2023-05-24T20:42:34.472 INFO:root:teuthology version: 0.0.1.dev105+g6fb4f0c 2023-05-24T20:42:34.473 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2023-05-24_20:32:11-rgw-wip-yuri-testing-2023-05-24-0738-reef-distro-default-smithi/7285505 --name yuriw-2023-05-24_20:32:11-rgw-wip-yuri-testing-2023-05-24-0738-reef-distro-default-smithi --description rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8} tasks/test_multi} --owner scheduled_yuriw@teuthology --verbose -- /home/teuthworker/archive/yuriw-2023-05-24_20:32:11-rgw-wip-yuri-testing-2023-05-24-0738-reef-distro-default-smithi/7285505/orig.config.yaml 2023-05-24T20:42:34.509 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-05-24T20:42:34.602 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-05-24_20:32:11-rgw-wip-yuri-testing-2023-05-24-0738-reef-distro-default-smithi/7285505 branch: wip-yuri-testing-2023-05-24-0738-reef description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8} tasks/test_multi} email: yweinste@redhat.com first_in_suite: false job_id: '7285505' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-05-24_20:32:11-rgw-wip-yuri-testing-2023-05-24-0738-reef-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri-testing-2023-05-24-0738-reef ceph: conf: client: debug rgw: 20 debug rgw sync: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync data inject err probability: 0.1 rgw sync log trim interval: 0 rgw sync meta inject err probability: 0.1 rgw sync obj etag verify: true setgroup: ceph setuser: ceph client.0: rgw data notify interval msec: 0 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) - \(POOL_APP_NOT_ENABLED\) sha1: 3395903b291ea37c38b39a05e6d9aa4f40306c80 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 3395903b291ea37c38b39a05e6d9aa4f40306c80 rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - enabled_features: - resharding endpoints: - c1.client.0 is_default: true is_master: true name: a zones: - endpoints: - c1.client.0 is_default: true is_master: true name: a1 - endpoints: - c1.client.1 name: a2 - enabled_features: - resharding endpoints: - c2.client.0 is_default: true name: b zones: - endpoints: - c2.client.0 is_default: true is_master: true name: b1 - endpoints: - c2.client.1 name: b2 rgw-multisite-tests: args: - tests.py selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-yuri-testing-2023-05-24-0738-reef sha1: 3395903b291ea37c38b39a05e6d9aa4f40306c80 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 seed: 8560 sha1: 3395903b291ea37c38b39a05e6d9aa4f40306c80 sleep_before_teardown: 0 subset: 111/12000 suite: rgw suite_branch: wip-yuri-testing-2023-05-24-0738-reef suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 3395903b291ea37c38b39a05e6d9aa4f40306c80 targets: smithi071.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA5CM/4hsfiqXE11jZAHgpOo2FbW4ZOzUXKXNJNz9fWWeDtJTYStjEV6efAd9vMM91OzJdMX3iMjxJTS8Y5i2uY= smithi157.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO+lmSy90RyAYeFfSD6aXgxRUJPCkLZkHkQxsxqLXA3D4XXntKSNvOhxkzpRSCW+Uo3pl5PdjBfPz+AOeUYtCwU= tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6fb4f0c910de3902d27909a642e8d50d731dd28c timestamp: 2023-05-24_20:32:11 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.23459 2023-05-24T20:42:34.603 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa; will attempt to use it 2023-05-24T20:42:34.603 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/tasks 2023-05-24T20:42:34.604 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-05-24T20:42:34.606 INFO:teuthology.task.internal:Checking packages... 2023-05-24T20:42:34.635 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3395903b291ea37c38b39a05e6d9aa4f40306c80' 2023-05-24T20:42:34.636 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-05-24T20:42:34.636 INFO:teuthology.packaging:ref: None 2023-05-24T20:42:34.637 INFO:teuthology.packaging:tag: None 2023-05-24T20:42:34.637 INFO:teuthology.packaging:branch: wip-yuri-testing-2023-05-24-0738-reef 2023-05-24T20:42:34.637 INFO:teuthology.packaging:sha1: 3395903b291ea37c38b39a05e6d9aa4f40306c80 2023-05-24T20:42:34.637 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri-testing-2023-05-24-0738-reef 2023-05-24T20:42:34.795 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-3443.g3395903b 2023-05-24T20:42:34.796 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-05-24T20:42:34.802 INFO:teuthology.task.internal:no buildpackages task found 2023-05-24T20:42:34.803 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-05-24T20:42:34.808 INFO:teuthology.task.internal:Saving configuration 2023-05-24T20:42:34.822 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-05-24T20:42:34.827 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-05-24T20:42:34.856 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi071.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-05-24_20:32:11-rgw-wip-yuri-testing-2023-05-24-0738-reef-distro-default-smithi/7285505', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-05-24 20:36:17.087875', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA5CM/4hsfiqXE11jZAHgpOo2FbW4ZOzUXKXNJNz9fWWeDtJTYStjEV6efAd9vMM91OzJdMX3iMjxJTS8Y5i2uY='} 2023-05-24T20:42:34.881 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi157.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-05-24_20:32:11-rgw-wip-yuri-testing-2023-05-24-0738-reef-distro-default-smithi/7285505', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-05-24 20:36:17.089397', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO+lmSy90RyAYeFfSD6aXgxRUJPCkLZkHkQxsxqLXA3D4XXntKSNvOhxkzpRSCW+Uo3pl5PdjBfPz+AOeUYtCwU='} 2023-05-24T20:42:34.881 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-05-24T20:42:34.887 INFO:teuthology.task.internal:roles: ubuntu@smithi071.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2023-05-24T20:42:34.888 INFO:teuthology.task.internal:roles: ubuntu@smithi157.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2023-05-24T20:42:34.888 INFO:teuthology.run_tasks:Running task console_log... 2023-05-24T20:42:34.966 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7faf13f01dc0>, signals=[15]) 2023-05-24T20:42:34.967 INFO:teuthology.run_tasks:Running task internal.connect... 2023-05-24T20:42:34.973 INFO:teuthology.task.internal:Opening connections... 2023-05-24T20:42:34.974 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi071.front.sepia.ceph.com 2023-05-24T20:42:34.975 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-05-24T20:42:35.044 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi157.front.sepia.ceph.com 2023-05-24T20:42:35.046 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-05-24T20:42:35.114 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-05-24T20:42:35.121 DEBUG:teuthology.orchestra.run.smithi071:> uname -m 2023-05-24T20:42:35.144 INFO:teuthology.orchestra.run.smithi071.stdout:x86_64 2023-05-24T20:42:35.145 DEBUG:teuthology.orchestra.run.smithi071:> cat /etc/os-release 2023-05-24T20:42:35.206 INFO:teuthology.orchestra.run.smithi071.stdout:NAME="CentOS Stream" 2023-05-24T20:42:35.207 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION="8" 2023-05-24T20:42:35.207 INFO:teuthology.orchestra.run.smithi071.stdout:ID="centos" 2023-05-24T20:42:35.207 INFO:teuthology.orchestra.run.smithi071.stdout:ID_LIKE="rhel fedora" 2023-05-24T20:42:35.207 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION_ID="8" 2023-05-24T20:42:35.208 INFO:teuthology.orchestra.run.smithi071.stdout:PLATFORM_ID="platform:el8" 2023-05-24T20:42:35.208 INFO:teuthology.orchestra.run.smithi071.stdout:PRETTY_NAME="CentOS Stream 8" 2023-05-24T20:42:35.208 INFO:teuthology.orchestra.run.smithi071.stdout:ANSI_COLOR="0;31" 2023-05-24T20:42:35.209 INFO:teuthology.orchestra.run.smithi071.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-05-24T20:42:35.209 INFO:teuthology.orchestra.run.smithi071.stdout:HOME_URL="https://centos.org/" 2023-05-24T20:42:35.209 INFO:teuthology.orchestra.run.smithi071.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-05-24T20:42:35.210 INFO:teuthology.orchestra.run.smithi071.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-05-24T20:42:35.210 INFO:teuthology.orchestra.run.smithi071.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-05-24T20:42:35.211 INFO:teuthology.lock.ops:Updating smithi071.front.sepia.ceph.com on lock server 2023-05-24T20:42:35.235 DEBUG:teuthology.orchestra.run.smithi157:> uname -m 2023-05-24T20:42:35.258 INFO:teuthology.orchestra.run.smithi157.stdout:x86_64 2023-05-24T20:42:35.258 DEBUG:teuthology.orchestra.run.smithi157:> cat /etc/os-release 2023-05-24T20:42:35.319 INFO:teuthology.orchestra.run.smithi157.stdout:NAME="CentOS Stream" 2023-05-24T20:42:35.320 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION="8" 2023-05-24T20:42:35.320 INFO:teuthology.orchestra.run.smithi157.stdout:ID="centos" 2023-05-24T20:42:35.321 INFO:teuthology.orchestra.run.smithi157.stdout:ID_LIKE="rhel fedora" 2023-05-24T20:42:35.321 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION_ID="8" 2023-05-24T20:42:35.321 INFO:teuthology.orchestra.run.smithi157.stdout:PLATFORM_ID="platform:el8" 2023-05-24T20:42:35.321 INFO:teuthology.orchestra.run.smithi157.stdout:PRETTY_NAME="CentOS Stream 8" 2023-05-24T20:42:35.321 INFO:teuthology.orchestra.run.smithi157.stdout:ANSI_COLOR="0;31" 2023-05-24T20:42:35.322 INFO:teuthology.orchestra.run.smithi157.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-05-24T20:42:35.322 INFO:teuthology.orchestra.run.smithi157.stdout:HOME_URL="https://centos.org/" 2023-05-24T20:42:35.322 INFO:teuthology.orchestra.run.smithi157.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-05-24T20:42:35.322 INFO:teuthology.orchestra.run.smithi157.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-05-24T20:42:35.323 INFO:teuthology.orchestra.run.smithi157.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-05-24T20:42:35.323 INFO:teuthology.lock.ops:Updating smithi157.front.sepia.ceph.com on lock server 2023-05-24T20:42:35.352 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-05-24T20:42:35.361 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-05-24T20:42:35.367 INFO:teuthology.task.internal:Checking for old test directory... 2023-05-24T20:42:35.367 DEBUG:teuthology.orchestra.run.smithi071:> test '!' -e /home/ubuntu/cephtest 2023-05-24T20:42:35.370 DEBUG:teuthology.orchestra.run.smithi157:> test '!' -e /home/ubuntu/cephtest 2023-05-24T20:42:35.390 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-05-24T20:42:35.396 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-05-24T20:42:35.397 DEBUG:teuthology.orchestra.run.smithi071:> test -z $(ls -A /var/lib/ceph) 2023-05-24T20:42:35.430 DEBUG:teuthology.orchestra.run.smithi157:> test -z $(ls -A /var/lib/ceph) 2023-05-24T20:42:35.462 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-05-24T20:42:35.521 INFO:teuthology.run_tasks:Running task kernel... 2023-05-24T20:42:35.536 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-05-24T20:42:35.537 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-05-24T20:42:35.537 DEBUG:teuthology.orchestra.run.smithi071:> test -f /run/.containerenv -o -f /.dockerenv 2023-05-24T20:42:35.538 DEBUG:teuthology.orchestra.run.smithi157:> test -f /run/.containerenv -o -f /.dockerenv 2023-05-24T20:42:35.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-24T20:42:35.557 DEBUG:teuthology.orchestra.run.smithi071:> uname -r 2023-05-24T20:42:35.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-24T20:42:35.559 DEBUG:teuthology.orchestra.run.smithi157:> uname -r 2023-05-24T20:42:35.617 INFO:teuthology.orchestra.run.smithi071.stdout:4.18.0-489.el8.x86_64 2023-05-24T20:42:35.617 INFO:teuthology.task.kernel:Running kernel on smithi071: 4.18.0-489.el8.x86_64 2023-05-24T20:42:35.617 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum install -y kernel 2023-05-24T20:42:35.620 INFO:teuthology.orchestra.run.smithi157.stdout:4.18.0-489.el8.x86_64 2023-05-24T20:42:35.620 INFO:teuthology.task.kernel:Running kernel on smithi157: 4.18.0-489.el8.x86_64 2023-05-24T20:42:35.620 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum install -y kernel 2023-05-24T20:42:40.343 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 8 - AppStream 14 MB/s | 29 MB 00:02 2023-05-24T20:42:41.003 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 8 - AppStream 10 MB/s | 29 MB 00:02 2023-05-24T20:42:48.035 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 8 - BaseOS 14 MB/s | 36 MB 00:02 2023-05-24T20:42:53.443 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 8 - Extras 105 kB/s | 18 kB 00:00 2023-05-24T20:42:53.836 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 8 - Extras common packages 29 kB/s | 6.6 kB 00:00 2023-05-24T20:42:54.187 INFO:teuthology.orchestra.run.smithi157.stdout:Copr repo for python3-asyncssh owned by ceph 17 kB/s | 3.5 kB 00:00 2023-05-24T20:42:55.485 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 8 - BaseOS 3.9 MB/s | 36 MB 00:09 2023-05-24T20:43:00.231 INFO:teuthology.orchestra.run.smithi157.stdout:Extra Packages for Enterprise Linux 2.4 MB/s | 14 MB 00:05 2023-05-24T20:43:01.084 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 8 - Extras 47 kB/s | 18 kB 00:00 2023-05-24T20:43:01.535 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 8 - Extras common packages 22 kB/s | 6.6 kB 00:00 2023-05-24T20:43:01.870 INFO:teuthology.orchestra.run.smithi071.stdout:Copr repo for python3-asyncssh owned by ceph 18 kB/s | 3.5 kB 00:00 2023-05-24T20:43:03.678 INFO:teuthology.orchestra.run.smithi157.stdout:lab-extras 304 kB/s | 24 kB 00:00 2023-05-24T20:43:04.490 INFO:teuthology.orchestra.run.smithi071.stdout:Extra Packages for Enterprise Linux 5.6 MB/s | 14 MB 00:02 2023-05-24T20:43:06.793 INFO:teuthology.orchestra.run.smithi157.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-05-24T20:43:06.794 INFO:teuthology.orchestra.run.smithi157.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-05-24T20:43:07.008 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-05-24T20:43:07.008 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:43:07.008 INFO:teuthology.orchestra.run.smithi157.stdout: Package Architecture Version Repository Size 2023-05-24T20:43:07.009 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:43:07.009 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-05-24T20:43:07.009 INFO:teuthology.orchestra.run.smithi157.stdout: kernel x86_64 4.18.0-492.el8 baseos 9.5 M 2023-05-24T20:43:07.009 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2023-05-24T20:43:07.009 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-core x86_64 4.18.0-492.el8 baseos 42 M 2023-05-24T20:43:07.009 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-modules x86_64 4.18.0-492.el8 baseos 35 M 2023-05-24T20:43:07.010 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:43:07.010 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-05-24T20:43:07.010 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:43:07.010 INFO:teuthology.orchestra.run.smithi157.stdout:Install 3 Packages 2023-05-24T20:43:07.010 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:43:07.011 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 87 M 2023-05-24T20:43:07.011 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 95 M 2023-05-24T20:43:07.011 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-05-24T20:43:07.870 INFO:teuthology.orchestra.run.smithi071.stdout:lab-extras 541 kB/s | 24 kB 00:00 2023-05-24T20:43:10.388 INFO:teuthology.orchestra.run.smithi157.stdout:(1/3): kernel-4.18.0-492.el8.x86_64.rpm 3.0 MB/s | 9.5 MB 00:03 2023-05-24T20:43:10.952 INFO:teuthology.orchestra.run.smithi071.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-05-24T20:43:10.953 INFO:teuthology.orchestra.run.smithi071.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-05-24T20:43:11.148 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-05-24T20:43:11.148 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:43:11.149 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2023-05-24T20:43:11.149 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:43:11.149 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-05-24T20:43:11.149 INFO:teuthology.orchestra.run.smithi071.stdout: kernel x86_64 4.18.0-492.el8 baseos 9.5 M 2023-05-24T20:43:11.149 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2023-05-24T20:43:11.150 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-core x86_64 4.18.0-492.el8 baseos 42 M 2023-05-24T20:43:11.150 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules x86_64 4.18.0-492.el8 baseos 35 M 2023-05-24T20:43:11.150 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:43:11.150 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-05-24T20:43:11.150 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:43:11.150 INFO:teuthology.orchestra.run.smithi071.stdout:Install 3 Packages 2023-05-24T20:43:11.150 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:43:11.151 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 87 M 2023-05-24T20:43:11.151 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 95 M 2023-05-24T20:43:11.151 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-05-24T20:43:12.838 INFO:teuthology.orchestra.run.smithi157.stdout:(2/3): kernel-modules-4.18.0-492.el8.x86_64.rpm 6.2 MB/s | 35 MB 00:05 2023-05-24T20:43:13.405 INFO:teuthology.orchestra.run.smithi157.stdout:(3/3): kernel-core-4.18.0-492.el8.x86_64.rpm 6.8 MB/s | 42 MB 00:06 2023-05-24T20:43:13.405 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-05-24T20:43:13.406 INFO:teuthology.orchestra.run.smithi157.stdout:Total 14 MB/s | 87 MB 00:06 2023-05-24T20:43:14.056 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-05-24T20:43:14.460 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-05-24T20:43:14.460 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-05-24T20:43:14.796 INFO:teuthology.orchestra.run.smithi071.stdout:(1/3): kernel-4.18.0-492.el8.x86_64.rpm 2.7 MB/s | 9.5 MB 00:03 2023-05-24T20:43:15.646 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-05-24T20:43:15.715 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-05-24T20:43:18.659 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-05-24T20:43:18.863 INFO:teuthology.orchestra.run.smithi071.stdout:(2/3): kernel-modules-4.18.0-492.el8.x86_64.rpm 4.6 MB/s | 35 MB 00:07 2023-05-24T20:43:19.209 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : kernel-core-4.18.0-492.el8.x86_64 1/3 2023-05-24T20:43:19.638 INFO:teuthology.orchestra.run.smithi071.stdout:(3/3): kernel-core-4.18.0-492.el8.x86_64.rpm 5.1 MB/s | 42 MB 00:08 2023-05-24T20:43:19.638 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-05-24T20:43:19.638 INFO:teuthology.orchestra.run.smithi071.stdout:Total 10 MB/s | 87 MB 00:08 2023-05-24T20:43:20.293 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-05-24T20:43:20.710 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-05-24T20:43:20.710 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-05-24T20:43:21.918 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-05-24T20:43:21.951 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-05-24T20:43:22.774 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-core-4.18.0-492.el8.x86_64 1/3 2023-05-24T20:43:23.150 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : kernel-modules-4.18.0-492.el8.x86_64 2/3 2023-05-24T20:43:24.709 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-05-24T20:43:25.245 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-core-4.18.0-492.el8.x86_64 1/3 2023-05-24T20:43:26.539 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-modules-4.18.0-492.el8.x86_64 2/3 2023-05-24T20:43:27.018 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : kernel-4.18.0-492.el8.x86_64 3/3 2023-05-24T20:43:28.838 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-core-4.18.0-492.el8.x86_64 1/3 2023-05-24T20:43:29.244 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-modules-4.18.0-492.el8.x86_64 2/3 2023-05-24T20:43:32.585 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-modules-4.18.0-492.el8.x86_64 2/3 2023-05-24T20:43:32.877 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-4.18.0-492.el8.x86_64 3/3 2023-05-24T20:43:48.590 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-core-4.18.0-492.el8.x86_64 3/3 2023-05-24T20:43:48.614 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-modules-4.18.0-492.el8.x86_64 3/3 2023-05-24T20:43:48.948 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-4.18.0-492.el8.x86_64 3/3 2023-05-24T20:43:48.949 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : kernel-4.18.0-492.el8.x86_64 1/3 2023-05-24T20:43:48.949 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : kernel-core-4.18.0-492.el8.x86_64 2/3 2023-05-24T20:43:49.603 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : kernel-modules-4.18.0-492.el8.x86_64 3/3 2023-05-24T20:43:49.603 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:43:49.603 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-05-24T20:43:49.604 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-4.18.0-492.el8.x86_64 kernel-core-4.18.0-492.el8.x86_64 2023-05-24T20:43:49.604 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-modules-4.18.0-492.el8.x86_64 2023-05-24T20:43:49.604 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:43:49.604 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-05-24T20:43:49.745 DEBUG:teuthology.orchestra.run.smithi157:> rpm -q kernel | sort -rV | head -n 1 2023-05-24T20:43:49.958 INFO:teuthology.orchestra.run.smithi157.stdout:kernel-4.18.0-492.el8.x86_64 2023-05-24T20:43:49.958 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-492.el8.x86_64 2023-05-24T20:43:49.959 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-492.el8.x86_64 2023-05-24T20:43:49.959 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-05-24T20:43:49.959 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-05-24T20:43:49.959 INFO:teuthology.task.kernel:Installing distro kernel on c2.mon.a... 2023-05-24T20:43:49.959 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi157.front.sepia.ceph.com, path=None, version=distro) 2023-05-24T20:43:49.960 DEBUG:teuthology.orchestra.run.smithi157:> rpm -q kernel | sort -rV | head -n 1 2023-05-24T20:43:50.163 INFO:teuthology.orchestra.run.smithi157.stdout:kernel-4.18.0-492.el8.x86_64 2023-05-24T20:43:50.164 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-492.el8.x86_64 2023-05-24T20:43:50.164 DEBUG:teuthology.orchestra.run.smithi157:> sudo rpm -qi grub2-tools 2023-05-24T20:43:50.197 INFO:teuthology.orchestra.run.smithi157.stdout:Name : grub2-tools 2023-05-24T20:43:50.198 INFO:teuthology.orchestra.run.smithi157.stdout:Epoch : 1 2023-05-24T20:43:50.198 INFO:teuthology.orchestra.run.smithi157.stdout:Version : 2.02 2023-05-24T20:43:50.198 INFO:teuthology.orchestra.run.smithi157.stdout:Release : 129.el8 2023-05-24T20:43:50.198 INFO:teuthology.orchestra.run.smithi157.stdout:Architecture: x86_64 2023-05-24T20:43:50.198 INFO:teuthology.orchestra.run.smithi157.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-05-24T20:43:50.199 INFO:teuthology.orchestra.run.smithi157.stdout:Group : System Environment/Base 2023-05-24T20:43:50.199 INFO:teuthology.orchestra.run.smithi157.stdout:Size : 9265586 2023-05-24T20:43:50.199 INFO:teuthology.orchestra.run.smithi157.stdout:License : GPLv3+ 2023-05-24T20:43:50.199 INFO:teuthology.orchestra.run.smithi157.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-05-24T20:43:50.199 INFO:teuthology.orchestra.run.smithi157.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-05-24T20:43:50.199 INFO:teuthology.orchestra.run.smithi157.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-05-24T20:43:50.199 INFO:teuthology.orchestra.run.smithi157.stdout:Build Host : kbuilder.bsys.centos.org 2023-05-24T20:43:50.200 INFO:teuthology.orchestra.run.smithi157.stdout:Relocations : (not relocatable) 2023-05-24T20:43:50.200 INFO:teuthology.orchestra.run.smithi157.stdout:Packager : CentOS BuildSystem 2023-05-24T20:43:50.200 INFO:teuthology.orchestra.run.smithi157.stdout:Vendor : CentOS 2023-05-24T20:43:50.200 INFO:teuthology.orchestra.run.smithi157.stdout:URL : http://www.gnu.org/software/grub/ 2023-05-24T20:43:50.200 INFO:teuthology.orchestra.run.smithi157.stdout:Summary : Support tools for GRUB. 2023-05-24T20:43:50.200 INFO:teuthology.orchestra.run.smithi157.stdout:Description : 2023-05-24T20:43:50.200 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:43:50.201 INFO:teuthology.orchestra.run.smithi157.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-05-24T20:43:50.201 INFO:teuthology.orchestra.run.smithi157.stdout:customizable bootloader with modular architecture. It supports a rich 2023-05-24T20:43:50.201 INFO:teuthology.orchestra.run.smithi157.stdout:variety of kernel formats, file systems, computer architectures and 2023-05-24T20:43:50.201 INFO:teuthology.orchestra.run.smithi157.stdout:hardware devices. 2023-05-24T20:43:50.201 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:43:50.201 INFO:teuthology.orchestra.run.smithi157.stdout:This subpackage provides tools for support of all platforms. 2023-05-24T20:43:50.202 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-05-24T20:43:50.202 INFO:teuthology.task.kernel:Updating grub on smithi157 to boot 4.18.0-492.el8.x86_64 2023-05-24T20:43:50.202 DEBUG:teuthology.orchestra.run.smithi157:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-05-24T20:43:51.753 INFO:teuthology.orchestra.run.smithi157.stderr:Generating grub configuration file ... 2023-05-24T20:43:54.217 INFO:teuthology.orchestra.run.smithi157.stderr:done 2023-05-24T20:43:54.218 DEBUG:teuthology.orchestra.run.smithi157:> mktemp 2023-05-24T20:43:54.253 INFO:teuthology.orchestra.run.smithi157.stdout:/tmp/tmp.1pyxxaSkZZ 2023-05-24T20:43:54.254 DEBUG:teuthology.orchestra.run.smithi157:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.1pyxxaSkZZ 2023-05-24T20:43:54.338 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0666 /tmp/tmp.1pyxxaSkZZ 2023-05-24T20:43:54.438 DEBUG:teuthology.orchestra.remote:smithi157:/tmp/tmp.1pyxxaSkZZ is 6KB 2023-05-24T20:43:54.486 DEBUG:teuthology.orchestra.run.smithi157:> rm -fr /tmp/tmp.1pyxxaSkZZ 2023-05-24T20:43:54.504 DEBUG:teuthology.orchestra.run.smithi157:> sudo /bin/ls /boot/loader/entries || true 2023-05-24T20:43:54.583 INFO:teuthology.orchestra.run.smithi157.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-05-24T20:43:54.583 INFO:teuthology.orchestra.run.smithi157.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-05-24T20:43:54.583 INFO:teuthology.orchestra.run.smithi157.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-05-24T20:43:54.583 INFO:teuthology.orchestra.run.smithi157.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-492.el8.x86_64.conf 2023-05-24T20:43:54.589 DEBUG:teuthology.orchestra.run.smithi157:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-492.el8.x86_64 2023-05-24T20:43:54.675 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-core-4.18.0-492.el8.x86_64 3/3 2023-05-24T20:43:54.698 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-modules-4.18.0-492.el8.x86_64 3/3 2023-05-24T20:43:54.757 DEBUG:teuthology.orchestra.run.smithi157:> sudo shutdown -r now 2023-05-24T20:43:55.032 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-4.18.0-492.el8.x86_64 3/3 2023-05-24T20:43:55.032 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-4.18.0-492.el8.x86_64 1/3 2023-05-24T20:43:55.033 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-core-4.18.0-492.el8.x86_64 2/3 2023-05-24T20:43:55.678 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-modules-4.18.0-492.el8.x86_64 3/3 2023-05-24T20:43:55.678 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:43:55.679 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-05-24T20:43:55.680 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-4.18.0-492.el8.x86_64 kernel-core-4.18.0-492.el8.x86_64 2023-05-24T20:43:55.680 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules-4.18.0-492.el8.x86_64 2023-05-24T20:43:55.680 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:43:55.681 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-05-24T20:43:55.839 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q kernel | sort -rV | head -n 1 2023-05-24T20:43:56.096 INFO:teuthology.orchestra.run.smithi071.stdout:kernel-4.18.0-492.el8.x86_64 2023-05-24T20:43:56.096 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-492.el8.x86_64 2023-05-24T20:43:56.097 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-492.el8.x86_64 2023-05-24T20:43:56.097 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-05-24T20:43:56.097 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-05-24T20:43:56.098 INFO:teuthology.task.kernel:Installing distro kernel on c1.mon.a... 2023-05-24T20:43:56.098 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi071.front.sepia.ceph.com, path=None, version=distro) 2023-05-24T20:43:56.098 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q kernel | sort -rV | head -n 1 2023-05-24T20:43:56.301 INFO:teuthology.orchestra.run.smithi071.stdout:kernel-4.18.0-492.el8.x86_64 2023-05-24T20:43:56.301 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-492.el8.x86_64 2023-05-24T20:43:56.302 DEBUG:teuthology.orchestra.run.smithi071:> sudo rpm -qi grub2-tools 2023-05-24T20:43:56.336 INFO:teuthology.orchestra.run.smithi071.stdout:Name : grub2-tools 2023-05-24T20:43:56.336 INFO:teuthology.orchestra.run.smithi071.stdout:Epoch : 1 2023-05-24T20:43:56.337 INFO:teuthology.orchestra.run.smithi071.stdout:Version : 2.02 2023-05-24T20:43:56.337 INFO:teuthology.orchestra.run.smithi071.stdout:Release : 129.el8 2023-05-24T20:43:56.337 INFO:teuthology.orchestra.run.smithi071.stdout:Architecture: x86_64 2023-05-24T20:43:56.337 INFO:teuthology.orchestra.run.smithi071.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-05-24T20:43:56.338 INFO:teuthology.orchestra.run.smithi071.stdout:Group : System Environment/Base 2023-05-24T20:43:56.338 INFO:teuthology.orchestra.run.smithi071.stdout:Size : 9265586 2023-05-24T20:43:56.338 INFO:teuthology.orchestra.run.smithi071.stdout:License : GPLv3+ 2023-05-24T20:43:56.338 INFO:teuthology.orchestra.run.smithi071.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-05-24T20:43:56.338 INFO:teuthology.orchestra.run.smithi071.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-05-24T20:43:56.338 INFO:teuthology.orchestra.run.smithi071.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-05-24T20:43:56.339 INFO:teuthology.orchestra.run.smithi071.stdout:Build Host : kbuilder.bsys.centos.org 2023-05-24T20:43:56.339 INFO:teuthology.orchestra.run.smithi071.stdout:Relocations : (not relocatable) 2023-05-24T20:43:56.339 INFO:teuthology.orchestra.run.smithi071.stdout:Packager : CentOS BuildSystem 2023-05-24T20:43:56.339 INFO:teuthology.orchestra.run.smithi071.stdout:Vendor : CentOS 2023-05-24T20:43:56.339 INFO:teuthology.orchestra.run.smithi071.stdout:URL : http://www.gnu.org/software/grub/ 2023-05-24T20:43:56.339 INFO:teuthology.orchestra.run.smithi071.stdout:Summary : Support tools for GRUB. 2023-05-24T20:43:56.340 INFO:teuthology.orchestra.run.smithi071.stdout:Description : 2023-05-24T20:43:56.340 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:43:56.340 INFO:teuthology.orchestra.run.smithi071.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-05-24T20:43:56.340 INFO:teuthology.orchestra.run.smithi071.stdout:customizable bootloader with modular architecture. It supports a rich 2023-05-24T20:43:56.340 INFO:teuthology.orchestra.run.smithi071.stdout:variety of kernel formats, file systems, computer architectures and 2023-05-24T20:43:56.341 INFO:teuthology.orchestra.run.smithi071.stdout:hardware devices. 2023-05-24T20:43:56.341 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:43:56.341 INFO:teuthology.orchestra.run.smithi071.stdout:This subpackage provides tools for support of all platforms. 2023-05-24T20:43:56.342 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-05-24T20:43:56.342 INFO:teuthology.task.kernel:Updating grub on smithi071 to boot 4.18.0-492.el8.x86_64 2023-05-24T20:43:56.343 DEBUG:teuthology.orchestra.run.smithi071:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-05-24T20:43:57.843 INFO:teuthology.orchestra.run.smithi071.stderr:Generating grub configuration file ... 2023-05-24T20:44:00.179 INFO:teuthology.orchestra.run.smithi071.stderr:done 2023-05-24T20:44:00.180 DEBUG:teuthology.orchestra.run.smithi071:> mktemp 2023-05-24T20:44:00.213 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.Q3WY0pLVPs 2023-05-24T20:44:00.213 DEBUG:teuthology.orchestra.run.smithi071:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Q3WY0pLVPs 2023-05-24T20:44:00.286 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.Q3WY0pLVPs 2023-05-24T20:44:00.424 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.Q3WY0pLVPs is 6KB 2023-05-24T20:44:00.472 DEBUG:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.Q3WY0pLVPs 2023-05-24T20:44:00.490 DEBUG:teuthology.orchestra.run.smithi071:> sudo /bin/ls /boot/loader/entries || true 2023-05-24T20:44:00.562 INFO:teuthology.orchestra.run.smithi071.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-05-24T20:44:00.563 INFO:teuthology.orchestra.run.smithi071.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-05-24T20:44:00.563 INFO:teuthology.orchestra.run.smithi071.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-05-24T20:44:00.563 INFO:teuthology.orchestra.run.smithi071.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-492.el8.x86_64.conf 2023-05-24T20:44:00.564 DEBUG:teuthology.orchestra.run.smithi071:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-492.el8.x86_64 2023-05-24T20:44:00.737 DEBUG:teuthology.orchestra.run.smithi071:> sudo shutdown -r now 2023-05-24T20:44:24.780 INFO:teuthology.task.kernel:Checking client c2.mon.a for new kernel version... 2023-05-24T20:44:24.781 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2023-05-24T20:44:24.781 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-05-24T20:44:30.744 INFO:teuthology.task.kernel:Checking client c1.mon.a for new kernel version... 2023-05-24T20:44:30.745 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2023-05-24T20:44:30.745 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-05-24T20:44:59.380 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2023-05-24T20:45:29.408 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2023-05-24T20:45:29.409 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-05-24T20:45:30.748 DEBUG:teuthology.orchestra.remote:timed out 2023-05-24T20:45:47.760 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2023-05-24T20:46:00.763 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2023-05-24T20:46:00.763 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-05-24T20:46:01.028 DEBUG:teuthology.orchestra.run.smithi071:> true 2023-05-24T20:46:01.309 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi071.front.sepia.ceph.com' 2023-05-24T20:46:01.309 INFO:teuthology.task.kernel:Checking kernel version of c1.mon.a, want "4.18.0-492.el8.x86_64"... 2023-05-24T20:46:01.309 DEBUG:teuthology.orchestra.run.smithi071:> uname -r 2023-05-24T20:46:01.330 INFO:teuthology.orchestra.run.smithi071.stdout:4.18.0-492.el8.x86_64 2023-05-24T20:46:01.330 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-492.el8.x86_64 vs 4.18.0-492.el8.x86_64 2023-05-24T20:46:01.330 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-05-24T20:46:01.330 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-05-24T20:46:02.331 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2023-05-24T20:46:02.332 DEBUG:teuthology.orchestra.run.smithi071:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-05-24T20:46:02.400 INFO:teuthology.orchestra.run.smithi071.stdout:ttyS1 2023-05-24T20:46:02.415 DEBUG:teuthology.parallel:result is None 2023-05-24T20:46:17.761 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2023-05-24T20:46:17.762 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-05-24T20:46:18.045 DEBUG:teuthology.orchestra.run.smithi157:> true 2023-05-24T20:46:18.325 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi157.front.sepia.ceph.com' 2023-05-24T20:46:18.325 INFO:teuthology.task.kernel:Checking kernel version of c2.mon.a, want "4.18.0-492.el8.x86_64"... 2023-05-24T20:46:18.326 DEBUG:teuthology.orchestra.run.smithi157:> uname -r 2023-05-24T20:46:18.345 INFO:teuthology.orchestra.run.smithi157.stdout:4.18.0-492.el8.x86_64 2023-05-24T20:46:18.346 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-492.el8.x86_64 vs 4.18.0-492.el8.x86_64 2023-05-24T20:46:18.346 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-05-24T20:46:18.346 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-05-24T20:46:19.346 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2023-05-24T20:46:19.347 DEBUG:teuthology.orchestra.run.smithi157:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-05-24T20:46:19.421 INFO:teuthology.orchestra.run.smithi157.stdout:ttyS1 2023-05-24T20:46:19.436 DEBUG:teuthology.parallel:result is None 2023-05-24T20:46:19.437 INFO:teuthology.run_tasks:Running task internal.base... 2023-05-24T20:46:19.445 INFO:teuthology.task.internal:Creating test directory... 2023-05-24T20:46:19.446 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-05-24T20:46:19.450 DEBUG:teuthology.orchestra.run.smithi157:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-05-24T20:46:19.472 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-05-24T20:46:19.489 INFO:teuthology.run_tasks:Running task internal.archive... 2023-05-24T20:46:19.506 INFO:teuthology.task.internal:Creating archive directory... 2023-05-24T20:46:19.507 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-05-24T20:46:19.512 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-05-24T20:46:19.543 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-05-24T20:46:19.551 INFO:teuthology.task.internal:Enabling coredump saving... 2023-05-24T20:46:19.551 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-05-24T20:46:19.580 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-05-24T20:46:19.620 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-05-24T20:46:19.621 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-05-24T20:46:19.631 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-05-24T20:46:19.632 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-05-24T20:46:19.633 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-05-24T20:46:19.640 INFO:teuthology.task.internal:Configuring sudo... 2023-05-24T20:46:19.641 DEBUG:teuthology.orchestra.run.smithi071:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-05-24T20:46:19.676 DEBUG:teuthology.orchestra.run.smithi157:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-05-24T20:46:19.706 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-05-24T20:46:19.715 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-05-24T20:46:19.716 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-05-24T20:46:19.747 DEBUG:teuthology.orchestra.run.smithi157:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-05-24T20:46:19.765 DEBUG:teuthology.orchestra.run.smithi071:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-05-24T20:46:19.855 DEBUG:teuthology.orchestra.run.smithi071:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-05-24T20:46:19.905 DEBUG:teuthology.orchestra.run.smithi071:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-05-24T20:46:19.993 DEBUG:teuthology.orchestra.run.smithi071:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-05-24T20:46:20.033 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-05-24T20:46:20.033 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-05-24T20:46:20.099 DEBUG:teuthology.orchestra.run.smithi157:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-05-24T20:46:20.148 DEBUG:teuthology.orchestra.run.smithi157:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-05-24T20:46:20.203 DEBUG:teuthology.orchestra.run.smithi157:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-05-24T20:46:20.294 DEBUG:teuthology.orchestra.run.smithi157:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-05-24T20:46:20.334 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-05-24T20:46:20.335 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-05-24T20:46:20.403 DEBUG:teuthology.orchestra.run.smithi071:> sudo service rsyslog restart 2023-05-24T20:46:20.405 DEBUG:teuthology.orchestra.run.smithi157:> sudo service rsyslog restart 2023-05-24T20:46:20.462 INFO:teuthology.orchestra.run.smithi071.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-05-24T20:46:20.500 INFO:teuthology.orchestra.run.smithi157.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-05-24T20:46:20.951 INFO:teuthology.run_tasks:Running task internal.timer... 2023-05-24T20:46:20.959 INFO:teuthology.task.internal:Starting timer... 2023-05-24T20:46:20.960 INFO:teuthology.run_tasks:Running task pcp... 2023-05-24T20:46:20.977 INFO:teuthology.run_tasks:Running task selinux... 2023-05-24T20:46:20.986 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-05-24T20:46:20.987 DEBUG:teuthology.orchestra.run.smithi071:> sudo service auditd rotate 2023-05-24T20:46:21.084 INFO:teuthology.orchestra.run.smithi071.stdout:Rotating logs: 2023-05-24T20:46:21.086 DEBUG:teuthology.orchestra.run.smithi157:> sudo service auditd rotate 2023-05-24T20:46:21.168 INFO:teuthology.orchestra.run.smithi157.stdout:Rotating logs: 2023-05-24T20:46:21.170 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-05-24T20:46:21.171 DEBUG:teuthology.orchestra.run.smithi071:> /usr/sbin/getenforce 2023-05-24T20:46:21.195 INFO:teuthology.orchestra.run.smithi071.stdout:Permissive 2023-05-24T20:46:21.196 DEBUG:teuthology.orchestra.run.smithi157:> /usr/sbin/getenforce 2023-05-24T20:46:21.239 INFO:teuthology.orchestra.run.smithi157.stdout:Permissive 2023-05-24T20:46:21.240 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi071.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi157.front.sepia.ceph.com': 'permissive'} 2023-05-24T20:46:21.240 DEBUG:teuthology.orchestra.run.smithi071:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-05-24T20:46:21.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-24T20:46:21.269 DEBUG:teuthology.orchestra.run.smithi157:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-05-24T20:46:21.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-24T20:46:21.309 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-05-24T20:46:21.309 DEBUG:teuthology.orchestra.run.smithi071:> sudo /usr/sbin/setenforce permissive 2023-05-24T20:46:21.337 DEBUG:teuthology.orchestra.run.smithi157:> sudo /usr/sbin/setenforce permissive 2023-05-24T20:46:21.378 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-05-24T20:46:21.390 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-05-24T20:46:21.395 INFO:teuthology.repo_utils:Fetching main from origin 2023-05-24T20:46:21.533 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-05-24T20:46:21.556 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-05-24T20:46:21.558 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi071.front.sepia.ceph.com,smithi157.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-05-24T20:52:49.143 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi071.front.sepia.ceph.com'), Remote(name='ubuntu@smithi157.front.sepia.ceph.com')] 2023-05-24T20:52:49.145 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2023-05-24T20:52:49.146 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-05-24T20:52:49.216 DEBUG:teuthology.orchestra.run.smithi071:> true 2023-05-24T20:52:49.283 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2023-05-24T20:52:49.284 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-05-24T20:52:49.354 DEBUG:teuthology.orchestra.run.smithi157:> true 2023-05-24T20:52:49.430 INFO:teuthology.run_tasks:Running task clock... 2023-05-24T20:52:49.441 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-05-24T20:52:49.442 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-05-24T20:52:49.443 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-05-24T20:52:49.447 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-05-24T20:52:49.447 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-05-24T20:52:49.478 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-05-24T20:52:49.492 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-05-24T20:52:49.505 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-05-24T20:52:49.520 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-05-24T20:52:49.558 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: ntpd: command not found 2023-05-24T20:52:49.570 INFO:teuthology.orchestra.run.smithi071.stdout:506 Cannot talk to daemon 2023-05-24T20:52:49.584 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-05-24T20:52:49.592 INFO:teuthology.orchestra.run.smithi157.stderr:sudo: ntpd: command not found 2023-05-24T20:52:49.597 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-05-24T20:52:49.604 INFO:teuthology.orchestra.run.smithi157.stdout:506 Cannot talk to daemon 2023-05-24T20:52:49.618 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-05-24T20:52:49.632 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-05-24T20:52:49.664 INFO:teuthology.orchestra.run.smithi071.stderr:bash: ntpq: command not found 2023-05-24T20:52:49.668 INFO:teuthology.orchestra.run.smithi071.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-05-24T20:52:49.668 INFO:teuthology.orchestra.run.smithi071.stdout:=============================================================================== 2023-05-24T20:52:49.668 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-05-24T20:52:49.669 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-05-24T20:52:49.669 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-05-24T20:52:49.669 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-05-24T20:52:49.694 INFO:teuthology.orchestra.run.smithi157.stderr:bash: ntpq: command not found 2023-05-24T20:52:49.698 INFO:teuthology.orchestra.run.smithi157.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-05-24T20:52:49.699 INFO:teuthology.orchestra.run.smithi157.stdout:=============================================================================== 2023-05-24T20:52:49.699 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-05-24T20:52:49.699 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-05-24T20:52:49.699 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-05-24T20:52:49.699 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-05-24T20:52:49.699 INFO:teuthology.run_tasks:Running task install... 2023-05-24T20:52:49.711 DEBUG:teuthology.task.install:project ceph 2023-05-24T20:52:49.711 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3395903b291ea37c38b39a05e6d9aa4f40306c80'}} 2023-05-24T20:52:49.712 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '3395903b291ea37c38b39a05e6d9aa4f40306c80'} 2023-05-24T20:52:49.712 INFO:teuthology.task.install:Using flavor: default 2023-05-24T20:52:49.722 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-05-24T20:52:49.722 INFO:teuthology.task.install:extra packages: [] 2023-05-24T20:52:49.723 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3395903b291ea37c38b39a05e6d9aa4f40306c80', 'tag': None, 'wait_for_package': False} 2023-05-24T20:52:49.723 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3395903b291ea37c38b39a05e6d9aa4f40306c80 2023-05-24T20:52:49.726 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3395903b291ea37c38b39a05e6d9aa4f40306c80', 'tag': None, 'wait_for_package': False} 2023-05-24T20:52:49.726 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3395903b291ea37c38b39a05e6d9aa4f40306c80 2023-05-24T20:52:49.868 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-05-24-0738-reef/3395903b291ea37c38b39a05e6d9aa4f40306c80/centos/8/flavors/default/ 2023-05-24T20:52:49.868 INFO:teuthology.task.install.rpm:Package version is 18.0.0-3443.g3395903b 2023-05-24T20:52:49.878 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-05-24-0738-reef/3395903b291ea37c38b39a05e6d9aa4f40306c80/centos/8/flavors/default/ 2023-05-24T20:52:49.878 INFO:teuthology.task.install.rpm:Package version is 18.0.0-3443.g3395903b 2023-05-24T20:52:49.975 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-05-24-0738-reef/3395903b291ea37c38b39a05e6d9aa4f40306c80/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-05-24-0738-reef/3395903b291ea37c38b39a05e6d9aa4f40306c80/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-05-24-0738-reef/3395903b291ea37c38b39a05e6d9aa4f40306c80/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-05-24T20:52:49.976 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-05-24T20:52:49.976 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-05-24T20:52:49.983 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-05-24-0738-reef/3395903b291ea37c38b39a05e6d9aa4f40306c80/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-05-24-0738-reef/3395903b291ea37c38b39a05e6d9aa4f40306c80/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-05-24-0738-reef/3395903b291ea37c38b39a05e6d9aa4f40306c80/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-05-24T20:52:49.984 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-05-24T20:52:49.984 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-05-24T20:52:50.004 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-05-24T20:52:50.004 DEBUG:teuthology.orchestra.run.smithi071:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3395903b291ea37c38b39a05e6d9aa4f40306c80/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-05-24T20:52:50.011 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-05-24T20:52:50.011 DEBUG:teuthology.orchestra.run.smithi157:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3395903b291ea37c38b39a05e6d9aa4f40306c80/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-05-24T20:52:50.072 DEBUG:teuthology.orchestra.run.smithi071:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-05-24T20:52:50.080 DEBUG:teuthology.orchestra.run.smithi157:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-05-24T20:52:50.149 DEBUG:teuthology.orchestra.run.smithi071:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-05-24T20:52:50.157 DEBUG:teuthology.orchestra.run.smithi157:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-05-24T20:52:50.216 INFO:teuthology.orchestra.run.smithi071.stdout:check_obsoletes = 1 2023-05-24T20:52:50.217 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum clean all 2023-05-24T20:52:50.223 INFO:teuthology.orchestra.run.smithi157.stdout:check_obsoletes = 1 2023-05-24T20:52:50.225 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum clean all 2023-05-24T20:52:50.472 INFO:teuthology.orchestra.run.smithi071.stdout:55 files removed 2023-05-24T20:52:50.486 INFO:teuthology.orchestra.run.smithi157.stdout:55 files removed 2023-05-24T20:52:50.500 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-radosgw 2023-05-24T20:52:50.514 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-radosgw 2023-05-24T20:52:51.569 INFO:teuthology.orchestra.run.smithi071.stdout:ceph packages for x86_64 119 kB/s | 80 kB 00:00 2023-05-24T20:52:51.583 INFO:teuthology.orchestra.run.smithi157.stdout:ceph packages for x86_64 122 kB/s | 80 kB 00:00 2023-05-24T20:52:51.937 INFO:teuthology.orchestra.run.smithi071.stdout:ceph noarch packages 28 kB/s | 9.4 kB 00:00 2023-05-24T20:52:51.959 INFO:teuthology.orchestra.run.smithi157.stdout:ceph noarch packages 28 kB/s | 9.4 kB 00:00 2023-05-24T20:52:52.512 INFO:teuthology.orchestra.run.smithi071.stdout:ceph source packages 2.7 kB/s | 1.5 kB 00:00 2023-05-24T20:52:52.526 INFO:teuthology.orchestra.run.smithi157.stdout:ceph source packages 2.8 kB/s | 1.5 kB 00:00 2023-05-24T20:52:53.712 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 8 - AppStream 25 MB/s | 29 MB 00:01 2023-05-24T20:52:54.043 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 8 - AppStream 19 MB/s | 29 MB 00:01 2023-05-24T20:53:00.979 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 8 - BaseOS 19 MB/s | 36 MB 00:01 2023-05-24T20:53:01.177 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 8 - BaseOS 19 MB/s | 36 MB 00:01 2023-05-24T20:53:06.545 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 8 - Extras 65 kB/s | 18 kB 00:00 2023-05-24T20:53:06.710 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 8 - Extras 68 kB/s | 18 kB 00:00 2023-05-24T20:53:07.063 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 8 - Extras common packages 18 kB/s | 6.6 kB 00:00 2023-05-24T20:53:07.314 INFO:teuthology.orchestra.run.smithi071.stdout:Copr repo for python3-asyncssh owned by ceph 31 kB/s | 3.5 kB 00:00 2023-05-24T20:53:08.408 INFO:teuthology.orchestra.run.smithi071.stdout:Extra Packages for Enterprise Linux 15 MB/s | 14 MB 00:00 2023-05-24T20:53:11.781 INFO:teuthology.orchestra.run.smithi071.stdout:lab-extras 398 kB/s | 24 kB 00:00 2023-05-24T20:53:12.077 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 8 - Extras common packages 1.3 kB/s | 6.6 kB 00:05 2023-05-24T20:53:12.362 INFO:teuthology.orchestra.run.smithi157.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2023-05-24T20:53:13.712 INFO:teuthology.orchestra.run.smithi157.stdout:Extra Packages for Enterprise Linux 11 MB/s | 14 MB 00:01 2023-05-24T20:53:15.248 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-05-24T20:53:15.250 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:53:15.251 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2023-05-24T20:53:15.251 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:53:15.251 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-05-24T20:53:15.251 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-radosgw x86_64 2:18.0.0-3443.g3395903b.el8 ceph 13 M 2023-05-24T20:53:15.251 INFO:teuthology.orchestra.run.smithi071.stdout:Upgrading: 2023-05-24T20:53:15.251 INFO:teuthology.orchestra.run.smithi071.stdout: librados2 x86_64 2:18.0.0-3443.g3395903b.el8 ceph 3.7 M 2023-05-24T20:53:15.252 INFO:teuthology.orchestra.run.smithi071.stdout: librbd1 x86_64 2:18.0.0-3443.g3395903b.el8 ceph 3.6 M 2023-05-24T20:53:15.252 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2023-05-24T20:53:15.252 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-base x86_64 2:18.0.0-3443.g3395903b.el8 ceph 5.3 M 2023-05-24T20:53:15.252 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-common x86_64 2:18.0.0-3443.g3395903b.el8 ceph 22 M 2023-05-24T20:53:15.253 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-selinux x86_64 2:18.0.0-3443.g3395903b.el8 ceph 24 k 2023-05-24T20:53:15.253 INFO:teuthology.orchestra.run.smithi071.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-05-24T20:53:15.253 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs2 x86_64 2:18.0.0-3443.g3395903b.el8 ceph 767 k 2023-05-24T20:53:15.253 INFO:teuthology.orchestra.run.smithi071.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-05-24T20:53:15.253 INFO:teuthology.orchestra.run.smithi071.stdout: librabbitmq x86_64 0.9.0-3.el8 baseos 47 k 2023-05-24T20:53:15.253 INFO:teuthology.orchestra.run.smithi071.stdout: libradosstriper1 x86_64 2:18.0.0-3443.g3395903b.el8 ceph 474 k 2023-05-24T20:53:15.253 INFO:teuthology.orchestra.run.smithi071.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-05-24T20:53:15.254 INFO:teuthology.orchestra.run.smithi071.stdout: librgw2 x86_64 2:18.0.0-3443.g3395903b.el8 ceph 6.5 M 2023-05-24T20:53:15.254 INFO:teuthology.orchestra.run.smithi071.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-05-24T20:53:15.254 INFO:teuthology.orchestra.run.smithi071.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-05-24T20:53:15.254 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-argparse x86_64 2:18.0.0-3443.g3395903b.el8 ceph 45 k 2023-05-24T20:53:15.254 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-common x86_64 2:18.0.0-3443.g3395903b.el8 ceph 119 k 2023-05-24T20:53:15.254 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cephfs x86_64 2:18.0.0-3443.g3395903b.el8 ceph 190 k 2023-05-24T20:53:15.254 INFO:teuthology.orchestra.run.smithi071.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-05-24T20:53:15.255 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rados x86_64 2:18.0.0-3443.g3395903b.el8 ceph 371 k 2023-05-24T20:53:15.255 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rbd x86_64 2:18.0.0-3443.g3395903b.el8 ceph 364 k 2023-05-24T20:53:15.255 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rgw x86_64 2:18.0.0-3443.g3395903b.el8 ceph 106 k 2023-05-24T20:53:15.255 INFO:teuthology.orchestra.run.smithi071.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-05-24T20:53:15.255 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:53:15.255 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-05-24T20:53:15.255 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:53:15.255 INFO:teuthology.orchestra.run.smithi071.stdout:Install 21 Packages 2023-05-24T20:53:15.256 INFO:teuthology.orchestra.run.smithi071.stdout:Upgrade 2 Packages 2023-05-24T20:53:15.256 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:53:15.256 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 59 M 2023-05-24T20:53:15.257 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-05-24T20:53:16.849 INFO:teuthology.orchestra.run.smithi071.stdout:(1/23): ceph-base-18.0.0-3443.g3395903b.el8.x86 3.6 MB/s | 5.3 MB 00:01 2023-05-24T20:53:16.907 INFO:teuthology.orchestra.run.smithi071.stdout:(2/23): ceph-selinux-18.0.0-3443.g3395903b.el8. 419 kB/s | 24 kB 00:00 2023-05-24T20:53:17.074 INFO:teuthology.orchestra.run.smithi071.stdout:(3/23): libcephfs2-18.0.0-3443.g3395903b.el8.x8 4.5 MB/s | 767 kB 00:00 2023-05-24T20:53:17.163 INFO:teuthology.orchestra.run.smithi157.stdout:lab-extras 383 kB/s | 24 kB 00:00 2023-05-24T20:53:17.191 INFO:teuthology.orchestra.run.smithi071.stdout:(4/23): libradosstriper1-18.0.0-3443.g3395903b. 4.0 MB/s | 474 kB 00:00 2023-05-24T20:53:17.616 INFO:teuthology.orchestra.run.smithi071.stdout:(5/23): ceph-radosgw-18.0.0-3443.g3395903b.el8. 5.8 MB/s | 13 MB 00:02 2023-05-24T20:53:17.683 INFO:teuthology.orchestra.run.smithi071.stdout:(6/23): python3-ceph-argparse-18.0.0-3443.g3395 672 kB/s | 45 kB 00:00 2023-05-24T20:53:17.767 INFO:teuthology.orchestra.run.smithi071.stdout:(7/23): python3-ceph-common-18.0.0-3443.g339590 1.4 MB/s | 119 kB 00:00 2023-05-24T20:53:17.859 INFO:teuthology.orchestra.run.smithi071.stdout:(8/23): python3-cephfs-18.0.0-3443.g3395903b.el 2.0 MB/s | 190 kB 00:00 2023-05-24T20:53:17.975 INFO:teuthology.orchestra.run.smithi071.stdout:(9/23): python3-rados-18.0.0-3443.g3395903b.el8 3.1 MB/s | 371 kB 00:00 2023-05-24T20:53:18.084 INFO:teuthology.orchestra.run.smithi071.stdout:(10/23): python3-rbd-18.0.0-3443.g3395903b.el8. 3.3 MB/s | 364 kB 00:00 2023-05-24T20:53:18.293 INFO:teuthology.orchestra.run.smithi071.stdout:(11/23): ceph-common-18.0.0-3443.g3395903b.el8. 7.4 MB/s | 22 MB 00:02 2023-05-24T20:53:18.318 INFO:teuthology.orchestra.run.smithi071.stdout:(12/23): python3-rgw-18.0.0-3443.g3395903b.el8. 456 kB/s | 106 kB 00:00 2023-05-24T20:53:18.402 INFO:teuthology.orchestra.run.smithi071.stdout:(13/23): librgw2-18.0.0-3443.g3395903b.el8.x86_ 5.4 MB/s | 6.5 MB 00:01 2023-05-24T20:53:18.477 INFO:teuthology.orchestra.run.smithi071.stdout:(14/23): librdkafka-0.11.4-3.el8.x86_64.rpm 1.9 MB/s | 354 kB 00:00 2023-05-24T20:53:18.503 INFO:teuthology.orchestra.run.smithi071.stdout:(15/23): python3-prettytable-0.7.2-14.el8.noarc 438 kB/s | 44 kB 00:00 2023-05-24T20:53:18.528 INFO:teuthology.orchestra.run.smithi071.stdout:(16/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.2 MB/s | 259 kB 00:00 2023-05-24T20:53:18.895 INFO:teuthology.orchestra.run.smithi071.stdout:(17/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 112 kB/s | 47 kB 00:00 2023-05-24T20:53:20.586 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-05-24T20:53:20.588 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:53:20.589 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2023-05-24T20:53:20.589 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:53:20.589 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-05-24T20:53:20.590 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-radosgw x86_64 2:18.0.0-3443.g3395903b.el8 ceph 13 M 2023-05-24T20:53:20.590 INFO:teuthology.orchestra.run.smithi157.stdout:Upgrading: 2023-05-24T20:53:20.590 INFO:teuthology.orchestra.run.smithi157.stdout: librados2 x86_64 2:18.0.0-3443.g3395903b.el8 ceph 3.7 M 2023-05-24T20:53:20.590 INFO:teuthology.orchestra.run.smithi157.stdout: librbd1 x86_64 2:18.0.0-3443.g3395903b.el8 ceph 3.6 M 2023-05-24T20:53:20.591 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2023-05-24T20:53:20.591 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-base x86_64 2:18.0.0-3443.g3395903b.el8 ceph 5.3 M 2023-05-24T20:53:20.591 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-common x86_64 2:18.0.0-3443.g3395903b.el8 ceph 22 M 2023-05-24T20:53:20.592 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-selinux x86_64 2:18.0.0-3443.g3395903b.el8 ceph 24 k 2023-05-24T20:53:20.592 INFO:teuthology.orchestra.run.smithi157.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-05-24T20:53:20.592 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs2 x86_64 2:18.0.0-3443.g3395903b.el8 ceph 767 k 2023-05-24T20:53:20.592 INFO:teuthology.orchestra.run.smithi157.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-05-24T20:53:20.593 INFO:teuthology.orchestra.run.smithi157.stdout: librabbitmq x86_64 0.9.0-3.el8 baseos 47 k 2023-05-24T20:53:20.593 INFO:teuthology.orchestra.run.smithi157.stdout: libradosstriper1 x86_64 2:18.0.0-3443.g3395903b.el8 ceph 474 k 2023-05-24T20:53:20.593 INFO:teuthology.orchestra.run.smithi157.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-05-24T20:53:20.593 INFO:teuthology.orchestra.run.smithi157.stdout: librgw2 x86_64 2:18.0.0-3443.g3395903b.el8 ceph 6.5 M 2023-05-24T20:53:20.593 INFO:teuthology.orchestra.run.smithi157.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-05-24T20:53:20.594 INFO:teuthology.orchestra.run.smithi157.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-05-24T20:53:20.594 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-argparse x86_64 2:18.0.0-3443.g3395903b.el8 ceph 45 k 2023-05-24T20:53:20.594 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-common x86_64 2:18.0.0-3443.g3395903b.el8 ceph 119 k 2023-05-24T20:53:20.594 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cephfs x86_64 2:18.0.0-3443.g3395903b.el8 ceph 190 k 2023-05-24T20:53:20.594 INFO:teuthology.orchestra.run.smithi157.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-05-24T20:53:20.594 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rados x86_64 2:18.0.0-3443.g3395903b.el8 ceph 371 k 2023-05-24T20:53:20.594 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rbd x86_64 2:18.0.0-3443.g3395903b.el8 ceph 364 k 2023-05-24T20:53:20.595 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rgw x86_64 2:18.0.0-3443.g3395903b.el8 ceph 106 k 2023-05-24T20:53:20.595 INFO:teuthology.orchestra.run.smithi157.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-05-24T20:53:20.595 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:53:20.595 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-05-24T20:53:20.595 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:53:20.595 INFO:teuthology.orchestra.run.smithi157.stdout:Install 21 Packages 2023-05-24T20:53:20.595 INFO:teuthology.orchestra.run.smithi157.stdout:Upgrade 2 Packages 2023-05-24T20:53:20.595 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:53:20.596 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 59 M 2023-05-24T20:53:20.596 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-05-24T20:53:21.855 INFO:teuthology.orchestra.run.smithi157.stdout:(1/23): ceph-base-18.0.0-3443.g3395903b.el8.x86 7.0 MB/s | 5.3 MB 00:00 2023-05-24T20:53:21.922 INFO:teuthology.orchestra.run.smithi157.stdout:(2/23): ceph-selinux-18.0.0-3443.g3395903b.el8. 365 kB/s | 24 kB 00:00 2023-05-24T20:53:22.064 INFO:teuthology.orchestra.run.smithi157.stdout:(3/23): libcephfs2-18.0.0-3443.g3395903b.el8.x8 5.3 MB/s | 767 kB 00:00 2023-05-24T20:53:22.223 INFO:teuthology.orchestra.run.smithi157.stdout:(4/23): libradosstriper1-18.0.0-3443.g3395903b. 2.9 MB/s | 474 kB 00:00 2023-05-24T20:53:22.631 INFO:teuthology.orchestra.run.smithi157.stdout:(5/23): ceph-radosgw-18.0.0-3443.g3395903b.el8. 8.4 MB/s | 13 MB 00:01 2023-05-24T20:53:22.723 INFO:teuthology.orchestra.run.smithi157.stdout:(6/23): python3-ceph-argparse-18.0.0-3443.g3395 488 kB/s | 45 kB 00:00 2023-05-24T20:53:22.815 INFO:teuthology.orchestra.run.smithi157.stdout:(7/23): python3-ceph-common-18.0.0-3443.g339590 1.3 MB/s | 119 kB 00:00 2023-05-24T20:53:22.924 INFO:teuthology.orchestra.run.smithi157.stdout:(8/23): python3-cephfs-18.0.0-3443.g3395903b.el 1.7 MB/s | 190 kB 00:00 2023-05-24T20:53:23.049 INFO:teuthology.orchestra.run.smithi157.stdout:(9/23): librgw2-18.0.0-3443.g3395903b.el8.x86_6 7.9 MB/s | 6.5 MB 00:00 2023-05-24T20:53:23.074 INFO:teuthology.orchestra.run.smithi157.stdout:(10/23): python3-rados-18.0.0-3443.g3395903b.el 2.4 MB/s | 371 kB 00:00 2023-05-24T20:53:23.275 INFO:teuthology.orchestra.run.smithi157.stdout:(11/23): ceph-common-18.0.0-3443.g3395903b.el8. 9.9 MB/s | 22 MB 00:02 2023-05-24T20:53:23.291 INFO:teuthology.orchestra.run.smithi157.stdout:(12/23): python3-rgw-18.0.0-3443.g3395903b.el8. 491 kB/s | 106 kB 00:00 2023-05-24T20:53:23.325 INFO:teuthology.orchestra.run.smithi157.stdout:(13/23): python3-rbd-18.0.0-3443.g3395903b.el8. 1.3 MB/s | 364 kB 00:00 2023-05-24T20:53:23.417 INFO:teuthology.orchestra.run.smithi157.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 479 kB/s | 44 kB 00:00 2023-05-24T20:53:23.443 INFO:teuthology.orchestra.run.smithi157.stdout:(15/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.7 MB/s | 259 kB 00:00 2023-05-24T20:53:23.496 INFO:teuthology.orchestra.run.smithi157.stdout:(16/23): librdkafka-0.11.4-3.el8.x86_64.rpm 1.6 MB/s | 354 kB 00:00 2023-05-24T20:53:23.677 INFO:teuthology.orchestra.run.smithi157.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 333 kB/s | 59 kB 00:00 2023-05-24T20:53:23.710 INFO:teuthology.orchestra.run.smithi157.stdout:(18/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.1 MB/s | 306 kB 00:00 2023-05-24T20:53:23.736 INFO:teuthology.orchestra.run.smithi157.stdout:(19/23): libunwind-1.3.1-3.el8.x86_64.rpm 1.2 MB/s | 75 kB 00:00 2023-05-24T20:53:23.761 INFO:teuthology.orchestra.run.smithi157.stdout:(20/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 136 kB/s | 47 kB 00:00 2023-05-24T20:53:23.919 INFO:teuthology.orchestra.run.smithi157.stdout:(21/23): thrift-0.13.0-2.el8.x86_64.rpm 8.3 MB/s | 1.7 MB 00:00 2023-05-24T20:53:24.132 INFO:teuthology.orchestra.run.smithi157.stdout:(22/23): librados2-18.0.0-3443.g3395903b.el8.x8 9.2 MB/s | 3.7 MB 00:00 2023-05-24T20:53:24.187 INFO:teuthology.orchestra.run.smithi157.stdout:(23/23): librbd1-18.0.0-3443.g3395903b.el8.x86_ 8.4 MB/s | 3.6 MB 00:00 2023-05-24T20:53:24.190 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-05-24T20:53:24.191 INFO:teuthology.orchestra.run.smithi157.stdout:Total 16 MB/s | 59 MB 00:03 2023-05-24T20:53:24.233 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-05-24T20:53:24.279 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-05-24T20:53:24.279 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-05-24T20:53:24.869 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-05-24T20:53:24.872 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-05-24T20:53:26.160 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-05-24T20:53:26.347 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2023-05-24T20:53:26.584 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/25 2023-05-24T20:53:26.791 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/25 2023-05-24T20:53:27.211 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/25 2023-05-24T20:53:27.458 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : librados2-2:18.0.0-3443.g3395903b.el8.x86_64 4/25 2023-05-24T20:53:27.554 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librados2-2:18.0.0-3443.g3395903b.el8.x86_64 4/25 2023-05-24T20:53:27.847 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rados-2:18.0.0-3443.g3395903b.el8.x86_64 5/25 2023-05-24T20:53:28.057 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libcephfs2-2:18.0.0-3443.g3395903b.el8.x86_64 6/25 2023-05-24T20:53:28.464 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libcephfs2-2:18.0.0-3443.g3395903b.el8.x86_64 6/25 2023-05-24T20:53:28.665 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : librbd1-2:18.0.0-3443.g3395903b.el8.x86_64 7/25 2023-05-24T20:53:28.727 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librbd1-2:18.0.0-3443.g3395903b.el8.x86_64 7/25 2023-05-24T20:53:28.931 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/25 2023-05-24T20:53:29.107 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2023-05-24T20:53:29.164 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2023-05-24T20:53:29.376 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-ceph-argparse-2:18.0.0-3443.g3395903b.el8. 10/25 2023-05-24T20:53:29.651 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cephfs-2:18.0.0-3443.g3395903b.el8.x86_64 11/25 2023-05-24T20:53:29.899 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rbd-2:18.0.0-3443.g3395903b.el8.x86_64 12/25 2023-05-24T20:53:30.065 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libradosstriper1-2:18.0.0-3443.g3395903b.el8.x86_6 13/25 2023-05-24T20:53:30.279 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libradosstriper1-2:18.0.0-3443.g3395903b.el8.x86_6 13/25 2023-05-24T20:53:30.492 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 14/25 2023-05-24T20:53:31.277 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 15/25 2023-05-24T20:53:31.415 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librgw2-2:18.0.0-3443.g3395903b.el8.x86_64 16/25 2023-05-24T20:53:31.492 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librgw2-2:18.0.0-3443.g3395903b.el8.x86_64 16/25 2023-05-24T20:53:31.706 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rgw-2:18.0.0-3443.g3395903b.el8.x86_64 17/25 2023-05-24T20:53:31.933 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-05-24T20:53:32.149 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-ceph-common-2:18.0.0-3443.g3395903b.el8.x8 19/25 2023-05-24T20:53:34.661 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-common-2:18.0.0-3443.g3395903b.el8.x86_64 20/25 2023-05-24T20:53:34.856 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-common-2:18.0.0-3443.g3395903b.el8.x86_64 20/25 2023-05-24T20:53:35.456 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-common-2:18.0.0-3443.g3395903b.el8.x86_64 20/25 2023-05-24T20:53:35.731 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-base-2:18.0.0-3443.g3395903b.el8.x86_64 21/25 2023-05-24T20:53:35.858 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-base-2:18.0.0-3443.g3395903b.el8.x86_64 21/25 2023-05-24T20:53:36.031 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-selinux-2:18.0.0-3443.g3395903b.el8.x86_64 22/25 2023-05-24T20:53:54.699 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-selinux-2:18.0.0-3443.g3395903b.el8.x86_64 22/25 2023-05-24T20:53:54.699 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /sys 2023-05-24T20:53:54.699 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /proc 2023-05-24T20:53:54.700 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /dev 2023-05-24T20:53:54.700 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /run 2023-05-24T20:53:54.700 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /mnt 2023-05-24T20:53:54.700 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /var/tmp 2023-05-24T20:53:54.700 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /home 2023-05-24T20:53:54.701 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /tmp 2023-05-24T20:53:54.701 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /dev 2023-05-24T20:53:54.701 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:53:56.078 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-radosgw-2:18.0.0-3443.g3395903b.el8.x86_64 23/25 2023-05-24T20:53:56.099 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-3443.g3395903b.el8.x86_64 23/25 2023-05-24T20:53:56.099 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-05-24T20:53:56.330 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-05-24T20:53:56.330 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-05-24T20:53:56.998 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-05-24T20:53:56.999 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-base-2:18.0.0-3443.g3395903b.el8.x86_64 1/25 2023-05-24T20:53:56.999 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-common-2:18.0.0-3443.g3395903b.el8.x86_64 2/25 2023-05-24T20:53:56.999 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-radosgw-2:18.0.0-3443.g3395903b.el8.x86_64 3/25 2023-05-24T20:53:56.999 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-selinux-2:18.0.0-3443.g3395903b.el8.x86_64 4/25 2023-05-24T20:53:56.999 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libcephfs2-2:18.0.0-3443.g3395903b.el8.x86_64 5/25 2023-05-24T20:53:57.000 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libradosstriper1-2:18.0.0-3443.g3395903b.el8.x86_6 6/25 2023-05-24T20:53:57.000 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librgw2-2:18.0.0-3443.g3395903b.el8.x86_64 7/25 2023-05-24T20:53:57.000 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-ceph-argparse-2:18.0.0-3443.g3395903b.el8. 8/25 2023-05-24T20:53:57.000 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-ceph-common-2:18.0.0-3443.g3395903b.el8.x8 9/25 2023-05-24T20:53:57.000 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cephfs-2:18.0.0-3443.g3395903b.el8.x86_64 10/25 2023-05-24T20:53:57.000 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rados-2:18.0.0-3443.g3395903b.el8.x86_64 11/25 2023-05-24T20:53:57.000 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rbd-2:18.0.0-3443.g3395903b.el8.x86_64 12/25 2023-05-24T20:53:57.001 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rgw-2:18.0.0-3443.g3395903b.el8.x86_64 13/25 2023-05-24T20:53:57.001 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-05-24T20:53:57.001 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-05-24T20:53:57.001 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-05-24T20:53:57.001 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 17/25 2023-05-24T20:53:57.001 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-05-24T20:53:57.001 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2023-05-24T20:53:57.002 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2023-05-24T20:53:57.002 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2023-05-24T20:53:57.002 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librados2-2:18.0.0-3443.g3395903b.el8.x86_64 22/25 2023-05-24T20:53:57.002 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-05-24T20:53:57.002 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librbd1-2:18.0.0-3443.g3395903b.el8.x86_64 24/25 2023-05-24T20:53:58.059 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-05-24T20:53:58.059 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:53:58.059 INFO:teuthology.orchestra.run.smithi157.stdout:Upgraded: 2023-05-24T20:53:58.060 INFO:teuthology.orchestra.run.smithi157.stdout: librados2-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:53:58.060 INFO:teuthology.orchestra.run.smithi157.stdout: librbd1-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:53:58.060 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-05-24T20:53:58.060 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-base-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:53:58.060 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-common-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:53:58.060 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-radosgw-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:53:58.060 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-selinux-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:53:58.060 INFO:teuthology.orchestra.run.smithi157.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-05-24T20:53:58.061 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs2-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:53:58.062 INFO:teuthology.orchestra.run.smithi157.stdout: liboath-2.6.2-3.el8.x86_64 2023-05-24T20:53:58.062 INFO:teuthology.orchestra.run.smithi157.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-05-24T20:53:58.062 INFO:teuthology.orchestra.run.smithi157.stdout: libradosstriper1-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:53:58.062 INFO:teuthology.orchestra.run.smithi157.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-05-24T20:53:58.062 INFO:teuthology.orchestra.run.smithi157.stdout: librgw2-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:53:58.062 INFO:teuthology.orchestra.run.smithi157.stdout: libunwind-1.3.1-3.el8.x86_64 2023-05-24T20:53:58.063 INFO:teuthology.orchestra.run.smithi157.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-05-24T20:53:58.063 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-argparse-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:53:58.063 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-common-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:53:58.063 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cephfs-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:53:58.063 INFO:teuthology.orchestra.run.smithi157.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-05-24T20:53:58.063 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rados-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:53:58.063 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rbd-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:53:58.063 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rgw-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:53:58.064 INFO:teuthology.orchestra.run.smithi157.stdout: thrift-0.13.0-2.el8.x86_64 2023-05-24T20:53:58.064 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:53:58.064 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-05-24T20:53:58.215 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-test 2023-05-24T20:53:58.966 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:00:41 ago on Wed 24 May 2023 08:53:17 PM UTC. 2023-05-24T20:54:00.093 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-05-24T20:54:00.094 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:54:00.094 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2023-05-24T20:54:00.094 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:54:00.094 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-05-24T20:54:00.095 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-test x86_64 2:18.0.0-3443.g3395903b.el8 ceph 47 M 2023-05-24T20:54:00.095 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2023-05-24T20:54:00.095 INFO:teuthology.orchestra.run.smithi157.stdout: jq x86_64 1.6-7.el8 appstream 202 k 2023-05-24T20:54:00.095 INFO:teuthology.orchestra.run.smithi157.stdout: libcephsqlite x86_64 2:18.0.0-3443.g3395903b.el8 ceph 137 k 2023-05-24T20:54:00.095 INFO:teuthology.orchestra.run.smithi157.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-05-24T20:54:00.095 INFO:teuthology.orchestra.run.smithi157.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-05-24T20:54:00.096 INFO:teuthology.orchestra.run.smithi157.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-05-24T20:54:00.096 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:54:00.096 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-05-24T20:54:00.096 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:54:00.096 INFO:teuthology.orchestra.run.smithi157.stdout:Install 6 Packages 2023-05-24T20:54:00.096 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:54:00.097 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 47 M 2023-05-24T20:54:00.097 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 210 M 2023-05-24T20:54:00.097 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-05-24T20:54:00.276 INFO:teuthology.orchestra.run.smithi157.stdout:(1/6): jq-1.6-7.el8.x86_64.rpm 1.5 MB/s | 202 kB 00:00 2023-05-24T20:54:00.343 INFO:teuthology.orchestra.run.smithi157.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 2.7 MB/s | 187 kB 00:00 2023-05-24T20:54:00.368 INFO:teuthology.orchestra.run.smithi157.stdout:(3/6): libcephsqlite-18.0.0-3443.g3395903b.el8. 608 kB/s | 137 kB 00:00 2023-05-24T20:54:00.419 INFO:teuthology.orchestra.run.smithi157.stdout:(4/6): socat-1.7.4.1-1.el8.x86_64.rpm 4.2 MB/s | 323 kB 00:00 2023-05-24T20:54:00.503 INFO:teuthology.orchestra.run.smithi157.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 526 kB/s | 71 kB 00:00 2023-05-24T20:54:03.095 INFO:teuthology.orchestra.run.smithi157.stdout:(6/6): ceph-test-18.0.0-3443.g3395903b.el8.x86_ 16 MB/s | 47 MB 00:02 2023-05-24T20:54:03.097 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-05-24T20:54:03.097 INFO:teuthology.orchestra.run.smithi157.stdout:Total 16 MB/s | 47 MB 00:03 2023-05-24T20:54:03.140 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-05-24T20:54:03.161 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-05-24T20:54:03.161 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-05-24T20:54:03.486 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-05-24T20:54:03.489 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-05-24T20:54:03.947 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-05-24T20:54:04.171 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-05-24T20:54:04.354 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-05-24T20:54:04.468 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-05-24T20:54:04.560 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-05-24T20:54:04.763 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : jq-1.6-7.el8.x86_64 4/6 2023-05-24T20:54:04.910 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libcephsqlite-2:18.0.0-3443.g3395903b.el8.x86_64 5/6 2023-05-24T20:54:09.384 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libcephsqlite-2:18.0.0-3443.g3395903b.el8.x86_64 5/6 2023-05-24T20:54:09.615 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-test-2:18.0.0-3443.g3395903b.el8.x86_64 6/6 2023-05-24T20:54:09.889 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-test-2:18.0.0-3443.g3395903b.el8.x86_64 6/6 2023-05-24T20:54:09.889 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-test-2:18.0.0-3443.g3395903b.el8.x86_64 1/6 2023-05-24T20:54:09.889 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libcephsqlite-2:18.0.0-3443.g3395903b.el8.x86_64 2/6 2023-05-24T20:54:09.890 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : jq-1.6-7.el8.x86_64 3/6 2023-05-24T20:54:09.890 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-05-24T20:54:09.890 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-05-24T20:54:10.561 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-05-24T20:54:10.561 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:54:10.561 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-05-24T20:54:10.561 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-test-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:54:10.562 INFO:teuthology.orchestra.run.smithi157.stdout: jq-1.6-7.el8.x86_64 2023-05-24T20:54:10.562 INFO:teuthology.orchestra.run.smithi157.stdout: libcephsqlite-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:54:10.562 INFO:teuthology.orchestra.run.smithi157.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-05-24T20:54:10.562 INFO:teuthology.orchestra.run.smithi157.stdout: socat-1.7.4.1-1.el8.x86_64 2023-05-24T20:54:10.562 INFO:teuthology.orchestra.run.smithi157.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-05-24T20:54:10.562 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:54:10.562 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-05-24T20:54:10.804 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph 2023-05-24T20:54:11.533 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:00:54 ago on Wed 24 May 2023 08:53:17 PM UTC. 2023-05-24T20:54:12.678 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-05-24T20:54:12.681 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:54:12.681 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2023-05-24T20:54:12.681 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:54:12.682 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-05-24T20:54:12.682 INFO:teuthology.orchestra.run.smithi157.stdout: ceph x86_64 2:18.0.0-3443.g3395903b.el8 ceph 6.4 k 2023-05-24T20:54:12.682 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2023-05-24T20:54:12.682 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mds x86_64 2:18.0.0-3443.g3395903b.el8 ceph 2.3 M 2023-05-24T20:54:12.682 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr x86_64 2:18.0.0-3443.g3395903b.el8 ceph 1.5 M 2023-05-24T20:54:12.682 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-modules-core noarch 2:18.0.0-3443.g3395903b.el8 ceph-noarch 240 k 2023-05-24T20:54:12.683 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mon x86_64 2:18.0.0-3443.g3395903b.el8 ceph 4.6 M 2023-05-24T20:54:12.683 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-osd x86_64 2:18.0.0-3443.g3395903b.el8 ceph 18 M 2023-05-24T20:54:12.683 INFO:teuthology.orchestra.run.smithi157.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-05-24T20:54:12.683 INFO:teuthology.orchestra.run.smithi157.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-05-24T20:54:12.683 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cffi x86_64 1.11.5-5.el8 baseos 237 k 2023-05-24T20:54:12.684 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-05-24T20:54:12.684 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-05-24T20:54:12.684 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cryptography x86_64 3.2.1-5.el8 baseos 559 k 2023-05-24T20:54:12.684 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-05-24T20:54:12.684 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-05-24T20:54:12.684 INFO:teuthology.orchestra.run.smithi157.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-05-24T20:54:12.684 INFO:teuthology.orchestra.run.smithi157.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-05-24T20:54:12.685 INFO:teuthology.orchestra.run.smithi157.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-05-24T20:54:12.685 INFO:teuthology.orchestra.run.smithi157.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-05-24T20:54:12.685 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-05-24T20:54:12.685 INFO:teuthology.orchestra.run.smithi157.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-05-24T20:54:12.685 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-05-24T20:54:12.685 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-05-24T20:54:12.685 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pytz noarch 2017.2-9.el8 appstream 54 k 2023-05-24T20:54:12.686 INFO:teuthology.orchestra.run.smithi157.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-05-24T20:54:12.686 INFO:teuthology.orchestra.run.smithi157.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-05-24T20:54:12.686 INFO:teuthology.orchestra.run.smithi157.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-05-24T20:54:12.686 INFO:teuthology.orchestra.run.smithi157.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-05-24T20:54:12.686 INFO:teuthology.orchestra.run.smithi157.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-05-24T20:54:12.686 INFO:teuthology.orchestra.run.smithi157.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-05-24T20:54:12.686 INFO:teuthology.orchestra.run.smithi157.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-05-24T20:54:12.686 INFO:teuthology.orchestra.run.smithi157.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-05-24T20:54:12.687 INFO:teuthology.orchestra.run.smithi157.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-05-24T20:54:12.687 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:54:12.687 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-05-24T20:54:12.687 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:54:12.687 INFO:teuthology.orchestra.run.smithi157.stdout:Install 32 Packages 2023-05-24T20:54:12.687 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:54:12.688 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 30 M 2023-05-24T20:54:12.688 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 119 M 2023-05-24T20:54:12.688 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-05-24T20:54:13.265 INFO:teuthology.orchestra.run.smithi157.stdout:(1/32): ceph-18.0.0-3443.g3395903b.el8.x86_64.r 20 kB/s | 6.4 kB 00:00 2023-05-24T20:54:14.157 INFO:teuthology.orchestra.run.smithi157.stdout:(2/32): ceph-mgr-18.0.0-3443.g3395903b.el8.x86_ 1.3 MB/s | 1.5 MB 00:01 2023-05-24T20:54:14.407 INFO:teuthology.orchestra.run.smithi157.stdout:(3/32): ceph-mds-18.0.0-3443.g3395903b.el8.x86_ 1.6 MB/s | 2.3 MB 00:01 2023-05-24T20:54:14.566 INFO:teuthology.orchestra.run.smithi157.stdout:(4/32): ceph-mgr-modules-core-18.0.0-3443.g3395 1.5 MB/s | 240 kB 00:00 2023-05-24T20:54:14.682 INFO:teuthology.orchestra.run.smithi157.stdout:(5/32): python3-mako-1.0.6-14.el8.noarch.rpm 1.3 MB/s | 157 kB 00:00 2023-05-24T20:54:14.724 INFO:teuthology.orchestra.run.smithi157.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 932 kB/s | 39 kB 00:00 2023-05-24T20:54:14.767 INFO:teuthology.orchestra.run.smithi157.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 2.4 MB/s | 103 kB 00:00 2023-05-24T20:54:14.808 INFO:teuthology.orchestra.run.smithi157.stdout:(8/32): python3-pytz-2017.2-9.el8.noarch.rpm 1.3 MB/s | 54 kB 00:00 2023-05-24T20:54:14.901 INFO:teuthology.orchestra.run.smithi157.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 4.9 MB/s | 457 kB 00:00 2023-05-24T20:54:15.342 INFO:teuthology.orchestra.run.smithi157.stdout:(10/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 538 kB/s | 237 kB 00:00 2023-05-24T20:54:15.530 INFO:teuthology.orchestra.run.smithi157.stdout:(11/32): python3-cryptography-3.2.1-5.el8.x86_6 2.9 MB/s | 559 kB 00:00 2023-05-24T20:54:15.626 INFO:teuthology.orchestra.run.smithi157.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 1.1 MB/s | 109 kB 00:00 2023-05-24T20:54:15.693 INFO:teuthology.orchestra.run.smithi157.stdout:(13/32): ceph-mon-18.0.0-3443.g3395903b.el8.x86 1.9 MB/s | 4.6 MB 00:02 2023-05-24T20:54:15.785 INFO:teuthology.orchestra.run.smithi157.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 275 kB/s | 44 kB 00:00 2023-05-24T20:54:15.810 INFO:teuthology.orchestra.run.smithi157.stdout:(15/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.5 MB/s | 185 kB 00:00 2023-05-24T20:54:15.852 INFO:teuthology.orchestra.run.smithi157.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 2.5 MB/s | 173 kB 00:00 2023-05-24T20:54:15.886 INFO:teuthology.orchestra.run.smithi157.stdout:(17/32): python3-cherrypy-18.4.0-1.el8.noarch.r 5.0 MB/s | 384 kB 00:00 2023-05-24T20:54:15.903 INFO:teuthology.orchestra.run.smithi157.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 221 kB/s | 11 kB 00:00 2023-05-24T20:54:15.920 INFO:teuthology.orchestra.run.smithi157.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 527 kB/s | 18 kB 00:00 2023-05-24T20:54:15.937 INFO:teuthology.orchestra.run.smithi157.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 1.4 MB/s | 49 kB 00:00 2023-05-24T20:54:15.954 INFO:teuthology.orchestra.run.smithi157.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 1.7 MB/s | 59 kB 00:00 2023-05-24T20:54:15.987 INFO:teuthology.orchestra.run.smithi157.stdout:(22/32): python3-pecan-1.3.2-9.el8.noarch.rpm 5.5 MB/s | 283 kB 00:00 2023-05-24T20:54:16.005 INFO:teuthology.orchestra.run.smithi157.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 309 kB/s | 16 kB 00:00 2023-05-24T20:54:16.021 INFO:teuthology.orchestra.run.smithi157.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 559 kB/s | 19 kB 00:00 2023-05-24T20:54:16.039 INFO:teuthology.orchestra.run.smithi157.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 713 kB/s | 24 kB 00:00 2023-05-24T20:54:16.064 INFO:teuthology.orchestra.run.smithi157.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 695 kB/s | 29 kB 00:00 2023-05-24T20:54:16.089 INFO:teuthology.orchestra.run.smithi157.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 548 kB/s | 27 kB 00:00 2023-05-24T20:54:16.123 INFO:teuthology.orchestra.run.smithi157.stdout:(28/32): python3-waitress-1.4.3-1.el8.noarch.rp 4.3 MB/s | 255 kB 00:00 2023-05-24T20:54:16.166 INFO:teuthology.orchestra.run.smithi157.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 3.2 MB/s | 251 kB 00:00 2023-05-24T20:54:16.198 INFO:teuthology.orchestra.run.smithi157.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.1 MB/s | 85 kB 00:00 2023-05-24T20:54:16.223 INFO:teuthology.orchestra.run.smithi157.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 405 kB/s | 23 kB 00:00 2023-05-24T20:54:20.423 INFO:teuthology.orchestra.run.smithi157.stdout:(32/32): ceph-osd-18.0.0-3443.g3395903b.el8.x86 2.8 MB/s | 18 MB 00:06 2023-05-24T20:54:20.425 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-05-24T20:54:20.426 INFO:teuthology.orchestra.run.smithi157.stdout:Total 3.9 MB/s | 30 MB 00:07 2023-05-24T20:54:20.508 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-05-24T20:54:20.545 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-05-24T20:54:20.546 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-05-24T20:54:20.846 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-05-24T20:54:20.849 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-05-24T20:54:21.929 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-05-24T20:54:22.171 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-05-24T20:54:22.390 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-05-24T20:54:22.651 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-05-24T20:54:22.854 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-05-24T20:54:23.035 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-05-24T20:54:23.224 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-05-24T20:54:23.359 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-05-24T20:54:23.545 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-05-24T20:54:23.787 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-05-24T20:54:23.999 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-05-24T20:54:24.228 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-05-24T20:54:24.444 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-05-24T20:54:24.830 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 13/32 2023-05-24T20:54:25.068 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 14/32 2023-05-24T20:54:25.264 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-05-24T20:54:25.482 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-05-24T20:54:25.682 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-05-24T20:54:26.000 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-05-24T20:54:26.203 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-05-24T20:54:26.401 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 20/32 2023-05-24T20:54:26.586 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-05-24T20:54:26.821 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-05-24T20:54:27.001 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-05-24T20:54:27.234 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-05-24T20:54:27.567 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-05-24T20:54:27.913 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-05-24T20:54:28.252 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-3443.g3395903b.el8. 27/32 2023-05-24T20:54:28.449 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-2:18.0.0-3443.g3395903b.el8.x86_64 28/32 2023-05-24T20:54:30.023 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-2:18.0.0-3443.g3395903b.el8.x86_64 28/32 2023-05-24T20:54:30.309 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-osd-2:18.0.0-3443.g3395903b.el8.x86_64 29/32 2023-05-24T20:54:30.767 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-osd-2:18.0.0-3443.g3395903b.el8.x86_64 29/32 2023-05-24T20:54:30.955 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mon-2:18.0.0-3443.g3395903b.el8.x86_64 30/32 2023-05-24T20:54:31.177 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mon-2:18.0.0-3443.g3395903b.el8.x86_64 30/32 2023-05-24T20:54:31.322 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mds-2:18.0.0-3443.g3395903b.el8.x86_64 31/32 2023-05-24T20:54:31.340 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mds-2:18.0.0-3443.g3395903b.el8.x86_64 31/32 2023-05-24T20:54:31.476 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-2:18.0.0-3443.g3395903b.el8.x86_64 32/32 2023-05-24T20:54:31.881 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-2:18.0.0-3443.g3395903b.el8.x86_64 32/32 2023-05-24T20:54:31.881 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-2:18.0.0-3443.g3395903b.el8.x86_64 1/32 2023-05-24T20:54:31.882 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mds-2:18.0.0-3443.g3395903b.el8.x86_64 2/32 2023-05-24T20:54:31.882 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-2:18.0.0-3443.g3395903b.el8.x86_64 3/32 2023-05-24T20:54:31.882 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mon-2:18.0.0-3443.g3395903b.el8.x86_64 4/32 2023-05-24T20:54:31.882 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-osd-2:18.0.0-3443.g3395903b.el8.x86_64 5/32 2023-05-24T20:54:31.882 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-3443.g3395903b.el8. 6/32 2023-05-24T20:54:31.882 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-05-24T20:54:31.883 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-05-24T20:54:31.883 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-05-24T20:54:31.883 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 10/32 2023-05-24T20:54:31.883 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-05-24T20:54:31.883 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 12/32 2023-05-24T20:54:31.883 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 13/32 2023-05-24T20:54:31.883 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-05-24T20:54:31.884 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-05-24T20:54:31.884 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-05-24T20:54:31.884 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-05-24T20:54:31.884 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-05-24T20:54:31.884 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-05-24T20:54:31.884 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-05-24T20:54:31.884 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-05-24T20:54:31.884 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-05-24T20:54:31.885 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-05-24T20:54:31.885 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-05-24T20:54:31.885 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-05-24T20:54:31.885 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-05-24T20:54:31.885 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-05-24T20:54:31.885 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-05-24T20:54:31.885 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-05-24T20:54:31.886 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-05-24T20:54:31.886 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-05-24T20:54:32.919 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-05-24T20:54:32.919 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:54:32.919 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-05-24T20:54:32.919 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:54:32.920 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mds-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:54:32.920 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:54:32.920 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-modules-core-2:18.0.0-3443.g3395903b.el8.noarch 2023-05-24T20:54:32.920 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mon-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:54:32.920 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-osd-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:54:32.920 INFO:teuthology.orchestra.run.smithi157.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-05-24T20:54:32.921 INFO:teuthology.orchestra.run.smithi157.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-05-24T20:54:32.921 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-05-24T20:54:32.921 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-05-24T20:54:32.921 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-05-24T20:54:32.921 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-05-24T20:54:32.921 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-6.2-6.el8.noarch 2023-05-24T20:54:32.921 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-05-24T20:54:32.922 INFO:teuthology.orchestra.run.smithi157.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-05-24T20:54:32.922 INFO:teuthology.orchestra.run.smithi157.stdout: python3-mako-1.0.6-14.el8.noarch 2023-05-24T20:54:32.922 INFO:teuthology.orchestra.run.smithi157.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-05-24T20:54:32.922 INFO:teuthology.orchestra.run.smithi157.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-05-24T20:54:32.922 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-05-24T20:54:32.922 INFO:teuthology.orchestra.run.smithi157.stdout: python3-portend-2.6-1.el8.noarch 2023-05-24T20:54:32.922 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-05-24T20:54:32.922 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pycparser-2.14-14.el8.noarch 2023-05-24T20:54:32.923 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pytz-2017.2-9.el8.noarch 2023-05-24T20:54:32.923 INFO:teuthology.orchestra.run.smithi157.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-05-24T20:54:32.923 INFO:teuthology.orchestra.run.smithi157.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-05-24T20:54:32.923 INFO:teuthology.orchestra.run.smithi157.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-05-24T20:54:32.923 INFO:teuthology.orchestra.run.smithi157.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-05-24T20:54:32.923 INFO:teuthology.orchestra.run.smithi157.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-05-24T20:54:32.923 INFO:teuthology.orchestra.run.smithi157.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-05-24T20:54:32.924 INFO:teuthology.orchestra.run.smithi157.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-05-24T20:54:32.924 INFO:teuthology.orchestra.run.smithi157.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-05-24T20:54:32.924 INFO:teuthology.orchestra.run.smithi157.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-05-24T20:54:32.924 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:54:32.924 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-05-24T20:54:33.167 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-base 2023-05-24T20:54:33.886 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:16 ago on Wed 24 May 2023 08:53:17 PM UTC. 2023-05-24T20:54:34.864 INFO:teuthology.orchestra.run.smithi157.stdout:Package ceph-base-2:18.0.0-3443.g3395903b.el8.x86_64 is already installed. 2023-05-24T20:54:35.002 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-05-24T20:54:35.002 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2023-05-24T20:54:35.003 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-05-24T20:54:35.097 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install cephadm 2023-05-24T20:54:35.815 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:18 ago on Wed 24 May 2023 08:53:17 PM UTC. 2023-05-24T20:54:36.915 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-05-24T20:54:36.916 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:54:36.916 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2023-05-24T20:54:36.917 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:54:36.917 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-05-24T20:54:36.917 INFO:teuthology.orchestra.run.smithi157.stdout: cephadm noarch 2:18.0.0-3443.g3395903b.el8 ceph-noarch 160 k 2023-05-24T20:54:36.917 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:54:36.917 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-05-24T20:54:36.918 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:54:36.918 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2023-05-24T20:54:36.918 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:54:36.918 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 160 k 2023-05-24T20:54:36.918 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 639 k 2023-05-24T20:54:36.918 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-05-24T20:54:37.119 INFO:teuthology.orchestra.run.smithi157.stdout:cephadm-18.0.0-3443.g3395903b.el8.noarch.rpm 792 kB/s | 160 kB 00:00 2023-05-24T20:54:37.120 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-05-24T20:54:37.120 INFO:teuthology.orchestra.run.smithi157.stdout:Total 787 kB/s | 160 kB 00:00 2023-05-24T20:54:37.121 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-05-24T20:54:37.128 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-05-24T20:54:37.128 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-05-24T20:54:37.130 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-05-24T20:54:37.133 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-05-24T20:54:37.176 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-05-24T20:54:37.702 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: cephadm-2:18.0.0-3443.g3395903b.el8.noarch 1/1 2023-05-24T20:54:37.859 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : cephadm-2:18.0.0-3443.g3395903b.el8.noarch 1/1 2023-05-24T20:54:38.088 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: cephadm-2:18.0.0-3443.g3395903b.el8.noarch 1/1 2023-05-24T20:54:38.619 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : cephadm-2:18.0.0-3443.g3395903b.el8.noarch 1/1 2023-05-24T20:54:38.619 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:54:38.619 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-05-24T20:54:38.620 INFO:teuthology.orchestra.run.smithi157.stdout: cephadm-2:18.0.0-3443.g3395903b.el8.noarch 2023-05-24T20:54:38.620 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:54:38.620 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-05-24T20:54:38.807 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-immutable-object-cache 2023-05-24T20:54:39.532 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:22 ago on Wed 24 May 2023 08:53:17 PM UTC. 2023-05-24T20:54:40.662 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-05-24T20:54:40.664 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:54:40.664 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2023-05-24T20:54:40.664 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:54:40.664 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-05-24T20:54:40.664 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-3443.g3395903b.el8 ceph 166 k 2023-05-24T20:54:40.665 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:54:40.665 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-05-24T20:54:40.665 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:54:40.665 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2023-05-24T20:54:40.665 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:54:40.665 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 166 k 2023-05-24T20:54:40.665 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 503 k 2023-05-24T20:54:40.666 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-05-24T20:54:41.690 INFO:teuthology.orchestra.run.smithi157.stdout:ceph-immutable-object-cache-18.0.0-3443.g339590 162 kB/s | 166 kB 00:01 2023-05-24T20:54:41.691 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-05-24T20:54:41.691 INFO:teuthology.orchestra.run.smithi157.stdout:Total 162 kB/s | 166 kB 00:01 2023-05-24T20:54:41.692 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-05-24T20:54:41.704 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-05-24T20:54:41.704 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-05-24T20:54:41.756 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-05-24T20:54:41.760 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-05-24T20:54:41.909 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-05-24T20:54:42.087 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-3443.g3395903b. 1/1 2023-05-24T20:54:42.507 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-3443.g3395903b. 1/1 2023-05-24T20:54:43.051 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-3443.g3395903b. 1/1 2023-05-24T20:54:43.051 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:54:43.052 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-05-24T20:54:43.052 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-immutable-object-cache-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:54:43.052 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:54:43.052 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-05-24T20:54:43.257 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mgr 2023-05-24T20:54:43.984 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:26 ago on Wed 24 May 2023 08:53:17 PM UTC. 2023-05-24T20:54:44.967 INFO:teuthology.orchestra.run.smithi157.stdout:Package ceph-mgr-2:18.0.0-3443.g3395903b.el8.x86_64 is already installed. 2023-05-24T20:54:45.105 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-05-24T20:54:45.106 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2023-05-24T20:54:45.106 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-05-24T20:54:45.199 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mgr-dashboard 2023-05-24T20:54:45.908 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:28 ago on Wed 24 May 2023 08:53:17 PM UTC. 2023-05-24T20:54:47.019 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-05-24T20:54:47.020 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:54:47.020 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2023-05-24T20:54:47.020 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:54:47.021 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-05-24T20:54:47.021 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-dashboard noarch 2:18.0.0-3443.g3395903b.el8 ceph-noarch 1.7 M 2023-05-24T20:54:47.021 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2023-05-24T20:54:47.021 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-grafana-dashboards noarch 2:18.0.0-3443.g3395903b.el8 ceph-noarch 23 k 2023-05-24T20:54:47.021 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-prometheus-alerts noarch 2:18.0.0-3443.g3395903b.el8 ceph-noarch 15 k 2023-05-24T20:54:47.021 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-05-24T20:54:47.021 INFO:teuthology.orchestra.run.smithi157.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-05-24T20:54:47.021 INFO:teuthology.orchestra.run.smithi157.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-05-24T20:54:47.022 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:54:47.022 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-05-24T20:54:47.022 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:54:47.022 INFO:teuthology.orchestra.run.smithi157.stdout:Install 6 Packages 2023-05-24T20:54:47.022 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:54:47.023 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 2.0 M 2023-05-24T20:54:47.023 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 8.0 M 2023-05-24T20:54:47.023 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-05-24T20:54:47.391 INFO:teuthology.orchestra.run.smithi157.stdout:(1/6): ceph-prometheus-alerts-18.0.0-3443.g3395 99 kB/s | 15 kB 00:00 2023-05-24T20:54:47.416 INFO:teuthology.orchestra.run.smithi157.stdout:(2/6): ceph-grafana-dashboards-18.0.0-3443.g339 134 kB/s | 23 kB 00:00 2023-05-24T20:54:47.517 INFO:teuthology.orchestra.run.smithi157.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 336 kB/s | 34 kB 00:00 2023-05-24T20:54:47.559 INFO:teuthology.orchestra.run.smithi157.stdout:(4/6): ceph-mgr-dashboard-18.0.0-3443.g3395903b 5.4 MB/s | 1.7 MB 00:00 2023-05-24T20:54:47.592 INFO:teuthology.orchestra.run.smithi157.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 2.6 MB/s | 196 kB 00:00 2023-05-24T20:54:47.709 INFO:teuthology.orchestra.run.smithi157.stdout:(6/6): python3-jwt-1.6.1-2.el8.noarch.rpm 134 kB/s | 43 kB 00:00 2023-05-24T20:54:47.711 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-05-24T20:54:47.712 INFO:teuthology.orchestra.run.smithi157.stdout:Total 2.9 MB/s | 2.0 MB 00:00 2023-05-24T20:54:47.722 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-05-24T20:54:47.730 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-05-24T20:54:47.731 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-05-24T20:54:47.820 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-05-24T20:54:47.823 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-05-24T20:54:48.089 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-05-24T20:54:48.275 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-05-24T20:54:48.520 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-05-24T20:54:48.671 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-05-24T20:54:48.791 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-3443.g3395903b.el8.n 4/6 2023-05-24T20:54:49.116 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-3443.g3395903b.el8. 5/6 2023-05-24T20:54:49.319 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-3443.g3395903b.el8.noarc 6/6 2023-05-24T20:54:49.606 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-3443.g3395903b.el8.noarc 6/6 2023-05-24T20:54:49.607 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-3443.g3395903b.el8. 1/6 2023-05-24T20:54:49.607 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-3443.g3395903b.el8.noarc 2/6 2023-05-24T20:54:49.607 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-3443.g3395903b.el8.n 3/6 2023-05-24T20:54:49.607 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-05-24T20:54:49.608 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-05-24T20:54:50.150 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-05-24T20:54:50.150 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:54:50.151 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-05-24T20:54:50.151 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-grafana-dashboards-2:18.0.0-3443.g3395903b.el8.noarch 2023-05-24T20:54:50.151 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-dashboard-2:18.0.0-3443.g3395903b.el8.noarch 2023-05-24T20:54:50.151 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-prometheus-alerts-2:18.0.0-3443.g3395903b.el8.noarch 2023-05-24T20:54:50.151 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-05-24T20:54:50.151 INFO:teuthology.orchestra.run.smithi157.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-05-24T20:54:50.151 INFO:teuthology.orchestra.run.smithi157.stdout: python3-routes-2.4.1-12.el8.noarch 2023-05-24T20:54:50.152 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:54:50.152 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-05-24T20:54:50.251 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mgr-diskprediction-local 2023-05-24T20:54:50.975 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:33 ago on Wed 24 May 2023 08:53:17 PM UTC. 2023-05-24T20:54:52.088 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-05-24T20:54:52.089 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================================ 2023-05-24T20:54:52.089 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2023-05-24T20:54:52.089 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================================ 2023-05-24T20:54:52.089 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-05-24T20:54:52.090 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-3443.g3395903b.el8 ceph-noarch 7.4 M 2023-05-24T20:54:52.090 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2023-05-24T20:54:52.090 INFO:teuthology.orchestra.run.smithi157.stdout: libgfortran x86_64 8.5.0-19.el8 baseos 645 k 2023-05-24T20:54:52.090 INFO:teuthology.orchestra.run.smithi157.stdout: libquadmath x86_64 8.5.0-19.el8 baseos 172 k 2023-05-24T20:54:52.090 INFO:teuthology.orchestra.run.smithi157.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-05-24T20:54:52.090 INFO:teuthology.orchestra.run.smithi157.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-05-24T20:54:52.090 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-05-24T20:54:52.091 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-05-24T20:54:52.091 INFO:teuthology.orchestra.run.smithi157.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-05-24T20:54:52.091 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:54:52.091 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-05-24T20:54:52.091 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================================ 2023-05-24T20:54:52.091 INFO:teuthology.orchestra.run.smithi157.stdout:Install 8 Packages 2023-05-24T20:54:52.091 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:54:52.092 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 35 M 2023-05-24T20:54:52.092 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 210 M 2023-05-24T20:54:52.092 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-05-24T20:54:52.774 INFO:teuthology.orchestra.run.smithi157.stdout:(1/8): openblas-0.3.15-4.el8.x86_64.rpm 15 MB/s | 4.7 MB 00:00 2023-05-24T20:54:52.849 INFO:teuthology.orchestra.run.smithi157.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 13 MB/s | 4.9 MB 00:00 2023-05-24T20:54:52.941 INFO:teuthology.orchestra.run.smithi157.stdout:(3/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 22 MB/s | 3.7 MB 00:00 2023-05-24T20:54:52.966 INFO:teuthology.orchestra.run.smithi157.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.9 MB/s | 225 kB 00:00 2023-05-24T20:54:53.049 INFO:teuthology.orchestra.run.smithi157.stdout:(5/8): ceph-mgr-diskprediction-local-18.0.0-344 13 MB/s | 7.4 MB 00:00 2023-05-24T20:54:53.291 INFO:teuthology.orchestra.run.smithi157.stdout:(6/8): python3-scipy-1.0.0-21.module_el8.5.0+77 39 MB/s | 14 MB 00:00 2023-05-24T20:54:53.417 INFO:teuthology.orchestra.run.smithi157.stdout:(7/8): libquadmath-8.5.0-19.el8.x86_64.rpm 469 kB/s | 172 kB 00:00 2023-05-24T20:54:53.484 INFO:teuthology.orchestra.run.smithi157.stdout:(8/8): libgfortran-8.5.0-19.el8.x86_64.rpm 1.2 MB/s | 645 kB 00:00 2023-05-24T20:54:53.484 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-05-24T20:54:53.485 INFO:teuthology.orchestra.run.smithi157.stdout:Total 25 MB/s | 35 MB 00:01 2023-05-24T20:54:53.715 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-05-24T20:54:53.733 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-05-24T20:54:53.733 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-05-24T20:54:54.129 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-05-24T20:54:54.132 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-05-24T20:54:54.711 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-05-24T20:54:54.893 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libquadmath-8.5.0-19.el8.x86_64 1/8 2023-05-24T20:54:55.036 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libquadmath-8.5.0-19.el8.x86_64 1/8 2023-05-24T20:54:55.184 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libgfortran-8.5.0-19.el8.x86_64 2/8 2023-05-24T20:54:55.782 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libgfortran-8.5.0-19.el8.x86_64 2/8 2023-05-24T20:54:55.951 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-05-24T20:54:56.561 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-05-24T20:54:56.709 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-05-24T20:54:57.427 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-05-24T20:54:58.030 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-05-24T20:54:58.217 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-05-24T20:55:00.008 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-05-24T20:55:01.206 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-05-24T20:55:01.425 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-3443.g3395903 8/8 2023-05-24T20:55:01.749 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-3443.g3395903 8/8 2023-05-24T20:55:01.749 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-3443.g3395903 1/8 2023-05-24T20:55:01.750 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-05-24T20:55:01.750 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-05-24T20:55:01.750 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-05-24T20:55:01.750 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-05-24T20:55:01.750 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-05-24T20:55:01.750 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libgfortran-8.5.0-19.el8.x86_64 7/8 2023-05-24T20:55:02.315 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libquadmath-8.5.0-19.el8.x86_64 8/8 2023-05-24T20:55:02.315 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:55:02.315 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-05-24T20:55:02.315 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-diskprediction-local-2:18.0.0-3443.g3395903b.el8.noarch 2023-05-24T20:55:02.315 INFO:teuthology.orchestra.run.smithi157.stdout: libgfortran-8.5.0-19.el8.x86_64 2023-05-24T20:55:02.315 INFO:teuthology.orchestra.run.smithi157.stdout: libquadmath-8.5.0-19.el8.x86_64 2023-05-24T20:55:02.316 INFO:teuthology.orchestra.run.smithi157.stdout: openblas-0.3.15-4.el8.x86_64 2023-05-24T20:55:02.316 INFO:teuthology.orchestra.run.smithi157.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-05-24T20:55:02.316 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-05-24T20:55:02.316 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-05-24T20:55:02.316 INFO:teuthology.orchestra.run.smithi157.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-05-24T20:55:02.316 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:55:02.317 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-05-24T20:55:02.425 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mgr-rook 2023-05-24T20:55:03.148 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:46 ago on Wed 24 May 2023 08:53:17 PM UTC. 2023-05-24T20:55:04.296 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-05-24T20:55:04.298 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:55:04.298 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2023-05-24T20:55:04.298 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:55:04.298 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-05-24T20:55:04.298 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-rook noarch 2:18.0.0-3443.g3395903b.el8 ceph-noarch 47 k 2023-05-24T20:55:04.299 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2023-05-24T20:55:04.299 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-05-24T20:55:04.299 INFO:teuthology.orchestra.run.smithi157.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-05-24T20:55:04.299 INFO:teuthology.orchestra.run.smithi157.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-05-24T20:55:04.299 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-05-24T20:55:04.299 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-05-24T20:55:04.299 INFO:teuthology.orchestra.run.smithi157.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-05-24T20:55:04.299 INFO:teuthology.orchestra.run.smithi157.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-05-24T20:55:04.300 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-05-24T20:55:04.300 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-05-24T20:55:04.300 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-05-24T20:55:04.300 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-05-24T20:55:04.300 INFO:teuthology.orchestra.run.smithi157.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-05-24T20:55:04.300 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:55:04.300 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-05-24T20:55:04.301 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:55:04.301 INFO:teuthology.orchestra.run.smithi157.stdout:Install 13 Packages 2023-05-24T20:55:04.301 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:55:04.301 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 1.9 M 2023-05-24T20:55:04.302 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 25 M 2023-05-24T20:55:04.302 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-05-24T20:55:07.624 INFO:teuthology.orchestra.run.smithi157.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 320 kB/s | 27 kB 00:00 2023-05-24T20:55:07.649 INFO:teuthology.orchestra.run.smithi157.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 184 kB/s | 20 kB 00:00 2023-05-24T20:55:07.716 INFO:teuthology.orchestra.run.smithi157.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.4 MB/s | 126 kB 00:00 2023-05-24T20:55:07.767 INFO:teuthology.orchestra.run.smithi157.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 941 kB/s | 110 kB 00:00 2023-05-24T20:55:08.000 INFO:teuthology.orchestra.run.smithi157.stdout:(5/13): ceph-mgr-rook-18.0.0-3443.g3395903b.el8 103 kB/s | 47 kB 00:00 2023-05-24T20:55:08.051 INFO:teuthology.orchestra.run.smithi157.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 150 kB/s | 43 kB 00:00 2023-05-24T20:55:08.126 INFO:teuthology.orchestra.run.smithi157.stdout:(7/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 379 kB/s | 155 kB 00:00 2023-05-24T20:55:08.143 INFO:teuthology.orchestra.run.smithi157.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 177 kB/s | 16 kB 00:00 2023-05-24T20:55:08.160 INFO:teuthology.orchestra.run.smithi157.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 207 kB/s | 33 kB 00:00 2023-05-24T20:55:08.210 INFO:teuthology.orchestra.run.smithi157.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 970 kB/s | 81 kB 00:00 2023-05-24T20:55:08.260 INFO:teuthology.orchestra.run.smithi157.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 1.2 MB/s | 61 kB 00:00 2023-05-24T20:55:08.277 INFO:teuthology.orchestra.run.smithi157.stdout:(12/13): python3-rsa-4.8-1.el8.noarch.rpm 526 kB/s | 61 kB 00:00 2023-05-24T20:55:08.344 INFO:teuthology.orchestra.run.smithi157.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 5.8 MB/s | 1.2 MB 00:00 2023-05-24T20:55:08.344 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-05-24T20:55:08.345 INFO:teuthology.orchestra.run.smithi157.stdout:Total 488 kB/s | 1.9 MB 00:04 2023-05-24T20:55:08.405 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-05-24T20:55:08.421 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-05-24T20:55:08.421 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-05-24T20:55:08.507 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-05-24T20:55:08.510 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-05-24T20:55:08.994 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-05-24T20:55:09.217 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-05-24T20:55:09.399 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-05-24T20:55:09.643 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rsa-4.8-1.el8.noarch 3/13 2023-05-24T20:55:09.888 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-05-24T20:55:10.071 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-05-24T20:55:10.294 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-05-24T20:55:10.572 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-05-24T20:55:10.770 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-05-24T20:55:11.820 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-05-24T20:55:12.071 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-05-24T20:55:12.297 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-05-24T20:55:12.474 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-05-24T20:55:12.639 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-rook-2:18.0.0-3443.g3395903b.el8.noarch 13/13 2023-05-24T20:55:12.942 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-3443.g3395903b.el8.noarch 13/13 2023-05-24T20:55:12.943 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-rook-2:18.0.0-3443.g3395903b.el8.noarch 1/13 2023-05-24T20:55:12.943 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-05-24T20:55:12.943 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-05-24T20:55:12.944 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-05-24T20:55:12.944 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-05-24T20:55:12.944 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-05-24T20:55:12.944 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-05-24T20:55:12.945 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-05-24T20:55:12.945 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-05-24T20:55:12.945 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-05-24T20:55:12.945 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-05-24T20:55:12.946 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 12/13 2023-05-24T20:55:13.553 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-05-24T20:55:13.553 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:55:13.554 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-05-24T20:55:13.554 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-rook-2:18.0.0-3443.g3395903b.el8.noarch 2023-05-24T20:55:13.554 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-05-24T20:55:13.554 INFO:teuthology.orchestra.run.smithi157.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-05-24T20:55:13.554 INFO:teuthology.orchestra.run.smithi157.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-05-24T20:55:13.554 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-05-24T20:55:13.555 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-05-24T20:55:13.555 INFO:teuthology.orchestra.run.smithi157.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-05-24T20:55:13.555 INFO:teuthology.orchestra.run.smithi157.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-05-24T20:55:13.555 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-05-24T20:55:13.555 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-05-24T20:55:13.555 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-05-24T20:55:13.555 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rsa-4.8-1.el8.noarch 2023-05-24T20:55:13.556 INFO:teuthology.orchestra.run.smithi157.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-05-24T20:55:13.556 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:55:13.556 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-05-24T20:55:13.670 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mgr-cephadm 2023-05-24T20:55:14.390 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:57 ago on Wed 24 May 2023 08:53:17 PM UTC. 2023-05-24T20:55:15.510 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-05-24T20:55:15.511 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:55:15.511 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2023-05-24T20:55:15.511 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:55:15.511 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-05-24T20:55:15.512 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-cephadm noarch 2:18.0.0-3443.g3395903b.el8 ceph-noarch 127 k 2023-05-24T20:55:15.512 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2023-05-24T20:55:15.512 INFO:teuthology.orchestra.run.smithi157.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-05-24T20:55:15.512 INFO:teuthology.orchestra.run.smithi157.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-05-24T20:55:15.512 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-05-24T20:55:15.512 INFO:teuthology.orchestra.run.smithi157.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-05-24T20:55:15.512 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:55:15.513 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-05-24T20:55:15.513 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:55:15.513 INFO:teuthology.orchestra.run.smithi157.stdout:Install 5 Packages 2023-05-24T20:55:15.513 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:55:15.514 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 6.0 M 2023-05-24T20:55:15.514 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 26 M 2023-05-24T20:55:15.514 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-05-24T20:55:15.925 INFO:teuthology.orchestra.run.smithi157.stdout:(1/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 3.1 MB/s | 538 kB 00:00 2023-05-24T20:55:15.950 INFO:teuthology.orchestra.run.smithi157.stdout:(2/5): ceph-mgr-cephadm-18.0.0-3443.g3395903b.e 654 kB/s | 127 kB 00:00 2023-05-24T20:55:16.072 INFO:teuthology.orchestra.run.smithi157.stdout:(3/5): python3-babel-2.5.1-7.el8.noarch.rpm 15 MB/s | 4.8 MB 00:00 2023-05-24T20:55:16.093 INFO:teuthology.orchestra.run.smithi157.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 424 kB/s | 60 kB 00:00 2023-05-24T20:55:16.151 INFO:teuthology.orchestra.run.smithi157.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 2.1 MB/s | 476 kB 00:00 2023-05-24T20:55:16.152 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-05-24T20:55:16.152 INFO:teuthology.orchestra.run.smithi157.stdout:Total 9.3 MB/s | 6.0 MB 00:00 2023-05-24T20:55:16.204 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-05-24T20:55:16.215 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-05-24T20:55:16.215 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-05-24T20:55:16.318 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-05-24T20:55:16.321 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-05-24T20:55:16.580 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-05-24T20:55:16.913 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2023-05-24T20:55:17.851 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-05-24T20:55:18.210 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2023-05-24T20:55:18.427 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-05-24T20:55:18.597 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-3443.g3395903b.el8.noarch 5/5 2023-05-24T20:55:18.898 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-3443.g3395903b.el8.noarch 5/5 2023-05-24T20:55:18.899 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-3443.g3395903b.el8.noarch 1/5 2023-05-24T20:55:18.899 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2023-05-24T20:55:18.899 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2023-05-24T20:55:18.899 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-05-24T20:55:19.539 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2023-05-24T20:55:19.539 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:55:19.540 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-05-24T20:55:19.540 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-cephadm-2:18.0.0-3443.g3395903b.el8.noarch 2023-05-24T20:55:19.540 INFO:teuthology.orchestra.run.smithi157.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-05-24T20:55:19.541 INFO:teuthology.orchestra.run.smithi157.stdout: python3-babel-2.5.1-7.el8.noarch 2023-05-24T20:55:19.541 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-05-24T20:55:19.541 INFO:teuthology.orchestra.run.smithi157.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-05-24T20:55:19.541 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:55:19.542 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-05-24T20:55:19.645 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-fuse 2023-05-24T20:55:20.367 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:03 ago on Wed 24 May 2023 08:53:17 PM UTC. 2023-05-24T20:55:21.491 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-05-24T20:55:21.491 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:55:21.491 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2023-05-24T20:55:21.492 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:55:21.492 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-05-24T20:55:21.492 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-fuse x86_64 2:18.0.0-3443.g3395903b.el8 ceph 844 k 2023-05-24T20:55:21.492 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2023-05-24T20:55:21.492 INFO:teuthology.orchestra.run.smithi157.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-05-24T20:55:21.492 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:55:21.492 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-05-24T20:55:21.493 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:55:21.493 INFO:teuthology.orchestra.run.smithi157.stdout:Install 2 Packages 2023-05-24T20:55:21.493 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:55:21.493 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 927 k 2023-05-24T20:55:21.494 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 3.0 M 2023-05-24T20:55:21.494 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-05-24T20:55:21.843 INFO:teuthology.orchestra.run.smithi157.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 271 kB/s | 83 kB 00:00 2023-05-24T20:55:21.876 INFO:teuthology.orchestra.run.smithi157.stdout:(2/2): ceph-fuse-18.0.0-3443.g3395903b.el8.x86_ 2.4 MB/s | 844 kB 00:00 2023-05-24T20:55:21.877 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-05-24T20:55:21.877 INFO:teuthology.orchestra.run.smithi157.stdout:Total 2.4 MB/s | 927 kB 00:00 2023-05-24T20:55:21.887 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-05-24T20:55:21.908 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-05-24T20:55:21.908 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-05-24T20:55:21.993 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-05-24T20:55:21.996 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-05-24T20:55:22.164 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-05-24T20:55:22.441 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-05-24T20:55:22.688 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-fuse-2:18.0.0-3443.g3395903b.el8.x86_64 2/2 2023-05-24T20:55:23.084 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-fuse-2:18.0.0-3443.g3395903b.el8.x86_64 2/2 2023-05-24T20:55:23.084 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-fuse-2:18.0.0-3443.g3395903b.el8.x86_64 1/2 2023-05-24T20:55:23.645 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-05-24T20:55:23.645 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:55:23.646 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-05-24T20:55:23.646 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-fuse-2:18.0.0-3443.g3395903b.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-05-24T20:55:23.646 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:55:23.646 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-05-24T20:55:23.859 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-volume 2023-05-24T20:55:24.584 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:07 ago on Wed 24 May 2023 08:53:17 PM UTC. 2023-05-24T20:55:25.704 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-05-24T20:55:25.705 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:55:25.705 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2023-05-24T20:55:25.706 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:55:25.706 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-05-24T20:55:25.706 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-volume noarch 2:18.0.0-3443.g3395903b.el8 ceph-noarch 264 k 2023-05-24T20:55:25.706 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:55:25.706 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-05-24T20:55:25.706 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:55:25.707 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2023-05-24T20:55:25.707 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:55:25.707 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 264 k 2023-05-24T20:55:25.707 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 1.3 M 2023-05-24T20:55:25.707 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-05-24T20:55:26.105 INFO:teuthology.orchestra.run.smithi157.stdout:ceph-volume-18.0.0-3443.g3395903b.el8.noarch.rp 662 kB/s | 264 kB 00:00 2023-05-24T20:55:26.105 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-05-24T20:55:26.106 INFO:teuthology.orchestra.run.smithi157.stdout:Total 660 kB/s | 264 kB 00:00 2023-05-24T20:55:26.106 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-05-24T20:55:26.116 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-05-24T20:55:26.117 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-05-24T20:55:26.137 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-05-24T20:55:26.140 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-05-24T20:55:26.307 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-05-24T20:55:26.554 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-volume-2:18.0.0-3443.g3395903b.el8.noarch 1/1 2023-05-24T20:55:26.968 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-volume-2:18.0.0-3443.g3395903b.el8.noarch 1/1 2023-05-24T20:55:27.496 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-volume-2:18.0.0-3443.g3395903b.el8.noarch 1/1 2023-05-24T20:55:27.496 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:55:27.496 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-05-24T20:55:27.496 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-volume-2:18.0.0-3443.g3395903b.el8.noarch 2023-05-24T20:55:27.497 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:55:27.497 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-05-24T20:55:27.678 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install librados-devel 2023-05-24T20:55:28.411 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:11 ago on Wed 24 May 2023 08:53:17 PM UTC. 2023-05-24T20:55:29.180 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] gperftools-libs-2.7-9.el8.x86_64.rpm: Curl error (7): Couldn't connect to server for http://mirror.es.its.nyu.edu/epel/8/Everything/x86_64/Packages/g/gperftools-libs-2.7-9.el8.x86_64.rpm [Failed to connect to mirror.es.its.nyu.edu port 80: Connection timed out] 2023-05-24T20:55:29.181 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] liboath-2.6.2-3.el8.x86_64.rpm: Curl error (7): Couldn't connect to server for http://mirror.es.its.nyu.edu/epel/8/Everything/x86_64/Packages/l/liboath-2.6.2-3.el8.x86_64.rpm [Failed to connect to mirror.es.its.nyu.edu port 80: Connection timed out] 2023-05-24T20:55:29.181 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] libunwind-1.3.1-3.el8.x86_64.rpm: Curl error (7): Couldn't connect to server for http://mirror.es.its.nyu.edu/epel/8/Everything/x86_64/Packages/l/libunwind-1.3.1-3.el8.x86_64.rpm [Failed to connect to mirror.es.its.nyu.edu port 80: Connection timed out] 2023-05-24T20:55:29.537 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-05-24T20:55:29.537 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:55:29.537 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2023-05-24T20:55:29.537 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:55:29.538 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-05-24T20:55:29.538 INFO:teuthology.orchestra.run.smithi157.stdout: librados-devel x86_64 2:18.0.0-3443.g3395903b.el8 ceph 139 k 2023-05-24T20:55:29.538 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:55:29.538 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-05-24T20:55:29.538 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:55:29.538 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2023-05-24T20:55:29.538 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:55:29.539 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 139 k 2023-05-24T20:55:29.539 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 512 k 2023-05-24T20:55:29.539 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-05-24T20:55:29.911 INFO:teuthology.orchestra.run.smithi071.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 458 B/s | 59 kB 02:11 2023-05-24T20:55:29.936 INFO:teuthology.orchestra.run.smithi071.stdout:(19/23): libunwind-1.3.1-3.el8.x86_64.rpm 586 B/s | 75 kB 02:11 2023-05-24T20:55:30.128 INFO:teuthology.orchestra.run.smithi071.stdout:(20/23): gperftools-libs-2.7-9.el8.x86_64.rpm 2.3 kB/s | 306 kB 02:11 2023-05-24T20:55:30.144 INFO:teuthology.orchestra.run.smithi157.stdout:librados-devel-18.0.0-3443.g3395903b.el8.x86_64 230 kB/s | 139 kB 00:00 2023-05-24T20:55:30.145 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-05-24T20:55:30.145 INFO:teuthology.orchestra.run.smithi157.stdout:Total 229 kB/s | 139 kB 00:00 2023-05-24T20:55:30.145 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-05-24T20:55:30.151 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-05-24T20:55:30.152 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-05-24T20:55:30.205 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-05-24T20:55:30.208 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-05-24T20:55:30.328 INFO:teuthology.orchestra.run.smithi071.stdout:(21/23): librados2-18.0.0-3443.g3395903b.el8.x8 9.3 MB/s | 3.7 MB 00:00 2023-05-24T20:55:30.545 INFO:teuthology.orchestra.run.smithi071.stdout:(22/23): librbd1-18.0.0-3443.g3395903b.el8.x86_ 8.6 MB/s | 3.6 MB 00:00 2023-05-24T20:55:30.587 INFO:teuthology.orchestra.run.smithi071.stdout:(23/23): thrift-0.13.0-2.el8.x86_64.rpm 2.6 MB/s | 1.7 MB 00:00 2023-05-24T20:55:30.591 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-05-24T20:55:30.591 INFO:teuthology.orchestra.run.smithi071.stdout:Total 445 kB/s | 59 MB 02:15 2023-05-24T20:55:30.630 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-05-24T20:55:30.674 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-05-24T20:55:30.676 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-05-24T20:55:30.676 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-05-24T20:55:31.262 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-05-24T20:55:31.266 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-05-24T20:55:31.828 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librados-devel-2:18.0.0-3443.g3395903b.el8.x86_64 1/1 2023-05-24T20:55:32.143 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librados-devel-2:18.0.0-3443.g3395903b.el8.x86_64 1/1 2023-05-24T20:55:32.423 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-05-24T20:55:32.640 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2023-05-24T20:55:32.667 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librados-devel-2:18.0.0-3443.g3395903b.el8.x86_64 1/1 2023-05-24T20:55:32.667 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:55:32.667 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-05-24T20:55:32.667 INFO:teuthology.orchestra.run.smithi157.stdout: librados-devel-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:55:32.668 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:55:32.668 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-05-24T20:55:32.867 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install libcephfs2 2023-05-24T20:55:32.883 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/25 2023-05-24T20:55:33.067 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/25 2023-05-24T20:55:33.523 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/25 2023-05-24T20:55:33.589 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:16 ago on Wed 24 May 2023 08:53:17 PM UTC. 2023-05-24T20:55:33.725 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : librados2-2:18.0.0-3443.g3395903b.el8.x86_64 4/25 2023-05-24T20:55:33.821 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados2-2:18.0.0-3443.g3395903b.el8.x86_64 4/25 2023-05-24T20:55:34.136 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rados-2:18.0.0-3443.g3395903b.el8.x86_64 5/25 2023-05-24T20:55:34.350 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephfs2-2:18.0.0-3443.g3395903b.el8.x86_64 6/25 2023-05-24T20:55:34.577 INFO:teuthology.orchestra.run.smithi157.stdout:Package libcephfs2-2:18.0.0-3443.g3395903b.el8.x86_64 is already installed. 2023-05-24T20:55:34.717 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-05-24T20:55:34.718 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2023-05-24T20:55:34.718 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-05-24T20:55:34.747 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephfs2-2:18.0.0-3443.g3395903b.el8.x86_64 6/25 2023-05-24T20:55:34.807 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install libcephfs-devel 2023-05-24T20:55:34.967 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : librbd1-2:18.0.0-3443.g3395903b.el8.x86_64 7/25 2023-05-24T20:55:35.027 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librbd1-2:18.0.0-3443.g3395903b.el8.x86_64 7/25 2023-05-24T20:55:35.272 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/25 2023-05-24T20:55:35.433 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2023-05-24T20:55:35.505 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2023-05-24T20:55:35.544 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:18 ago on Wed 24 May 2023 08:53:17 PM UTC. 2023-05-24T20:55:35.702 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-ceph-argparse-2:18.0.0-3443.g3395903b.el8. 10/25 2023-05-24T20:55:35.961 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cephfs-2:18.0.0-3443.g3395903b.el8.x86_64 11/25 2023-05-24T20:55:36.250 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rbd-2:18.0.0-3443.g3395903b.el8.x86_64 12/25 2023-05-24T20:55:36.416 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libradosstriper1-2:18.0.0-3443.g3395903b.el8.x86_6 13/25 2023-05-24T20:55:36.487 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libradosstriper1-2:18.0.0-3443.g3395903b.el8.x86_6 13/25 2023-05-24T20:55:36.667 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-05-24T20:55:36.668 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:55:36.668 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2023-05-24T20:55:36.668 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:55:36.668 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-05-24T20:55:36.668 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs-devel x86_64 2:18.0.0-3443.g3395903b.el8 ceph 30 k 2023-05-24T20:55:36.668 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:55:36.669 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-05-24T20:55:36.669 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:55:36.669 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2023-05-24T20:55:36.669 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:55:36.669 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 30 k 2023-05-24T20:55:36.670 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 137 k 2023-05-24T20:55:36.670 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-05-24T20:55:36.725 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 14/25 2023-05-24T20:55:37.024 INFO:teuthology.orchestra.run.smithi157.stdout:libcephfs-devel-18.0.0-3443.g3395903b.el8.x86_6 86 kB/s | 30 kB 00:00 2023-05-24T20:55:37.025 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-05-24T20:55:37.025 INFO:teuthology.orchestra.run.smithi157.stdout:Total 86 kB/s | 30 kB 00:00 2023-05-24T20:55:37.025 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-05-24T20:55:37.028 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-05-24T20:55:37.029 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-05-24T20:55:37.057 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-05-24T20:55:37.059 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-05-24T20:55:37.114 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-05-24T20:55:37.301 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libcephfs-devel-2:18.0.0-3443.g3395903b.el8.x86_64 1/1 2023-05-24T20:55:37.521 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 15/25 2023-05-24T20:55:37.561 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-3443.g3395903b.el8.x86_64 1/1 2023-05-24T20:55:37.708 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librgw2-2:18.0.0-3443.g3395903b.el8.x86_64 16/25 2023-05-24T20:55:37.766 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librgw2-2:18.0.0-3443.g3395903b.el8.x86_64 16/25 2023-05-24T20:55:37.974 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rgw-2:18.0.0-3443.g3395903b.el8.x86_64 17/25 2023-05-24T20:55:38.062 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libcephfs-devel-2:18.0.0-3443.g3395903b.el8.x86_64 1/1 2023-05-24T20:55:38.062 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:55:38.062 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-05-24T20:55:38.062 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs-devel-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:55:38.062 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:55:38.063 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-05-24T20:55:38.171 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install librados2 2023-05-24T20:55:38.174 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-05-24T20:55:38.383 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-ceph-common-2:18.0.0-3443.g3395903b.el8.x8 19/25 2023-05-24T20:55:38.891 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:21 ago on Wed 24 May 2023 08:53:17 PM UTC. 2023-05-24T20:55:39.876 INFO:teuthology.orchestra.run.smithi157.stdout:Package librados2-2:18.0.0-3443.g3395903b.el8.x86_64 is already installed. 2023-05-24T20:55:40.013 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-05-24T20:55:40.014 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2023-05-24T20:55:40.014 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-05-24T20:55:40.104 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install librbd1 2023-05-24T20:55:40.821 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:23 ago on Wed 24 May 2023 08:53:17 PM UTC. 2023-05-24T20:55:40.899 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-common-2:18.0.0-3443.g3395903b.el8.x86_64 20/25 2023-05-24T20:55:41.116 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-common-2:18.0.0-3443.g3395903b.el8.x86_64 20/25 2023-05-24T20:55:41.702 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-common-2:18.0.0-3443.g3395903b.el8.x86_64 20/25 2023-05-24T20:55:41.798 INFO:teuthology.orchestra.run.smithi157.stdout:Package librbd1-2:18.0.0-3443.g3395903b.el8.x86_64 is already installed. 2023-05-24T20:55:41.938 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-05-24T20:55:41.939 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2023-05-24T20:55:41.940 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-05-24T20:55:42.017 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-base-2:18.0.0-3443.g3395903b.el8.x86_64 21/25 2023-05-24T20:55:42.027 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install python3-rados 2023-05-24T20:55:42.109 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-base-2:18.0.0-3443.g3395903b.el8.x86_64 21/25 2023-05-24T20:55:42.282 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-selinux-2:18.0.0-3443.g3395903b.el8.x86_64 22/25 2023-05-24T20:55:42.754 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:25 ago on Wed 24 May 2023 08:53:17 PM UTC. 2023-05-24T20:55:43.734 INFO:teuthology.orchestra.run.smithi157.stdout:Package python3-rados-2:18.0.0-3443.g3395903b.el8.x86_64 is already installed. 2023-05-24T20:55:43.873 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-05-24T20:55:43.874 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2023-05-24T20:55:43.875 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-05-24T20:55:43.960 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install python3-rgw 2023-05-24T20:55:44.684 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:27 ago on Wed 24 May 2023 08:53:17 PM UTC. 2023-05-24T20:55:45.659 INFO:teuthology.orchestra.run.smithi157.stdout:Package python3-rgw-2:18.0.0-3443.g3395903b.el8.x86_64 is already installed. 2023-05-24T20:55:45.798 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-05-24T20:55:45.799 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2023-05-24T20:55:45.800 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-05-24T20:55:45.884 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install python3-cephfs 2023-05-24T20:55:46.600 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:29 ago on Wed 24 May 2023 08:53:17 PM UTC. 2023-05-24T20:55:47.596 INFO:teuthology.orchestra.run.smithi157.stdout:Package python3-cephfs-2:18.0.0-3443.g3395903b.el8.x86_64 is already installed. 2023-05-24T20:55:47.736 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-05-24T20:55:47.737 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2023-05-24T20:55:47.737 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-05-24T20:55:47.831 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install python3-rbd 2023-05-24T20:55:48.551 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:31 ago on Wed 24 May 2023 08:53:17 PM UTC. 2023-05-24T20:55:49.536 INFO:teuthology.orchestra.run.smithi157.stdout:Package python3-rbd-2:18.0.0-3443.g3395903b.el8.x86_64 is already installed. 2023-05-24T20:55:49.674 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-05-24T20:55:49.675 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2023-05-24T20:55:49.675 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-05-24T20:55:49.766 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install rbd-fuse 2023-05-24T20:55:50.487 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:33 ago on Wed 24 May 2023 08:53:17 PM UTC. 2023-05-24T20:55:51.608 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-05-24T20:55:51.609 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:55:51.609 INFO:teuthology.orchestra.run.smithi157.stdout: Package Architecture Version Repository Size 2023-05-24T20:55:51.610 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:55:51.610 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-05-24T20:55:51.610 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-fuse x86_64 2:18.0.0-3443.g3395903b.el8 ceph 94 k 2023-05-24T20:55:51.610 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:55:51.610 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-05-24T20:55:51.611 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:55:51.611 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2023-05-24T20:55:51.611 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:55:51.611 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 94 k 2023-05-24T20:55:51.612 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 280 k 2023-05-24T20:55:51.612 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-05-24T20:55:51.988 INFO:teuthology.orchestra.run.smithi157.stdout:rbd-fuse-18.0.0-3443.g3395903b.el8.x86_64.rpm 250 kB/s | 94 kB 00:00 2023-05-24T20:55:51.989 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-05-24T20:55:51.989 INFO:teuthology.orchestra.run.smithi157.stdout:Total 249 kB/s | 94 kB 00:00 2023-05-24T20:55:51.989 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-05-24T20:55:52.000 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-05-24T20:55:52.001 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-05-24T20:55:52.048 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-05-24T20:55:52.051 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-05-24T20:55:52.155 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-05-24T20:55:52.390 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : rbd-fuse-2:18.0.0-3443.g3395903b.el8.x86_64 1/1 2023-05-24T20:55:52.670 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: rbd-fuse-2:18.0.0-3443.g3395903b.el8.x86_64 1/1 2023-05-24T20:55:53.222 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : rbd-fuse-2:18.0.0-3443.g3395903b.el8.x86_64 1/1 2023-05-24T20:55:53.223 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:55:53.223 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-05-24T20:55:53.223 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-fuse-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:55:53.223 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:55:53.223 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-05-24T20:55:53.389 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install rbd-mirror 2023-05-24T20:55:54.118 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:37 ago on Wed 24 May 2023 08:53:17 PM UTC. 2023-05-24T20:55:55.246 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-05-24T20:55:55.247 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:55:55.247 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2023-05-24T20:55:55.247 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:55:55.247 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-05-24T20:55:55.248 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-mirror x86_64 2:18.0.0-3443.g3395903b.el8 ceph 3.4 M 2023-05-24T20:55:55.248 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:55:55.248 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-05-24T20:55:55.248 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:55:55.248 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2023-05-24T20:55:55.249 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:55:55.249 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 3.4 M 2023-05-24T20:55:55.249 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 16 M 2023-05-24T20:55:55.249 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-05-24T20:55:56.657 INFO:teuthology.orchestra.run.smithi157.stdout:rbd-mirror-18.0.0-3443.g3395903b.el8.x86_64.rpm 2.4 MB/s | 3.4 MB 00:01 2023-05-24T20:55:56.658 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-05-24T20:55:56.659 INFO:teuthology.orchestra.run.smithi157.stdout:Total 2.4 MB/s | 3.4 MB 00:01 2023-05-24T20:55:56.659 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-05-24T20:55:56.673 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-05-24T20:55:56.673 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-05-24T20:55:56.737 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-05-24T20:55:56.741 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-05-24T20:55:57.180 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-05-24T20:55:57.349 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : rbd-mirror-2:18.0.0-3443.g3395903b.el8.x86_64 1/1 2023-05-24T20:55:57.766 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: rbd-mirror-2:18.0.0-3443.g3395903b.el8.x86_64 1/1 2023-05-24T20:55:58.285 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : rbd-mirror-2:18.0.0-3443.g3395903b.el8.x86_64 1/1 2023-05-24T20:55:58.285 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:55:58.286 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-05-24T20:55:58.286 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-mirror-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:55:58.286 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:55:58.286 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-05-24T20:55:58.481 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install rbd-nbd 2023-05-24T20:55:59.207 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:42 ago on Wed 24 May 2023 08:53:17 PM UTC. 2023-05-24T20:56:00.334 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-05-24T20:56:00.335 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:56:00.335 INFO:teuthology.orchestra.run.smithi157.stdout: Package Architecture Version Repository Size 2023-05-24T20:56:00.335 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:56:00.335 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-05-24T20:56:00.335 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-nbd x86_64 2:18.0.0-3443.g3395903b.el8 ceph 179 k 2023-05-24T20:56:00.335 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:56:00.335 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-05-24T20:56:00.336 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-05-24T20:56:00.336 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2023-05-24T20:56:00.336 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:56:00.336 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 179 k 2023-05-24T20:56:00.336 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 568 k 2023-05-24T20:56:00.336 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-05-24T20:56:00.519 INFO:teuthology.orchestra.run.smithi157.stdout:rbd-nbd-18.0.0-3443.g3395903b.el8.x86_64.rpm 975 kB/s | 179 kB 00:00 2023-05-24T20:56:00.520 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-05-24T20:56:00.520 INFO:teuthology.orchestra.run.smithi157.stdout:Total 968 kB/s | 179 kB 00:00 2023-05-24T20:56:00.521 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-05-24T20:56:00.532 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-05-24T20:56:00.532 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-05-24T20:56:00.580 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-05-24T20:56:00.583 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-05-24T20:56:00.696 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-05-24T20:56:00.880 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : rbd-nbd-2:18.0.0-3443.g3395903b.el8.x86_64 1/1 2023-05-24T20:56:00.930 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-selinux-2:18.0.0-3443.g3395903b.el8.x86_64 22/25 2023-05-24T20:56:00.930 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /sys 2023-05-24T20:56:00.930 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /proc 2023-05-24T20:56:00.930 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /dev 2023-05-24T20:56:00.930 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /run 2023-05-24T20:56:00.930 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /mnt 2023-05-24T20:56:00.931 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /var/tmp 2023-05-24T20:56:00.931 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /home 2023-05-24T20:56:00.931 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /tmp 2023-05-24T20:56:00.931 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /dev 2023-05-24T20:56:00.931 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:56:01.177 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: rbd-nbd-2:18.0.0-3443.g3395903b.el8.x86_64 1/1 2023-05-24T20:56:01.687 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : rbd-nbd-2:18.0.0-3443.g3395903b.el8.x86_64 1/1 2023-05-24T20:56:01.688 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:56:01.688 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-05-24T20:56:01.688 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-nbd-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:56:01.688 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:56:01.688 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-05-24T20:56:01.857 DEBUG:teuthology.parallel:result is None 2023-05-24T20:56:02.330 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-radosgw-2:18.0.0-3443.g3395903b.el8.x86_64 23/25 2023-05-24T20:56:02.350 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-3443.g3395903b.el8.x86_64 23/25 2023-05-24T20:56:02.350 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-05-24T20:56:02.571 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-05-24T20:56:02.572 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-05-24T20:56:03.234 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-05-24T20:56:03.234 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-base-2:18.0.0-3443.g3395903b.el8.x86_64 1/25 2023-05-24T20:56:03.235 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-common-2:18.0.0-3443.g3395903b.el8.x86_64 2/25 2023-05-24T20:56:03.235 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-radosgw-2:18.0.0-3443.g3395903b.el8.x86_64 3/25 2023-05-24T20:56:03.235 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-selinux-2:18.0.0-3443.g3395903b.el8.x86_64 4/25 2023-05-24T20:56:03.235 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephfs2-2:18.0.0-3443.g3395903b.el8.x86_64 5/25 2023-05-24T20:56:03.235 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libradosstriper1-2:18.0.0-3443.g3395903b.el8.x86_6 6/25 2023-05-24T20:56:03.235 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librgw2-2:18.0.0-3443.g3395903b.el8.x86_64 7/25 2023-05-24T20:56:03.235 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-ceph-argparse-2:18.0.0-3443.g3395903b.el8. 8/25 2023-05-24T20:56:03.236 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-ceph-common-2:18.0.0-3443.g3395903b.el8.x8 9/25 2023-05-24T20:56:03.236 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cephfs-2:18.0.0-3443.g3395903b.el8.x86_64 10/25 2023-05-24T20:56:03.236 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rados-2:18.0.0-3443.g3395903b.el8.x86_64 11/25 2023-05-24T20:56:03.236 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rbd-2:18.0.0-3443.g3395903b.el8.x86_64 12/25 2023-05-24T20:56:03.236 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rgw-2:18.0.0-3443.g3395903b.el8.x86_64 13/25 2023-05-24T20:56:03.236 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-05-24T20:56:03.236 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-05-24T20:56:03.237 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-05-24T20:56:03.237 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 17/25 2023-05-24T20:56:03.237 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-05-24T20:56:03.237 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2023-05-24T20:56:03.237 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2023-05-24T20:56:03.237 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2023-05-24T20:56:03.238 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados2-2:18.0.0-3443.g3395903b.el8.x86_64 22/25 2023-05-24T20:56:03.238 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-05-24T20:56:03.238 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librbd1-2:18.0.0-3443.g3395903b.el8.x86_64 24/25 2023-05-24T20:56:04.247 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-05-24T20:56:04.247 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:56:04.248 INFO:teuthology.orchestra.run.smithi071.stdout:Upgraded: 2023-05-24T20:56:04.248 INFO:teuthology.orchestra.run.smithi071.stdout: librados2-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:56:04.248 INFO:teuthology.orchestra.run.smithi071.stdout: librbd1-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:56:04.248 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-05-24T20:56:04.248 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-base-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:56:04.248 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-common-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:56:04.248 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-radosgw-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:56:04.249 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-selinux-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:56:04.249 INFO:teuthology.orchestra.run.smithi071.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-05-24T20:56:04.249 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs2-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:56:04.249 INFO:teuthology.orchestra.run.smithi071.stdout: liboath-2.6.2-3.el8.x86_64 2023-05-24T20:56:04.250 INFO:teuthology.orchestra.run.smithi071.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-05-24T20:56:04.250 INFO:teuthology.orchestra.run.smithi071.stdout: libradosstriper1-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:56:04.250 INFO:teuthology.orchestra.run.smithi071.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-05-24T20:56:04.250 INFO:teuthology.orchestra.run.smithi071.stdout: librgw2-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:56:04.250 INFO:teuthology.orchestra.run.smithi071.stdout: libunwind-1.3.1-3.el8.x86_64 2023-05-24T20:56:04.251 INFO:teuthology.orchestra.run.smithi071.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-05-24T20:56:04.251 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-argparse-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:56:04.251 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-common-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:56:04.251 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cephfs-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:56:04.251 INFO:teuthology.orchestra.run.smithi071.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-05-24T20:56:04.251 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rados-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:56:04.251 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rbd-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:56:04.252 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rgw-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:56:04.252 INFO:teuthology.orchestra.run.smithi071.stdout: thrift-0.13.0-2.el8.x86_64 2023-05-24T20:56:04.252 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:56:04.252 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-05-24T20:56:04.399 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-test 2023-05-24T20:56:05.148 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:54 ago on Wed 24 May 2023 08:53:11 PM UTC. 2023-05-24T20:56:06.282 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-05-24T20:56:06.283 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:56:06.283 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2023-05-24T20:56:06.283 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:56:06.283 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-05-24T20:56:06.283 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-test x86_64 2:18.0.0-3443.g3395903b.el8 ceph 47 M 2023-05-24T20:56:06.283 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2023-05-24T20:56:06.284 INFO:teuthology.orchestra.run.smithi071.stdout: jq x86_64 1.6-7.el8 appstream 202 k 2023-05-24T20:56:06.284 INFO:teuthology.orchestra.run.smithi071.stdout: libcephsqlite x86_64 2:18.0.0-3443.g3395903b.el8 ceph 137 k 2023-05-24T20:56:06.284 INFO:teuthology.orchestra.run.smithi071.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-05-24T20:56:06.284 INFO:teuthology.orchestra.run.smithi071.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-05-24T20:56:06.284 INFO:teuthology.orchestra.run.smithi071.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-05-24T20:56:06.284 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:56:06.284 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-05-24T20:56:06.285 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:56:06.285 INFO:teuthology.orchestra.run.smithi071.stdout:Install 6 Packages 2023-05-24T20:56:06.285 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:56:06.285 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 47 M 2023-05-24T20:56:06.286 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 210 M 2023-05-24T20:56:06.286 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-05-24T20:56:06.604 INFO:teuthology.orchestra.run.smithi071.stdout:(1/6): jq-1.6-7.el8.x86_64.rpm 1.5 MB/s | 202 kB 00:00 2023-05-24T20:56:06.647 INFO:teuthology.orchestra.run.smithi071.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 4.3 MB/s | 187 kB 00:00 2023-05-24T20:56:06.689 INFO:teuthology.orchestra.run.smithi071.stdout:(3/6): libcephsqlite-18.0.0-3443.g3395903b.el8. 628 kB/s | 137 kB 00:00 2023-05-24T20:56:06.714 INFO:teuthology.orchestra.run.smithi071.stdout:(4/6): socat-1.7.4.1-1.el8.x86_64.rpm 4.7 MB/s | 323 kB 00:00 2023-05-24T20:56:06.806 INFO:teuthology.orchestra.run.smithi071.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 604 kB/s | 71 kB 00:00 2023-05-24T20:56:10.672 INFO:teuthology.orchestra.run.smithi071.stdout:(6/6): ceph-test-18.0.0-3443.g3395903b.el8.x86_ 11 MB/s | 47 MB 00:04 2023-05-24T20:56:10.674 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-05-24T20:56:10.675 INFO:teuthology.orchestra.run.smithi071.stdout:Total 11 MB/s | 47 MB 00:04 2023-05-24T20:56:10.716 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-05-24T20:56:10.737 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-05-24T20:56:10.737 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-05-24T20:56:11.062 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-05-24T20:56:11.065 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-05-24T20:56:11.525 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-05-24T20:56:11.737 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-05-24T20:56:11.953 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-05-24T20:56:12.075 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-05-24T20:56:12.143 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-05-24T20:56:12.337 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : jq-1.6-7.el8.x86_64 4/6 2023-05-24T20:56:12.509 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephsqlite-2:18.0.0-3443.g3395903b.el8.x86_64 5/6 2023-05-24T20:56:17.000 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephsqlite-2:18.0.0-3443.g3395903b.el8.x86_64 5/6 2023-05-24T20:56:17.247 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-test-2:18.0.0-3443.g3395903b.el8.x86_64 6/6 2023-05-24T20:56:17.536 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-test-2:18.0.0-3443.g3395903b.el8.x86_64 6/6 2023-05-24T20:56:17.537 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-test-2:18.0.0-3443.g3395903b.el8.x86_64 1/6 2023-05-24T20:56:17.537 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephsqlite-2:18.0.0-3443.g3395903b.el8.x86_64 2/6 2023-05-24T20:56:17.537 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : jq-1.6-7.el8.x86_64 3/6 2023-05-24T20:56:17.537 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-05-24T20:56:17.538 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-05-24T20:56:18.133 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-05-24T20:56:18.133 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:56:18.134 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-05-24T20:56:18.134 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-test-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:56:18.134 INFO:teuthology.orchestra.run.smithi071.stdout: jq-1.6-7.el8.x86_64 2023-05-24T20:56:18.134 INFO:teuthology.orchestra.run.smithi071.stdout: libcephsqlite-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:56:18.134 INFO:teuthology.orchestra.run.smithi071.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-05-24T20:56:18.134 INFO:teuthology.orchestra.run.smithi071.stdout: socat-1.7.4.1-1.el8.x86_64 2023-05-24T20:56:18.134 INFO:teuthology.orchestra.run.smithi071.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-05-24T20:56:18.135 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:56:18.135 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-05-24T20:56:18.375 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph 2023-05-24T20:56:19.092 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:03:08 ago on Wed 24 May 2023 08:53:11 PM UTC. 2023-05-24T20:56:20.196 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-05-24T20:56:20.199 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:56:20.199 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2023-05-24T20:56:20.200 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:56:20.200 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-05-24T20:56:20.200 INFO:teuthology.orchestra.run.smithi071.stdout: ceph x86_64 2:18.0.0-3443.g3395903b.el8 ceph 6.4 k 2023-05-24T20:56:20.200 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2023-05-24T20:56:20.200 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mds x86_64 2:18.0.0-3443.g3395903b.el8 ceph 2.3 M 2023-05-24T20:56:20.200 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr x86_64 2:18.0.0-3443.g3395903b.el8 ceph 1.5 M 2023-05-24T20:56:20.200 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-modules-core noarch 2:18.0.0-3443.g3395903b.el8 ceph-noarch 240 k 2023-05-24T20:56:20.201 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mon x86_64 2:18.0.0-3443.g3395903b.el8 ceph 4.6 M 2023-05-24T20:56:20.201 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-osd x86_64 2:18.0.0-3443.g3395903b.el8 ceph 18 M 2023-05-24T20:56:20.201 INFO:teuthology.orchestra.run.smithi071.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-05-24T20:56:20.201 INFO:teuthology.orchestra.run.smithi071.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-05-24T20:56:20.201 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cffi x86_64 1.11.5-5.el8 baseos 237 k 2023-05-24T20:56:20.201 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-05-24T20:56:20.201 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-05-24T20:56:20.201 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cryptography x86_64 3.2.1-5.el8 baseos 559 k 2023-05-24T20:56:20.202 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-05-24T20:56:20.202 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-05-24T20:56:20.202 INFO:teuthology.orchestra.run.smithi071.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-05-24T20:56:20.202 INFO:teuthology.orchestra.run.smithi071.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-05-24T20:56:20.202 INFO:teuthology.orchestra.run.smithi071.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-05-24T20:56:20.202 INFO:teuthology.orchestra.run.smithi071.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-05-24T20:56:20.202 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-05-24T20:56:20.202 INFO:teuthology.orchestra.run.smithi071.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-05-24T20:56:20.203 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-05-24T20:56:20.203 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-05-24T20:56:20.203 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pytz noarch 2017.2-9.el8 appstream 54 k 2023-05-24T20:56:20.203 INFO:teuthology.orchestra.run.smithi071.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-05-24T20:56:20.203 INFO:teuthology.orchestra.run.smithi071.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-05-24T20:56:20.203 INFO:teuthology.orchestra.run.smithi071.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-05-24T20:56:20.203 INFO:teuthology.orchestra.run.smithi071.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-05-24T20:56:20.204 INFO:teuthology.orchestra.run.smithi071.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-05-24T20:56:20.204 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-05-24T20:56:20.204 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-05-24T20:56:20.204 INFO:teuthology.orchestra.run.smithi071.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-05-24T20:56:20.204 INFO:teuthology.orchestra.run.smithi071.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-05-24T20:56:20.204 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:56:20.204 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-05-24T20:56:20.204 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:56:20.205 INFO:teuthology.orchestra.run.smithi071.stdout:Install 32 Packages 2023-05-24T20:56:20.205 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:56:20.205 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 30 M 2023-05-24T20:56:20.205 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 119 M 2023-05-24T20:56:20.206 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-05-24T20:56:21.081 INFO:teuthology.orchestra.run.smithi071.stdout:(1/32): ceph-18.0.0-3443.g3395903b.el8.x86_64.r 18 kB/s | 6.4 kB 00:00 2023-05-24T20:56:22.156 INFO:teuthology.orchestra.run.smithi071.stdout:(2/32): ceph-mon-18.0.0-3443.g3395903b.el8.x86_ 4.3 MB/s | 4.6 MB 00:01 2023-05-24T20:56:23.664 INFO:teuthology.orchestra.run.smithi071.stdout:(3/32): ceph-mgr-18.0.0-3443.g3395903b.el8.x86_ 537 kB/s | 1.5 MB 00:02 2023-05-24T20:56:24.073 INFO:teuthology.orchestra.run.smithi071.stdout:(4/32): ceph-mgr-modules-core-18.0.0-3443.g3395 588 kB/s | 240 kB 00:00 2023-05-24T20:56:24.206 INFO:teuthology.orchestra.run.smithi071.stdout:(5/32): python3-mako-1.0.6-14.el8.noarch.rpm 1.2 MB/s | 157 kB 00:00 2023-05-24T20:56:24.282 INFO:teuthology.orchestra.run.smithi071.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 516 kB/s | 39 kB 00:00 2023-05-24T20:56:24.316 INFO:teuthology.orchestra.run.smithi071.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 3.0 MB/s | 103 kB 00:00 2023-05-24T20:56:24.357 INFO:teuthology.orchestra.run.smithi071.stdout:(8/32): python3-pytz-2017.2-9.el8.noarch.rpm 1.3 MB/s | 54 kB 00:00 2023-05-24T20:56:24.541 INFO:teuthology.orchestra.run.smithi071.stdout:(9/32): ceph-osd-18.0.0-3443.g3395903b.el8.x86_ 7.4 MB/s | 18 MB 00:02 2023-05-24T20:56:24.566 INFO:teuthology.orchestra.run.smithi071.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.1 MB/s | 457 kB 00:00 2023-05-24T20:56:24.991 INFO:teuthology.orchestra.run.smithi071.stdout:(11/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 528 kB/s | 237 kB 00:00 2023-05-24T20:56:25.083 INFO:teuthology.orchestra.run.smithi071.stdout:(12/32): python3-cryptography-3.2.1-5.el8.x86_6 1.1 MB/s | 559 kB 00:00 2023-05-24T20:56:25.109 INFO:teuthology.orchestra.run.smithi071.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 933 kB/s | 109 kB 00:00 2023-05-24T20:56:25.151 INFO:teuthology.orchestra.run.smithi071.stdout:(14/32): ceph-mds-18.0.0-3443.g3395903b.el8.x86 525 kB/s | 2.3 MB 00:04 2023-05-24T20:56:25.259 INFO:teuthology.orchestra.run.smithi071.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 248 kB/s | 44 kB 00:00 2023-05-24T20:56:25.301 INFO:teuthology.orchestra.run.smithi071.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 964 kB/s | 185 kB 00:00 2023-05-24T20:56:25.327 INFO:teuthology.orchestra.run.smithi071.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 985 kB/s | 173 kB 00:00 2023-05-24T20:56:25.352 INFO:teuthology.orchestra.run.smithi071.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 222 kB/s | 11 kB 00:00 2023-05-24T20:56:25.394 INFO:teuthology.orchestra.run.smithi071.stdout:(19/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.8 MB/s | 384 kB 00:00 2023-05-24T20:56:25.411 INFO:teuthology.orchestra.run.smithi071.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 209 kB/s | 18 kB 00:00 2023-05-24T20:56:25.428 INFO:teuthology.orchestra.run.smithi071.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 645 kB/s | 49 kB 00:00 2023-05-24T20:56:25.445 INFO:teuthology.orchestra.run.smithi071.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 1.1 MB/s | 59 kB 00:00 2023-05-24T20:56:25.478 INFO:teuthology.orchestra.run.smithi071.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 309 kB/s | 16 kB 00:00 2023-05-24T20:56:25.495 INFO:teuthology.orchestra.run.smithi071.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 373 kB/s | 19 kB 00:00 2023-05-24T20:56:25.521 INFO:teuthology.orchestra.run.smithi071.stdout:(25/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.5 MB/s | 283 kB 00:00 2023-05-24T20:56:25.537 INFO:teuthology.orchestra.run.smithi071.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 408 kB/s | 24 kB 00:00 2023-05-24T20:56:25.555 INFO:teuthology.orchestra.run.smithi071.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 497 kB/s | 29 kB 00:00 2023-05-24T20:56:25.571 INFO:teuthology.orchestra.run.smithi071.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 547 kB/s | 27 kB 00:00 2023-05-24T20:56:25.613 INFO:teuthology.orchestra.run.smithi071.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 3.3 MB/s | 255 kB 00:00 2023-05-24T20:56:25.638 INFO:teuthology.orchestra.run.smithi071.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.2 MB/s | 85 kB 00:00 2023-05-24T20:56:25.664 INFO:teuthology.orchestra.run.smithi071.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 457 kB/s | 23 kB 00:00 2023-05-24T20:56:25.689 INFO:teuthology.orchestra.run.smithi071.stdout:(32/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.8 MB/s | 251 kB 00:00 2023-05-24T20:56:25.690 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-05-24T20:56:25.690 INFO:teuthology.orchestra.run.smithi071.stdout:Total 5.4 MB/s | 30 MB 00:05 2023-05-24T20:56:25.773 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-05-24T20:56:25.811 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-05-24T20:56:25.811 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-05-24T20:56:26.111 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-05-24T20:56:26.114 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-05-24T20:56:27.214 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-05-24T20:56:27.436 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-05-24T20:56:27.646 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-05-24T20:56:27.905 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-05-24T20:56:28.162 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-05-24T20:56:28.343 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-05-24T20:56:28.539 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-05-24T20:56:28.725 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-05-24T20:56:28.902 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-05-24T20:56:29.178 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-05-24T20:56:29.373 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-05-24T20:56:29.620 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-05-24T20:56:29.848 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-05-24T20:56:30.259 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 13/32 2023-05-24T20:56:30.499 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 14/32 2023-05-24T20:56:30.671 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-05-24T20:56:30.928 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-05-24T20:56:31.172 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-05-24T20:56:31.482 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-05-24T20:56:31.668 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-05-24T20:56:31.865 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 20/32 2023-05-24T20:56:32.076 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-05-24T20:56:32.323 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-05-24T20:56:32.533 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-05-24T20:56:32.750 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-05-24T20:56:33.091 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-05-24T20:56:33.410 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-05-24T20:56:33.734 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-3443.g3395903b.el8. 27/32 2023-05-24T20:56:33.889 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-2:18.0.0-3443.g3395903b.el8.x86_64 28/32 2023-05-24T20:56:35.457 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-2:18.0.0-3443.g3395903b.el8.x86_64 28/32 2023-05-24T20:56:35.681 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-osd-2:18.0.0-3443.g3395903b.el8.x86_64 29/32 2023-05-24T20:56:36.137 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-osd-2:18.0.0-3443.g3395903b.el8.x86_64 29/32 2023-05-24T20:56:36.520 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mon-2:18.0.0-3443.g3395903b.el8.x86_64 30/32 2023-05-24T20:56:36.746 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mon-2:18.0.0-3443.g3395903b.el8.x86_64 30/32 2023-05-24T20:56:36.912 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mds-2:18.0.0-3443.g3395903b.el8.x86_64 31/32 2023-05-24T20:56:36.930 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mds-2:18.0.0-3443.g3395903b.el8.x86_64 31/32 2023-05-24T20:56:37.058 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-2:18.0.0-3443.g3395903b.el8.x86_64 32/32 2023-05-24T20:56:37.459 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-2:18.0.0-3443.g3395903b.el8.x86_64 32/32 2023-05-24T20:56:37.459 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-2:18.0.0-3443.g3395903b.el8.x86_64 1/32 2023-05-24T20:56:37.459 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mds-2:18.0.0-3443.g3395903b.el8.x86_64 2/32 2023-05-24T20:56:37.459 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-2:18.0.0-3443.g3395903b.el8.x86_64 3/32 2023-05-24T20:56:37.459 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mon-2:18.0.0-3443.g3395903b.el8.x86_64 4/32 2023-05-24T20:56:37.460 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-osd-2:18.0.0-3443.g3395903b.el8.x86_64 5/32 2023-05-24T20:56:37.460 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-3443.g3395903b.el8. 6/32 2023-05-24T20:56:37.460 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-05-24T20:56:37.460 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-05-24T20:56:37.460 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-05-24T20:56:37.460 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 10/32 2023-05-24T20:56:37.460 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-05-24T20:56:37.461 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 12/32 2023-05-24T20:56:37.461 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 13/32 2023-05-24T20:56:37.461 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-05-24T20:56:37.461 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-05-24T20:56:37.461 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-05-24T20:56:37.461 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-05-24T20:56:37.461 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-05-24T20:56:37.461 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-05-24T20:56:37.462 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-05-24T20:56:37.462 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-05-24T20:56:37.462 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-05-24T20:56:37.462 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-05-24T20:56:37.462 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-05-24T20:56:37.462 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-05-24T20:56:37.462 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-05-24T20:56:37.462 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-05-24T20:56:37.463 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-05-24T20:56:37.463 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-05-24T20:56:37.463 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-05-24T20:56:37.463 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-05-24T20:56:38.443 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-05-24T20:56:38.443 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:56:38.444 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-05-24T20:56:38.444 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:56:38.444 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mds-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:56:38.444 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:56:38.444 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-modules-core-2:18.0.0-3443.g3395903b.el8.noarch 2023-05-24T20:56:38.444 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mon-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:56:38.445 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-osd-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:56:38.445 INFO:teuthology.orchestra.run.smithi071.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-05-24T20:56:38.445 INFO:teuthology.orchestra.run.smithi071.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-05-24T20:56:38.445 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-05-24T20:56:38.445 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-05-24T20:56:38.445 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-05-24T20:56:38.445 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-05-24T20:56:38.445 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-6.2-6.el8.noarch 2023-05-24T20:56:38.446 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-05-24T20:56:38.446 INFO:teuthology.orchestra.run.smithi071.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-05-24T20:56:38.446 INFO:teuthology.orchestra.run.smithi071.stdout: python3-mako-1.0.6-14.el8.noarch 2023-05-24T20:56:38.446 INFO:teuthology.orchestra.run.smithi071.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-05-24T20:56:38.446 INFO:teuthology.orchestra.run.smithi071.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-05-24T20:56:38.446 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-05-24T20:56:38.447 INFO:teuthology.orchestra.run.smithi071.stdout: python3-portend-2.6-1.el8.noarch 2023-05-24T20:56:38.447 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-05-24T20:56:38.447 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pycparser-2.14-14.el8.noarch 2023-05-24T20:56:38.447 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pytz-2017.2-9.el8.noarch 2023-05-24T20:56:38.447 INFO:teuthology.orchestra.run.smithi071.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-05-24T20:56:38.447 INFO:teuthology.orchestra.run.smithi071.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-05-24T20:56:38.448 INFO:teuthology.orchestra.run.smithi071.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-05-24T20:56:38.448 INFO:teuthology.orchestra.run.smithi071.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-05-24T20:56:38.449 INFO:teuthology.orchestra.run.smithi071.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-05-24T20:56:38.449 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-05-24T20:56:38.449 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-05-24T20:56:38.450 INFO:teuthology.orchestra.run.smithi071.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-05-24T20:56:38.450 INFO:teuthology.orchestra.run.smithi071.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-05-24T20:56:38.450 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:56:38.450 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-05-24T20:56:38.684 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-base 2023-05-24T20:56:39.402 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:03:28 ago on Wed 24 May 2023 08:53:11 PM UTC. 2023-05-24T20:56:40.383 INFO:teuthology.orchestra.run.smithi071.stdout:Package ceph-base-2:18.0.0-3443.g3395903b.el8.x86_64 is already installed. 2023-05-24T20:56:40.519 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-05-24T20:56:40.520 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2023-05-24T20:56:40.521 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-05-24T20:56:40.620 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install cephadm 2023-05-24T20:56:41.342 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:03:30 ago on Wed 24 May 2023 08:53:11 PM UTC. 2023-05-24T20:56:42.469 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-05-24T20:56:42.470 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:56:42.470 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2023-05-24T20:56:42.470 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:56:42.471 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-05-24T20:56:42.471 INFO:teuthology.orchestra.run.smithi071.stdout: cephadm noarch 2:18.0.0-3443.g3395903b.el8 ceph-noarch 160 k 2023-05-24T20:56:42.471 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:56:42.472 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-05-24T20:56:42.472 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:56:42.472 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2023-05-24T20:56:42.473 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:56:42.473 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 160 k 2023-05-24T20:56:42.473 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 639 k 2023-05-24T20:56:42.473 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-05-24T20:56:43.003 INFO:teuthology.orchestra.run.smithi071.stdout:cephadm-18.0.0-3443.g3395903b.el8.noarch.rpm 300 kB/s | 160 kB 00:00 2023-05-24T20:56:43.004 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-05-24T20:56:43.004 INFO:teuthology.orchestra.run.smithi071.stdout:Total 299 kB/s | 160 kB 00:00 2023-05-24T20:56:43.005 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-05-24T20:56:43.013 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-05-24T20:56:43.013 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-05-24T20:56:43.015 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-05-24T20:56:43.018 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-05-24T20:56:43.062 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-05-24T20:56:43.517 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: cephadm-2:18.0.0-3443.g3395903b.el8.noarch 1/1 2023-05-24T20:56:43.667 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : cephadm-2:18.0.0-3443.g3395903b.el8.noarch 1/1 2023-05-24T20:56:43.896 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: cephadm-2:18.0.0-3443.g3395903b.el8.noarch 1/1 2023-05-24T20:56:44.409 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : cephadm-2:18.0.0-3443.g3395903b.el8.noarch 1/1 2023-05-24T20:56:44.409 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:56:44.409 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-05-24T20:56:44.409 INFO:teuthology.orchestra.run.smithi071.stdout: cephadm-2:18.0.0-3443.g3395903b.el8.noarch 2023-05-24T20:56:44.409 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:56:44.410 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-05-24T20:56:44.581 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-immutable-object-cache 2023-05-24T20:56:45.303 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:03:34 ago on Wed 24 May 2023 08:53:11 PM UTC. 2023-05-24T20:56:46.421 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-05-24T20:56:46.422 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:56:46.422 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2023-05-24T20:56:46.422 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:56:46.422 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-05-24T20:56:46.422 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-3443.g3395903b.el8 ceph 166 k 2023-05-24T20:56:46.423 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:56:46.423 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-05-24T20:56:46.423 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:56:46.423 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2023-05-24T20:56:46.423 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:56:46.424 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 166 k 2023-05-24T20:56:46.424 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 503 k 2023-05-24T20:56:46.424 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-05-24T20:56:47.217 INFO:teuthology.orchestra.run.smithi071.stdout:ceph-immutable-object-cache-18.0.0-3443.g339590 209 kB/s | 166 kB 00:00 2023-05-24T20:56:47.218 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-05-24T20:56:47.218 INFO:teuthology.orchestra.run.smithi071.stdout:Total 209 kB/s | 166 kB 00:00 2023-05-24T20:56:47.219 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-05-24T20:56:47.232 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-05-24T20:56:47.232 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-05-24T20:56:47.284 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-05-24T20:56:47.287 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-05-24T20:56:47.980 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-05-24T20:56:48.560 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-3443.g3395903b. 1/1 2023-05-24T20:56:49.016 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-3443.g3395903b. 1/1 2023-05-24T20:56:49.543 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-3443.g3395903b. 1/1 2023-05-24T20:56:49.543 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:56:49.544 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-05-24T20:56:49.544 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-immutable-object-cache-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:56:49.544 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:56:49.544 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-05-24T20:56:49.730 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr 2023-05-24T20:56:50.450 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:03:39 ago on Wed 24 May 2023 08:53:11 PM UTC. 2023-05-24T20:56:51.414 INFO:teuthology.orchestra.run.smithi071.stdout:Package ceph-mgr-2:18.0.0-3443.g3395903b.el8.x86_64 is already installed. 2023-05-24T20:56:51.551 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-05-24T20:56:51.553 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2023-05-24T20:56:51.553 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-05-24T20:56:51.663 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-dashboard 2023-05-24T20:56:52.370 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:03:41 ago on Wed 24 May 2023 08:53:11 PM UTC. 2023-05-24T20:56:53.484 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-05-24T20:56:53.485 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:56:53.486 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2023-05-24T20:56:53.486 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:56:53.486 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-05-24T20:56:53.486 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-dashboard noarch 2:18.0.0-3443.g3395903b.el8 ceph-noarch 1.7 M 2023-05-24T20:56:53.486 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2023-05-24T20:56:53.486 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-grafana-dashboards noarch 2:18.0.0-3443.g3395903b.el8 ceph-noarch 23 k 2023-05-24T20:56:53.487 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-prometheus-alerts noarch 2:18.0.0-3443.g3395903b.el8 ceph-noarch 15 k 2023-05-24T20:56:53.487 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-05-24T20:56:53.487 INFO:teuthology.orchestra.run.smithi071.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-05-24T20:56:53.487 INFO:teuthology.orchestra.run.smithi071.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-05-24T20:56:53.488 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:56:53.489 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-05-24T20:56:53.489 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:56:53.489 INFO:teuthology.orchestra.run.smithi071.stdout:Install 6 Packages 2023-05-24T20:56:53.489 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:56:53.490 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 2.0 M 2023-05-24T20:56:53.490 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 8.0 M 2023-05-24T20:56:53.491 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-05-24T20:56:53.892 INFO:teuthology.orchestra.run.smithi071.stdout:(1/6): ceph-prometheus-alerts-18.0.0-3443.g3395 96 kB/s | 15 kB 00:00 2023-05-24T20:56:53.917 INFO:teuthology.orchestra.run.smithi071.stdout:(2/6): ceph-grafana-dashboards-18.0.0-3443.g339 130 kB/s | 23 kB 00:00 2023-05-24T20:56:54.042 INFO:teuthology.orchestra.run.smithi071.stdout:(3/6): ceph-mgr-dashboard-18.0.0-3443.g3395903b 5.6 MB/s | 1.7 MB 00:00 2023-05-24T20:56:54.076 INFO:teuthology.orchestra.run.smithi071.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 212 kB/s | 34 kB 00:00 2023-05-24T20:56:54.218 INFO:teuthology.orchestra.run.smithi071.stdout:(5/6): python3-jwt-1.6.1-2.el8.noarch.rpm 131 kB/s | 43 kB 00:00 2023-05-24T20:56:54.293 INFO:teuthology.orchestra.run.smithi071.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 785 kB/s | 196 kB 00:00 2023-05-24T20:56:54.294 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-05-24T20:56:54.295 INFO:teuthology.orchestra.run.smithi071.stdout:Total 2.5 MB/s | 2.0 MB 00:00 2023-05-24T20:56:54.306 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-05-24T20:56:54.315 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-05-24T20:56:54.315 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-05-24T20:56:54.397 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-05-24T20:56:54.401 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-05-24T20:56:54.683 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-05-24T20:56:54.872 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-05-24T20:56:55.109 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-05-24T20:56:55.268 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-05-24T20:56:55.423 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-3443.g3395903b.el8.n 4/6 2023-05-24T20:56:55.744 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-3443.g3395903b.el8. 5/6 2023-05-24T20:56:55.917 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-3443.g3395903b.el8.noarc 6/6 2023-05-24T20:56:56.196 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-3443.g3395903b.el8.noarc 6/6 2023-05-24T20:56:56.196 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-3443.g3395903b.el8. 1/6 2023-05-24T20:56:56.196 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-3443.g3395903b.el8.noarc 2/6 2023-05-24T20:56:56.196 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-3443.g3395903b.el8.n 3/6 2023-05-24T20:56:56.197 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-05-24T20:56:56.197 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-05-24T20:56:56.747 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-05-24T20:56:56.747 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:56:56.747 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-05-24T20:56:56.747 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-grafana-dashboards-2:18.0.0-3443.g3395903b.el8.noarch 2023-05-24T20:56:56.747 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-dashboard-2:18.0.0-3443.g3395903b.el8.noarch 2023-05-24T20:56:56.748 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-prometheus-alerts-2:18.0.0-3443.g3395903b.el8.noarch 2023-05-24T20:56:56.748 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-05-24T20:56:56.748 INFO:teuthology.orchestra.run.smithi071.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-05-24T20:56:56.748 INFO:teuthology.orchestra.run.smithi071.stdout: python3-routes-2.4.1-12.el8.noarch 2023-05-24T20:56:56.748 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:56:56.748 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-05-24T20:56:56.867 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-diskprediction-local 2023-05-24T20:56:57.578 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:03:46 ago on Wed 24 May 2023 08:53:11 PM UTC. 2023-05-24T20:56:58.691 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-05-24T20:56:58.693 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================================ 2023-05-24T20:56:58.693 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2023-05-24T20:56:58.693 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================================ 2023-05-24T20:56:58.693 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-05-24T20:56:58.693 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-3443.g3395903b.el8 ceph-noarch 7.4 M 2023-05-24T20:56:58.693 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2023-05-24T20:56:58.694 INFO:teuthology.orchestra.run.smithi071.stdout: libgfortran x86_64 8.5.0-19.el8 baseos 645 k 2023-05-24T20:56:58.694 INFO:teuthology.orchestra.run.smithi071.stdout: libquadmath x86_64 8.5.0-19.el8 baseos 172 k 2023-05-24T20:56:58.694 INFO:teuthology.orchestra.run.smithi071.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-05-24T20:56:58.694 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-05-24T20:56:58.694 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-05-24T20:56:58.694 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-05-24T20:56:58.694 INFO:teuthology.orchestra.run.smithi071.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-05-24T20:56:58.694 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:56:58.695 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-05-24T20:56:58.695 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================================ 2023-05-24T20:56:58.695 INFO:teuthology.orchestra.run.smithi071.stdout:Install 8 Packages 2023-05-24T20:56:58.695 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:56:58.696 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 35 M 2023-05-24T20:56:58.696 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 210 M 2023-05-24T20:56:58.696 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-05-24T20:56:59.607 INFO:teuthology.orchestra.run.smithi071.stdout:(1/8): ceph-mgr-diskprediction-local-18.0.0-344 14 MB/s | 7.4 MB 00:00 2023-05-24T20:56:59.907 INFO:teuthology.orchestra.run.smithi071.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 5.8 MB/s | 4.9 MB 00:00 2023-05-24T20:56:59.966 INFO:teuthology.orchestra.run.smithi071.stdout:(3/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 10 MB/s | 3.7 MB 00:00 2023-05-24T20:56:59.991 INFO:teuthology.orchestra.run.smithi071.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.6 MB/s | 225 kB 00:00 2023-05-24T20:57:00.100 INFO:teuthology.orchestra.run.smithi071.stdout:(5/8): openblas-0.3.15-4.el8.x86_64.rpm 4.6 MB/s | 4.7 MB 00:01 2023-05-24T20:57:00.408 INFO:teuthology.orchestra.run.smithi071.stdout:(6/8): python3-scipy-1.0.0-21.module_el8.5.0+77 31 MB/s | 14 MB 00:00 2023-05-24T20:57:00.518 INFO:teuthology.orchestra.run.smithi071.stdout:(7/8): libquadmath-8.5.0-19.el8.x86_64.rpm 413 kB/s | 172 kB 00:00 2023-05-24T20:57:00.984 INFO:teuthology.orchestra.run.smithi071.stdout:(8/8): libgfortran-8.5.0-19.el8.x86_64.rpm 650 kB/s | 645 kB 00:00 2023-05-24T20:57:00.984 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-05-24T20:57:00.985 INFO:teuthology.orchestra.run.smithi071.stdout:Total 15 MB/s | 35 MB 00:02 2023-05-24T20:57:01.219 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-05-24T20:57:01.238 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-05-24T20:57:01.238 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-05-24T20:57:01.616 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-05-24T20:57:01.619 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-05-24T20:57:02.176 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-05-24T20:57:02.322 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libquadmath-8.5.0-19.el8.x86_64 1/8 2023-05-24T20:57:02.455 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libquadmath-8.5.0-19.el8.x86_64 1/8 2023-05-24T20:57:02.598 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libgfortran-8.5.0-19.el8.x86_64 2/8 2023-05-24T20:57:03.180 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libgfortran-8.5.0-19.el8.x86_64 2/8 2023-05-24T20:57:03.364 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-05-24T20:57:03.985 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-05-24T20:57:04.575 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-05-24T20:57:05.549 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-05-24T20:57:05.834 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-05-24T20:57:06.064 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-05-24T20:57:07.847 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-05-24T20:57:09.079 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-05-24T20:57:09.262 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-3443.g3395903 8/8 2023-05-24T20:57:09.561 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-3443.g3395903 8/8 2023-05-24T20:57:09.561 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-3443.g3395903 1/8 2023-05-24T20:57:09.561 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-05-24T20:57:09.562 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-05-24T20:57:09.562 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-05-24T20:57:09.562 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-05-24T20:57:09.563 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-05-24T20:57:09.563 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libgfortran-8.5.0-19.el8.x86_64 7/8 2023-05-24T20:57:10.157 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libquadmath-8.5.0-19.el8.x86_64 8/8 2023-05-24T20:57:10.157 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:57:10.157 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-05-24T20:57:10.158 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-diskprediction-local-2:18.0.0-3443.g3395903b.el8.noarch 2023-05-24T20:57:10.158 INFO:teuthology.orchestra.run.smithi071.stdout: libgfortran-8.5.0-19.el8.x86_64 2023-05-24T20:57:10.158 INFO:teuthology.orchestra.run.smithi071.stdout: libquadmath-8.5.0-19.el8.x86_64 2023-05-24T20:57:10.158 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-0.3.15-4.el8.x86_64 2023-05-24T20:57:10.159 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-05-24T20:57:10.159 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-05-24T20:57:10.160 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-05-24T20:57:10.160 INFO:teuthology.orchestra.run.smithi071.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-05-24T20:57:10.161 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:57:10.161 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-05-24T20:57:10.275 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-rook 2023-05-24T20:57:10.989 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:03:59 ago on Wed 24 May 2023 08:53:11 PM UTC. 2023-05-24T20:57:12.104 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-05-24T20:57:12.105 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:57:12.106 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2023-05-24T20:57:12.106 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:57:12.106 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-05-24T20:57:12.106 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-rook noarch 2:18.0.0-3443.g3395903b.el8 ceph-noarch 47 k 2023-05-24T20:57:12.106 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2023-05-24T20:57:12.106 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-05-24T20:57:12.106 INFO:teuthology.orchestra.run.smithi071.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-05-24T20:57:12.107 INFO:teuthology.orchestra.run.smithi071.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-05-24T20:57:12.107 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-05-24T20:57:12.107 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-05-24T20:57:12.107 INFO:teuthology.orchestra.run.smithi071.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-05-24T20:57:12.107 INFO:teuthology.orchestra.run.smithi071.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-05-24T20:57:12.107 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-05-24T20:57:12.107 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-05-24T20:57:12.108 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-05-24T20:57:12.108 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-05-24T20:57:12.108 INFO:teuthology.orchestra.run.smithi071.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-05-24T20:57:12.108 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:57:12.108 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-05-24T20:57:12.108 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:57:12.108 INFO:teuthology.orchestra.run.smithi071.stdout:Install 13 Packages 2023-05-24T20:57:12.109 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:57:12.109 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 1.9 M 2023-05-24T20:57:12.109 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 25 M 2023-05-24T20:57:12.110 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-05-24T20:57:12.348 INFO:teuthology.orchestra.run.smithi071.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 271 kB/s | 27 kB 00:00 2023-05-24T20:57:12.373 INFO:teuthology.orchestra.run.smithi071.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 162 kB/s | 20 kB 00:00 2023-05-24T20:57:12.423 INFO:teuthology.orchestra.run.smithi071.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.6 MB/s | 126 kB 00:00 2023-05-24T20:57:12.457 INFO:teuthology.orchestra.run.smithi071.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 1.3 MB/s | 110 kB 00:00 2023-05-24T20:57:12.483 INFO:teuthology.orchestra.run.smithi071.stdout:(5/13): ceph-mgr-rook-18.0.0-3443.g3395903b.el8 204 kB/s | 47 kB 00:00 2023-05-24T20:57:12.658 INFO:teuthology.orchestra.run.smithi071.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 189 kB/s | 33 kB 00:00 2023-05-24T20:57:12.708 INFO:teuthology.orchestra.run.smithi071.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 325 kB/s | 16 kB 00:00 2023-05-24T20:57:12.775 INFO:teuthology.orchestra.run.smithi071.stdout:(8/13): python3-requests-oauthlib-1.0.0-1.el8.n 134 kB/s | 43 kB 00:00 2023-05-24T20:57:12.825 INFO:teuthology.orchestra.run.smithi071.stdout:(9/13): python3-google-auth-1.1.1-10.el8.noarch 696 kB/s | 81 kB 00:00 2023-05-24T20:57:12.859 INFO:teuthology.orchestra.run.smithi071.stdout:(10/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 356 kB/s | 155 kB 00:00 2023-05-24T20:57:12.884 INFO:teuthology.orchestra.run.smithi071.stdout:(11/13): python3-rsa-4.8-1.el8.noarch.rpm 1.1 MB/s | 61 kB 00:00 2023-05-24T20:57:13.076 INFO:teuthology.orchestra.run.smithi071.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 282 kB/s | 61 kB 00:00 2023-05-24T20:57:13.410 INFO:teuthology.orchestra.run.smithi071.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 1.8 MB/s | 1.2 MB 00:00 2023-05-24T20:57:13.413 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-05-24T20:57:13.413 INFO:teuthology.orchestra.run.smithi071.stdout:Total 1.5 MB/s | 1.9 MB 00:01 2023-05-24T20:57:13.473 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-05-24T20:57:13.490 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-05-24T20:57:13.490 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-05-24T20:57:13.574 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-05-24T20:57:13.578 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-05-24T20:57:14.077 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-05-24T20:57:14.322 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-05-24T20:57:14.512 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-05-24T20:57:14.789 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rsa-4.8-1.el8.noarch 3/13 2023-05-24T20:57:15.010 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-05-24T20:57:15.200 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-05-24T20:57:15.432 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-05-24T20:57:15.707 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-05-24T20:57:15.923 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-05-24T20:57:16.955 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-05-24T20:57:17.201 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-05-24T20:57:17.418 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-05-24T20:57:17.663 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-05-24T20:57:17.819 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-rook-2:18.0.0-3443.g3395903b.el8.noarch 13/13 2023-05-24T20:57:18.118 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-3443.g3395903b.el8.noarch 13/13 2023-05-24T20:57:18.118 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-rook-2:18.0.0-3443.g3395903b.el8.noarch 1/13 2023-05-24T20:57:18.118 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-05-24T20:57:18.118 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-05-24T20:57:18.118 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-05-24T20:57:18.119 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-05-24T20:57:18.119 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-05-24T20:57:18.119 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-05-24T20:57:18.119 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-05-24T20:57:18.119 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-05-24T20:57:18.119 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-05-24T20:57:18.119 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-05-24T20:57:18.119 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 12/13 2023-05-24T20:57:18.735 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-05-24T20:57:18.735 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:57:18.736 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-05-24T20:57:18.736 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-rook-2:18.0.0-3443.g3395903b.el8.noarch 2023-05-24T20:57:18.736 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-05-24T20:57:18.736 INFO:teuthology.orchestra.run.smithi071.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-05-24T20:57:18.736 INFO:teuthology.orchestra.run.smithi071.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-05-24T20:57:18.736 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-05-24T20:57:18.737 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-05-24T20:57:18.738 INFO:teuthology.orchestra.run.smithi071.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-05-24T20:57:18.738 INFO:teuthology.orchestra.run.smithi071.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-05-24T20:57:18.738 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-05-24T20:57:18.738 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-05-24T20:57:18.738 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-05-24T20:57:18.738 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rsa-4.8-1.el8.noarch 2023-05-24T20:57:18.738 INFO:teuthology.orchestra.run.smithi071.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-05-24T20:57:18.739 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:57:18.739 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-05-24T20:57:18.845 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-cephadm 2023-05-24T20:57:19.556 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:04:08 ago on Wed 24 May 2023 08:53:11 PM UTC. 2023-05-24T20:57:20.668 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-05-24T20:57:20.669 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:57:20.670 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2023-05-24T20:57:20.670 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:57:20.670 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-05-24T20:57:20.670 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-cephadm noarch 2:18.0.0-3443.g3395903b.el8 ceph-noarch 127 k 2023-05-24T20:57:20.670 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2023-05-24T20:57:20.670 INFO:teuthology.orchestra.run.smithi071.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-05-24T20:57:20.670 INFO:teuthology.orchestra.run.smithi071.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-05-24T20:57:20.671 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-05-24T20:57:20.671 INFO:teuthology.orchestra.run.smithi071.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-05-24T20:57:20.671 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:57:20.671 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-05-24T20:57:20.671 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:57:20.671 INFO:teuthology.orchestra.run.smithi071.stdout:Install 5 Packages 2023-05-24T20:57:20.671 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:57:20.672 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 6.0 M 2023-05-24T20:57:20.672 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 26 M 2023-05-24T20:57:20.672 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-05-24T20:57:21.200 INFO:teuthology.orchestra.run.smithi071.stdout:(1/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 2.8 MB/s | 538 kB 00:00 2023-05-24T20:57:21.225 INFO:teuthology.orchestra.run.smithi071.stdout:(2/5): ceph-mgr-cephadm-18.0.0-3443.g3395903b.e 597 kB/s | 127 kB 00:00 2023-05-24T20:57:21.359 INFO:teuthology.orchestra.run.smithi071.stdout:(3/5): python3-babel-2.5.1-7.el8.noarch.rpm 14 MB/s | 4.8 MB 00:00 2023-05-24T20:57:21.734 INFO:teuthology.orchestra.run.smithi071.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 118 kB/s | 60 kB 00:00 2023-05-24T20:57:21.992 INFO:teuthology.orchestra.run.smithi071.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 600 kB/s | 476 kB 00:00 2023-05-24T20:57:21.994 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-05-24T20:57:21.995 INFO:teuthology.orchestra.run.smithi071.stdout:Total 4.5 MB/s | 6.0 MB 00:01 2023-05-24T20:57:22.048 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-05-24T20:57:22.059 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-05-24T20:57:22.059 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-05-24T20:57:22.164 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-05-24T20:57:22.167 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-05-24T20:57:22.429 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-05-24T20:57:22.723 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2023-05-24T20:57:23.658 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-05-24T20:57:24.006 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2023-05-24T20:57:24.223 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-05-24T20:57:24.418 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-3443.g3395903b.el8.noarch 5/5 2023-05-24T20:57:24.706 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-3443.g3395903b.el8.noarch 5/5 2023-05-24T20:57:24.707 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-3443.g3395903b.el8.noarch 1/5 2023-05-24T20:57:24.707 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2023-05-24T20:57:24.707 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2023-05-24T20:57:24.707 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-05-24T20:57:25.249 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2023-05-24T20:57:25.250 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:57:25.250 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-05-24T20:57:25.250 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-cephadm-2:18.0.0-3443.g3395903b.el8.noarch 2023-05-24T20:57:25.250 INFO:teuthology.orchestra.run.smithi071.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-05-24T20:57:25.250 INFO:teuthology.orchestra.run.smithi071.stdout: python3-babel-2.5.1-7.el8.noarch 2023-05-24T20:57:25.251 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-05-24T20:57:25.251 INFO:teuthology.orchestra.run.smithi071.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-05-24T20:57:25.251 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:57:25.251 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-05-24T20:57:25.402 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-fuse 2023-05-24T20:57:26.122 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:04:15 ago on Wed 24 May 2023 08:53:11 PM UTC. 2023-05-24T20:57:27.241 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-05-24T20:57:27.241 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:57:27.242 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2023-05-24T20:57:27.242 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:57:27.242 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-05-24T20:57:27.242 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-fuse x86_64 2:18.0.0-3443.g3395903b.el8 ceph 844 k 2023-05-24T20:57:27.242 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2023-05-24T20:57:27.243 INFO:teuthology.orchestra.run.smithi071.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-05-24T20:57:27.243 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:57:27.243 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-05-24T20:57:27.243 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:57:27.243 INFO:teuthology.orchestra.run.smithi071.stdout:Install 2 Packages 2023-05-24T20:57:27.243 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:57:27.244 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 927 k 2023-05-24T20:57:27.244 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 3.0 M 2023-05-24T20:57:27.244 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-05-24T20:57:27.541 INFO:teuthology.orchestra.run.smithi071.stdout:(1/2): ceph-fuse-18.0.0-3443.g3395903b.el8.x86_ 3.2 MB/s | 844 kB 00:00 2023-05-24T20:57:27.566 INFO:teuthology.orchestra.run.smithi071.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 296 kB/s | 83 kB 00:00 2023-05-24T20:57:27.567 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-05-24T20:57:27.567 INFO:teuthology.orchestra.run.smithi071.stdout:Total 2.8 MB/s | 927 kB 00:00 2023-05-24T20:57:27.577 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-05-24T20:57:27.599 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-05-24T20:57:27.599 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-05-24T20:57:27.684 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-05-24T20:57:27.687 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-05-24T20:57:27.864 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-05-24T20:57:28.143 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-05-24T20:57:28.350 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-fuse-2:18.0.0-3443.g3395903b.el8.x86_64 2/2 2023-05-24T20:57:28.742 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-fuse-2:18.0.0-3443.g3395903b.el8.x86_64 2/2 2023-05-24T20:57:28.742 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-fuse-2:18.0.0-3443.g3395903b.el8.x86_64 1/2 2023-05-24T20:57:29.285 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-05-24T20:57:29.285 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:57:29.285 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-05-24T20:57:29.285 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-fuse-2:18.0.0-3443.g3395903b.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-05-24T20:57:29.286 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:57:29.286 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-05-24T20:57:29.473 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-volume 2023-05-24T20:57:30.192 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:04:19 ago on Wed 24 May 2023 08:53:11 PM UTC. 2023-05-24T20:57:31.305 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-05-24T20:57:31.305 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:57:31.306 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2023-05-24T20:57:31.306 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:57:31.306 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-05-24T20:57:31.306 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-volume noarch 2:18.0.0-3443.g3395903b.el8 ceph-noarch 264 k 2023-05-24T20:57:31.306 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:57:31.306 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-05-24T20:57:31.306 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:57:31.307 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2023-05-24T20:57:31.307 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:57:31.307 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 264 k 2023-05-24T20:57:31.307 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 1.3 M 2023-05-24T20:57:31.308 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-05-24T20:57:31.519 INFO:teuthology.orchestra.run.smithi071.stdout:ceph-volume-18.0.0-3443.g3395903b.el8.noarch.rp 1.2 MB/s | 264 kB 00:00 2023-05-24T20:57:31.520 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-05-24T20:57:31.520 INFO:teuthology.orchestra.run.smithi071.stdout:Total 1.2 MB/s | 264 kB 00:00 2023-05-24T20:57:31.521 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-05-24T20:57:31.531 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-05-24T20:57:31.532 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-05-24T20:57:31.555 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-05-24T20:57:31.558 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-05-24T20:57:31.724 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-05-24T20:57:31.983 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-volume-2:18.0.0-3443.g3395903b.el8.noarch 1/1 2023-05-24T20:57:32.393 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-volume-2:18.0.0-3443.g3395903b.el8.noarch 1/1 2023-05-24T20:57:32.922 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-volume-2:18.0.0-3443.g3395903b.el8.noarch 1/1 2023-05-24T20:57:32.922 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:57:32.923 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-05-24T20:57:32.923 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-volume-2:18.0.0-3443.g3395903b.el8.noarch 2023-05-24T20:57:32.923 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:57:32.923 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-05-24T20:57:33.110 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install librados-devel 2023-05-24T20:57:33.841 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:04:22 ago on Wed 24 May 2023 08:53:11 PM UTC. 2023-05-24T20:57:34.956 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-05-24T20:57:34.957 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:57:34.957 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2023-05-24T20:57:34.957 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:57:34.957 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-05-24T20:57:34.957 INFO:teuthology.orchestra.run.smithi071.stdout: librados-devel x86_64 2:18.0.0-3443.g3395903b.el8 ceph 139 k 2023-05-24T20:57:34.958 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:57:34.958 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-05-24T20:57:34.958 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:57:34.958 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2023-05-24T20:57:34.958 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:57:34.958 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 139 k 2023-05-24T20:57:34.958 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 512 k 2023-05-24T20:57:34.958 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-05-24T20:57:35.148 INFO:teuthology.orchestra.run.smithi071.stdout:librados-devel-18.0.0-3443.g3395903b.el8.x86_64 720 kB/s | 139 kB 00:00 2023-05-24T20:57:35.149 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-05-24T20:57:35.149 INFO:teuthology.orchestra.run.smithi071.stdout:Total 715 kB/s | 139 kB 00:00 2023-05-24T20:57:35.149 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-05-24T20:57:35.154 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-05-24T20:57:35.154 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-05-24T20:57:35.204 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-05-24T20:57:35.207 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-05-24T20:57:35.332 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-05-24T20:57:35.564 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librados-devel-2:18.0.0-3443.g3395903b.el8.x86_64 1/1 2023-05-24T20:57:35.850 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados-devel-2:18.0.0-3443.g3395903b.el8.x86_64 1/1 2023-05-24T20:57:36.382 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados-devel-2:18.0.0-3443.g3395903b.el8.x86_64 1/1 2023-05-24T20:57:36.383 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:57:36.383 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-05-24T20:57:36.383 INFO:teuthology.orchestra.run.smithi071.stdout: librados-devel-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:57:36.383 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:57:36.383 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-05-24T20:57:36.593 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install libcephfs2 2023-05-24T20:57:37.310 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:04:26 ago on Wed 24 May 2023 08:53:11 PM UTC. 2023-05-24T20:57:38.291 INFO:teuthology.orchestra.run.smithi071.stdout:Package libcephfs2-2:18.0.0-3443.g3395903b.el8.x86_64 is already installed. 2023-05-24T20:57:38.429 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-05-24T20:57:38.430 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2023-05-24T20:57:38.430 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-05-24T20:57:38.782 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install libcephfs-devel 2023-05-24T20:57:39.502 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:04:28 ago on Wed 24 May 2023 08:53:11 PM UTC. 2023-05-24T20:57:40.609 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-05-24T20:57:40.610 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:57:40.610 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2023-05-24T20:57:40.611 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:57:40.611 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-05-24T20:57:40.611 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs-devel x86_64 2:18.0.0-3443.g3395903b.el8 ceph 30 k 2023-05-24T20:57:40.611 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:57:40.611 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-05-24T20:57:40.611 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:57:40.611 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2023-05-24T20:57:40.612 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:57:40.612 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 30 k 2023-05-24T20:57:40.612 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 137 k 2023-05-24T20:57:40.612 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-05-24T20:57:40.754 INFO:teuthology.orchestra.run.smithi071.stdout:libcephfs-devel-18.0.0-3443.g3395903b.el8.x86_6 212 kB/s | 30 kB 00:00 2023-05-24T20:57:40.755 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-05-24T20:57:40.755 INFO:teuthology.orchestra.run.smithi071.stdout:Total 210 kB/s | 30 kB 00:00 2023-05-24T20:57:40.756 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-05-24T20:57:40.759 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-05-24T20:57:40.759 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-05-24T20:57:40.779 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-05-24T20:57:40.784 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-05-24T20:57:40.839 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-05-24T20:57:41.030 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephfs-devel-2:18.0.0-3443.g3395903b.el8.x86_64 1/1 2023-05-24T20:57:41.313 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-3443.g3395903b.el8.x86_64 1/1 2023-05-24T20:57:41.820 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephfs-devel-2:18.0.0-3443.g3395903b.el8.x86_64 1/1 2023-05-24T20:57:41.820 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:57:41.820 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-05-24T20:57:41.820 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs-devel-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:57:41.821 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:57:41.821 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-05-24T20:57:41.935 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install librados2 2023-05-24T20:57:42.654 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:04:31 ago on Wed 24 May 2023 08:53:11 PM UTC. 2023-05-24T20:57:43.622 INFO:teuthology.orchestra.run.smithi071.stdout:Package librados2-2:18.0.0-3443.g3395903b.el8.x86_64 is already installed. 2023-05-24T20:57:43.766 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-05-24T20:57:43.766 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2023-05-24T20:57:43.766 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-05-24T20:57:43.855 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install librbd1 2023-05-24T20:57:44.570 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:04:33 ago on Wed 24 May 2023 08:53:11 PM UTC. 2023-05-24T20:57:45.546 INFO:teuthology.orchestra.run.smithi071.stdout:Package librbd1-2:18.0.0-3443.g3395903b.el8.x86_64 is already installed. 2023-05-24T20:57:45.685 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-05-24T20:57:45.686 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2023-05-24T20:57:45.687 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-05-24T20:57:45.792 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-rados 2023-05-24T20:57:46.506 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:04:35 ago on Wed 24 May 2023 08:53:11 PM UTC. 2023-05-24T20:57:47.483 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-rados-2:18.0.0-3443.g3395903b.el8.x86_64 is already installed. 2023-05-24T20:57:47.621 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-05-24T20:57:47.623 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2023-05-24T20:57:47.623 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-05-24T20:57:47.724 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-rgw 2023-05-24T20:57:48.437 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:04:37 ago on Wed 24 May 2023 08:53:11 PM UTC. 2023-05-24T20:57:49.465 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-rgw-2:18.0.0-3443.g3395903b.el8.x86_64 is already installed. 2023-05-24T20:57:49.603 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-05-24T20:57:49.604 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2023-05-24T20:57:49.604 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-05-24T20:57:49.865 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-cephfs 2023-05-24T20:57:50.582 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:04:39 ago on Wed 24 May 2023 08:53:11 PM UTC. 2023-05-24T20:57:51.560 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-cephfs-2:18.0.0-3443.g3395903b.el8.x86_64 is already installed. 2023-05-24T20:57:51.699 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-05-24T20:57:51.700 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2023-05-24T20:57:51.700 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-05-24T20:57:51.803 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-rbd 2023-05-24T20:57:52.524 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:04:41 ago on Wed 24 May 2023 08:53:11 PM UTC. 2023-05-24T20:57:53.511 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-rbd-2:18.0.0-3443.g3395903b.el8.x86_64 is already installed. 2023-05-24T20:57:53.648 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-05-24T20:57:53.649 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2023-05-24T20:57:53.650 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-05-24T20:57:53.753 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install rbd-fuse 2023-05-24T20:57:54.471 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:04:43 ago on Wed 24 May 2023 08:53:11 PM UTC. 2023-05-24T20:57:55.600 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-05-24T20:57:55.616 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:57:55.616 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2023-05-24T20:57:55.616 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:57:55.616 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-05-24T20:57:55.616 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-fuse x86_64 2:18.0.0-3443.g3395903b.el8 ceph 94 k 2023-05-24T20:57:55.617 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:57:55.617 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-05-24T20:57:55.617 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:57:55.617 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2023-05-24T20:57:55.617 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:57:55.617 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 94 k 2023-05-24T20:57:55.617 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 280 k 2023-05-24T20:57:55.618 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-05-24T20:57:56.060 INFO:teuthology.orchestra.run.smithi071.stdout:rbd-fuse-18.0.0-3443.g3395903b.el8.x86_64.rpm 206 kB/s | 94 kB 00:00 2023-05-24T20:57:56.060 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-05-24T20:57:56.061 INFO:teuthology.orchestra.run.smithi071.stdout:Total 205 kB/s | 94 kB 00:00 2023-05-24T20:57:56.061 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-05-24T20:57:56.071 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-05-24T20:57:56.071 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-05-24T20:57:56.119 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-05-24T20:57:56.122 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-05-24T20:57:56.229 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-05-24T20:57:56.410 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-fuse-2:18.0.0-3443.g3395903b.el8.x86_64 1/1 2023-05-24T20:57:56.725 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-fuse-2:18.0.0-3443.g3395903b.el8.x86_64 1/1 2023-05-24T20:57:57.275 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-fuse-2:18.0.0-3443.g3395903b.el8.x86_64 1/1 2023-05-24T20:57:57.275 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:57:57.276 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-05-24T20:57:57.276 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-fuse-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:57:57.276 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:57:57.276 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-05-24T20:57:57.517 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install rbd-mirror 2023-05-24T20:57:58.244 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:04:47 ago on Wed 24 May 2023 08:53:11 PM UTC. 2023-05-24T20:57:59.355 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-05-24T20:57:59.355 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:57:59.355 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2023-05-24T20:57:59.356 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:57:59.356 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-05-24T20:57:59.356 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-mirror x86_64 2:18.0.0-3443.g3395903b.el8 ceph 3.4 M 2023-05-24T20:57:59.356 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:57:59.356 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-05-24T20:57:59.357 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:57:59.357 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2023-05-24T20:57:59.357 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:57:59.357 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 3.4 M 2023-05-24T20:57:59.357 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 16 M 2023-05-24T20:57:59.358 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-05-24T20:58:02.912 INFO:teuthology.orchestra.run.smithi071.stdout:rbd-mirror-18.0.0-3443.g3395903b.el8.x86_64.rpm 994 kB/s | 3.4 MB 00:03 2023-05-24T20:58:02.913 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-05-24T20:58:02.913 INFO:teuthology.orchestra.run.smithi071.stdout:Total 993 kB/s | 3.4 MB 00:03 2023-05-24T20:58:02.914 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-05-24T20:58:02.928 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-05-24T20:58:02.928 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-05-24T20:58:02.994 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-05-24T20:58:02.997 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-05-24T20:58:03.435 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-05-24T20:58:03.610 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-mirror-2:18.0.0-3443.g3395903b.el8.x86_64 1/1 2023-05-24T20:58:04.008 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-mirror-2:18.0.0-3443.g3395903b.el8.x86_64 1/1 2023-05-24T20:58:04.535 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-mirror-2:18.0.0-3443.g3395903b.el8.x86_64 1/1 2023-05-24T20:58:04.536 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:04.536 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-05-24T20:58:04.536 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-mirror-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:58:04.536 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:04.536 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-05-24T20:58:04.726 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install rbd-nbd 2023-05-24T20:58:05.448 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:04:54 ago on Wed 24 May 2023 08:53:11 PM UTC. 2023-05-24T20:58:06.562 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-05-24T20:58:06.562 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:58:06.563 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2023-05-24T20:58:06.563 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:58:06.563 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-05-24T20:58:06.563 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-nbd x86_64 2:18.0.0-3443.g3395903b.el8 ceph 179 k 2023-05-24T20:58:06.563 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:06.563 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-05-24T20:58:06.563 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-05-24T20:58:06.564 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2023-05-24T20:58:06.564 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:06.564 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 179 k 2023-05-24T20:58:06.564 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 568 k 2023-05-24T20:58:06.565 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-05-24T20:58:07.114 INFO:teuthology.orchestra.run.smithi071.stdout:rbd-nbd-18.0.0-3443.g3395903b.el8.x86_64.rpm 325 kB/s | 179 kB 00:00 2023-05-24T20:58:07.115 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-05-24T20:58:07.115 INFO:teuthology.orchestra.run.smithi071.stdout:Total 324 kB/s | 179 kB 00:00 2023-05-24T20:58:07.116 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-05-24T20:58:07.127 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-05-24T20:58:07.127 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-05-24T20:58:07.177 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-05-24T20:58:07.180 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-05-24T20:58:07.294 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-05-24T20:58:07.466 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-nbd-2:18.0.0-3443.g3395903b.el8.x86_64 1/1 2023-05-24T20:58:07.753 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-nbd-2:18.0.0-3443.g3395903b.el8.x86_64 1/1 2023-05-24T20:58:08.286 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-nbd-2:18.0.0-3443.g3395903b.el8.x86_64 1/1 2023-05-24T20:58:08.287 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:08.287 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-05-24T20:58:08.287 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-nbd-2:18.0.0-3443.g3395903b.el8.x86_64 2023-05-24T20:58:08.287 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:08.287 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-05-24T20:58:08.471 DEBUG:teuthology.parallel:result is None 2023-05-24T20:58:08.472 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3395903b291ea37c38b39a05e6d9aa4f40306c80 2023-05-24T20:58:08.616 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-05-24T20:58:08.639 INFO:teuthology.orchestra.run.smithi071.stdout:18.0.0-3443.g3395903b.el8 2023-05-24T20:58:08.640 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-3443.g3395903b.el8 2023-05-24T20:58:08.640 INFO:teuthology.task.install:The correct ceph version 18.0.0-3443.g3395903b is installed. 2023-05-24T20:58:08.641 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3395903b291ea37c38b39a05e6d9aa4f40306c80 2023-05-24T20:58:08.781 DEBUG:teuthology.orchestra.run.smithi157:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-05-24T20:58:08.809 INFO:teuthology.orchestra.run.smithi157.stdout:18.0.0-3443.g3395903b.el8 2023-05-24T20:58:08.809 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-3443.g3395903b.el8 2023-05-24T20:58:08.810 INFO:teuthology.task.install:The correct ceph version 18.0.0-3443.g3395903b is installed. 2023-05-24T20:58:08.810 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-05-24T20:58:08.811 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-05-24T20:58:08.811 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-05-24T20:58:08.839 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-05-24T20:58:08.840 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-05-24T20:58:08.883 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-05-24T20:58:08.883 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-05-24T20:58:08.884 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/daemon-helper 2023-05-24T20:58:08.910 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-05-24T20:58:08.978 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-05-24T20:58:08.978 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/daemon-helper 2023-05-24T20:58:09.010 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-05-24T20:58:09.079 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-05-24T20:58:09.080 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-05-24T20:58:09.080 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/adjust-ulimits 2023-05-24T20:58:09.107 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-05-24T20:58:09.174 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-05-24T20:58:09.174 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/adjust-ulimits 2023-05-24T20:58:09.203 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-05-24T20:58:09.271 INFO:teuthology.run_tasks:Running task ceph... 2023-05-24T20:58:09.351 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-05-24T20:58:09.352 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 777 /var/log/ceph 2023-05-24T20:58:09.354 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 777 /var/log/ceph 2023-05-24T20:58:09.381 INFO:tasks.ceph:Disabling ceph logrotate... 2023-05-24T20:58:09.382 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /etc/logrotate.d/ceph 2023-05-24T20:58:09.422 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /etc/logrotate.d/ceph 2023-05-24T20:58:09.449 INFO:tasks.ceph:Creating extra log directories... 2023-05-24T20:58:09.449 DEBUG:teuthology.orchestra.run.smithi071:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-05-24T20:58:09.489 DEBUG:teuthology.orchestra.run.smithi157:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-05-24T20:58:09.528 INFO:tasks.ceph:Creating ceph cluster c1... 2023-05-24T20:58:09.529 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-05-24T20:58:09.529 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-05-24_20:32:11-rgw-wip-yuri-testing-2023-05-24-0738-reef-distro-default-smithi/7285505', 'branch': 'wip-yuri-testing-2023-05-24-0738-reef', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8} tasks/test_multi}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7285505', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-05-24_20:32:11-rgw-wip-yuri-testing-2023-05-24-0738-reef-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri-testing-2023-05-24-0738-reef'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '3395903b291ea37c38b39a05e6d9aa4f40306c80', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '3395903b291ea37c38b39a05e6d9aa4f40306c80'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a1'}, {'endpoints': ['c1.client.1'], 'name': 'a2'}]}, {'enabled_features': ['resharding'], 'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'b', 'zones': [{'endpoints': ['c2.client.0'], 'is_default': True, 'is_master': True, 'name': 'b1'}, {'endpoints': ['c2.client.1'], 'name': 'b2'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-yuri-testing-2023-05-24-0738-reef', 'sha1': '3395903b291ea37c38b39a05e6d9aa4f40306c80'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'seed': 8560, 'sha1': '3395903b291ea37c38b39a05e6d9aa4f40306c80', 'sleep_before_teardown': 0, 'subset': '111/12000', 'suite': 'rgw', 'suite_branch': 'wip-yuri-testing-2023-05-24-0738-reef', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '3395903b291ea37c38b39a05e6d9aa4f40306c80', 'targets': {'smithi071.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA5CM/4hsfiqXE11jZAHgpOo2FbW4ZOzUXKXNJNz9fWWeDtJTYStjEV6efAd9vMM91OzJdMX3iMjxJTS8Y5i2uY=', 'smithi157.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO+lmSy90RyAYeFfSD6aXgxRUJPCkLZkHkQxsxqLXA3D4XXntKSNvOhxkzpRSCW+Uo3pl5PdjBfPz+AOeUYtCwU='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '3395903b291ea37c38b39a05e6d9aa4f40306c80', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6fb4f0c910de3902d27909a642e8d50d731dd28c', 'timestamp': '2023-05-24_20:32:11', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.23459'} 2023-05-24T20:58:09.530 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2023-05-24T20:58:09.557 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2023-05-24T20:58:09.576 DEBUG:teuthology.orchestra.run.smithi071:> sudo install -d -m0777 -- /var/run/ceph 2023-05-24T20:58:09.615 DEBUG:teuthology.orchestra.run.smithi157:> sudo install -d -m0777 -- /var/run/ceph 2023-05-24T20:58:09.643 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-05-24T20:58:09.644 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/scratch_devs of=/dev/stdout 2023-05-24T20:58:09.698 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-05-24T20:58:09.698 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_1 2023-05-24T20:58:09.755 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-05-24T20:58:09.756 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-05-24T20:58:09.756 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 6h/6d Inode: 368793 Links: 1 2023-05-24T20:58:09.756 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-05-24T20:58:09.756 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2023-05-24T20:58:09.756 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2023-05-24 20:58:03.757508955 +0000 2023-05-24T20:58:09.756 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2023-05-24 20:52:19.742424391 +0000 2023-05-24T20:58:09.757 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2023-05-24 20:52:19.742424391 +0000 2023-05-24T20:58:09.757 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: - 2023-05-24T20:58:09.757 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-05-24T20:58:09.821 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2023-05-24T20:58:09.822 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2023-05-24T20:58:09.822 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000258862 s, 2.0 MB/s 2023-05-24T20:58:09.823 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-05-24T20:58:09.881 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_2 2023-05-24T20:58:09.939 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-05-24T20:58:09.940 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-05-24T20:58:09.940 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 6h/6d Inode: 366089 Links: 1 2023-05-24T20:58:09.940 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-05-24T20:58:09.940 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2023-05-24T20:58:09.940 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2023-05-24 20:58:03.757508955 +0000 2023-05-24T20:58:09.941 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2023-05-24 20:52:20.254415225 +0000 2023-05-24T20:58:09.941 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2023-05-24 20:52:20.254415225 +0000 2023-05-24T20:58:09.941 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: - 2023-05-24T20:58:09.941 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-05-24T20:58:10.005 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2023-05-24T20:58:10.005 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2023-05-24T20:58:10.006 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000219241 s, 2.3 MB/s 2023-05-24T20:58:10.006 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-05-24T20:58:10.065 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_3 2023-05-24T20:58:10.123 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-05-24T20:58:10.124 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-05-24T20:58:10.124 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 6h/6d Inode: 368926 Links: 1 2023-05-24T20:58:10.124 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-05-24T20:58:10.124 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2023-05-24T20:58:10.124 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2023-05-24 20:58:03.757508955 +0000 2023-05-24T20:58:10.125 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2023-05-24 20:52:20.788405665 +0000 2023-05-24T20:58:10.125 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2023-05-24 20:52:20.788405665 +0000 2023-05-24T20:58:10.125 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: - 2023-05-24T20:58:10.125 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-05-24T20:58:10.189 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2023-05-24T20:58:10.189 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2023-05-24T20:58:10.190 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000224669 s, 2.3 MB/s 2023-05-24T20:58:10.191 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-05-24T20:58:10.249 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_4 2023-05-24T20:58:10.308 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-05-24T20:58:10.308 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-05-24T20:58:10.308 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 6h/6d Inode: 368639 Links: 1 2023-05-24T20:58:10.308 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-05-24T20:58:10.308 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2023-05-24T20:58:10.309 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2023-05-24 20:58:03.757508955 +0000 2023-05-24T20:58:10.309 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2023-05-24 20:52:21.307396373 +0000 2023-05-24T20:58:10.309 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2023-05-24 20:52:21.307396373 +0000 2023-05-24T20:58:10.309 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: - 2023-05-24T20:58:10.309 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-05-24T20:58:10.373 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2023-05-24T20:58:10.373 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2023-05-24T20:58:10.373 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000236204 s, 2.2 MB/s 2023-05-24T20:58:10.374 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-05-24T20:58:10.433 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-05-24T20:58:10.433 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi071.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-05-24T20:58:10.434 INFO:tasks.ceph:Generating config... 2023-05-24T20:58:10.435 INFO:tasks.ceph:[client] debug rgw = 20 2023-05-24T20:58:10.436 INFO:tasks.ceph:[client] debug rgw sync = 20 2023-05-24T20:58:10.436 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2023-05-24T20:58:10.436 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2023-05-24T20:58:10.436 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-05-24T20:58:10.436 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2023-05-24T20:58:10.437 INFO:tasks.ceph:[client] rgw data log num shards = 4 2023-05-24T20:58:10.437 INFO:tasks.ceph:[client] rgw md log max shards = 4 2023-05-24T20:58:10.437 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2023-05-24T20:58:10.437 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2023-05-24T20:58:10.437 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2023-05-24T20:58:10.438 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2023-05-24T20:58:10.438 INFO:tasks.ceph:[client] setgroup = ceph 2023-05-24T20:58:10.438 INFO:tasks.ceph:[client] setuser = ceph 2023-05-24T20:58:10.438 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2023-05-24T20:58:10.438 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-05-24T20:58:10.439 INFO:tasks.ceph:[mgr] debug ms = 1 2023-05-24T20:58:10.439 INFO:tasks.ceph:[mon] debug mon = 20 2023-05-24T20:58:10.439 INFO:tasks.ceph:[mon] debug ms = 1 2023-05-24T20:58:10.439 INFO:tasks.ceph:[mon] debug paxos = 20 2023-05-24T20:58:10.439 INFO:tasks.ceph:[osd] debug ms = 1 2023-05-24T20:58:10.440 INFO:tasks.ceph:[osd] debug osd = 20 2023-05-24T20:58:10.440 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2023-05-24T20:58:10.440 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2023-05-24T20:58:10.441 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2023-05-24T20:58:10.441 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2023-05-24T20:58:10.442 INFO:tasks.ceph:Setting up c1.mon.a... 2023-05-24T20:58:10.442 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2023-05-24T20:58:10.520 INFO:teuthology.orchestra.run.smithi071.stdout:creating /etc/ceph/c1.keyring 2023-05-24T20:58:10.522 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2023-05-24T20:58:10.568 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0644 /etc/ceph/c1.keyring 2023-05-24T20:58:10.594 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.71')] 2023-05-24T20:58:10.595 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.71'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2023-05-24T20:58:10.596 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-05-24T20:58:10.596 DEBUG:teuthology.orchestra.run.smithi071:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-05-24T20:58:10.653 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.71 --print /home/ubuntu/cephtest/c1.monmap 2023-05-24T20:58:10.735 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T20:58:10.735 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T20:58:10.736 INFO:teuthology.orchestra.run.smithi071.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2023-05-24T20:58:10.737 INFO:teuthology.orchestra.run.smithi071.stdout:monmaptool: generated fsid 39018ae0-8be5-4c54-a564-aa238e12e56e 2023-05-24T20:58:10.737 INFO:teuthology.orchestra.run.smithi071.stdout:setting min_mon_release = pacific 2023-05-24T20:58:10.737 INFO:teuthology.orchestra.run.smithi071.stdout:epoch 0 2023-05-24T20:58:10.737 INFO:teuthology.orchestra.run.smithi071.stdout:fsid 39018ae0-8be5-4c54-a564-aa238e12e56e 2023-05-24T20:58:10.737 INFO:teuthology.orchestra.run.smithi071.stdout:last_changed 2023-05-24T20:58:10.730628+0000 2023-05-24T20:58:10.737 INFO:teuthology.orchestra.run.smithi071.stdout:created 2023-05-24T20:58:10.730628+0000 2023-05-24T20:58:10.737 INFO:teuthology.orchestra.run.smithi071.stdout:min_mon_release 16 (pacific) 2023-05-24T20:58:10.738 INFO:teuthology.orchestra.run.smithi071.stdout:election_strategy: 1 2023-05-24T20:58:10.738 INFO:teuthology.orchestra.run.smithi071.stdout:0: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.a 2023-05-24T20:58:10.743 INFO:teuthology.orchestra.run.smithi071.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2023-05-24T20:58:10.745 DEBUG:teuthology.orchestra.run.smithi071:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-05-24T20:58:10.802 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID 39018ae0-8be5-4c54-a564-aa238e12e56e... 2023-05-24T20:58:10.803 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2023-05-24T20:58:10.845 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2023-05-24T20:58:10.888 INFO:teuthology.orchestra.run.smithi071.stdout:[global] 2023-05-24T20:58:10.889 INFO:teuthology.orchestra.run.smithi071.stdout: chdir = "" 2023-05-24T20:58:10.889 INFO:teuthology.orchestra.run.smithi071.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-05-24T20:58:10.889 INFO:teuthology.orchestra.run.smithi071.stdout: auth supported = cephx 2023-05-24T20:58:10.889 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:10.890 INFO:teuthology.orchestra.run.smithi071.stdout: filestore xattr use omap = true 2023-05-24T20:58:10.890 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:10.890 INFO:teuthology.orchestra.run.smithi071.stdout: mon clock drift allowed = 1.000 2023-05-24T20:58:10.890 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:10.890 INFO:teuthology.orchestra.run.smithi071.stdout: osd crush chooseleaf type = 0 2023-05-24T20:58:10.890 INFO:teuthology.orchestra.run.smithi071.stdout: auth debug = true 2023-05-24T20:58:10.890 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:10.890 INFO:teuthology.orchestra.run.smithi071.stdout: ms die on old message = true 2023-05-24T20:58:10.891 INFO:teuthology.orchestra.run.smithi071.stdout: ms die on bug = true 2023-05-24T20:58:10.891 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:10.891 INFO:teuthology.orchestra.run.smithi071.stdout: mon max pg per osd = 10000 # >= luminous 2023-05-24T20:58:10.891 INFO:teuthology.orchestra.run.smithi071.stdout: mon pg warn max object skew = 0 2023-05-24T20:58:10.891 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:10.891 INFO:teuthology.orchestra.run.smithi071.stdout: # disable pg_autoscaler by default for new pools 2023-05-24T20:58:10.891 INFO:teuthology.orchestra.run.smithi071.stdout: osd_pool_default_pg_autoscale_mode = off 2023-05-24T20:58:10.892 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:10.892 INFO:teuthology.orchestra.run.smithi071.stdout: osd pool default size = 2 2023-05-24T20:58:10.892 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:10.892 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd allow primary affinity = true 2023-05-24T20:58:10.892 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd allow pg remap = true 2023-05-24T20:58:10.892 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on legacy crush tunables = false 2023-05-24T20:58:10.892 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on crush straw calc version zero = false 2023-05-24T20:58:10.892 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on no sortbitwise = false 2023-05-24T20:58:10.893 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on osd down out interval zero = false 2023-05-24T20:58:10.893 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on too few osds = false 2023-05-24T20:58:10.893 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-05-24T20:58:10.893 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_pool_no_redundancy = false 2023-05-24T20:58:10.893 INFO:teuthology.orchestra.run.smithi071.stdout: mon_allow_pool_size_one = true 2023-05-24T20:58:10.893 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:10.893 INFO:teuthology.orchestra.run.smithi071.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-05-24T20:58:10.893 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:10.894 INFO:teuthology.orchestra.run.smithi071.stdout: osd default data pool replay window = 5 2023-05-24T20:58:10.894 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:10.894 INFO:teuthology.orchestra.run.smithi071.stdout: mon allow pool delete = true 2023-05-24T20:58:10.894 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:10.894 INFO:teuthology.orchestra.run.smithi071.stdout: mon cluster log file level = debug 2023-05-24T20:58:10.894 INFO:teuthology.orchestra.run.smithi071.stdout: debug asserts on shutdown = true 2023-05-24T20:58:10.894 INFO:teuthology.orchestra.run.smithi071.stdout: mon health detail to clog = false 2023-05-24T20:58:10.894 INFO:teuthology.orchestra.run.smithi071.stdout: mon host = 172.21.15.71 2023-05-24T20:58:10.895 INFO:teuthology.orchestra.run.smithi071.stdout: fsid = 39018ae0-8be5-4c54-a564-aa238e12e56e 2023-05-24T20:58:10.895 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:10.895 INFO:teuthology.orchestra.run.smithi071.stdout:[osd] 2023-05-24T20:58:10.895 INFO:teuthology.orchestra.run.smithi071.stdout: osd journal size = 100 2023-05-24T20:58:10.895 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:10.895 INFO:teuthology.orchestra.run.smithi071.stdout: osd scrub load threshold = 5.0 2023-05-24T20:58:10.895 INFO:teuthology.orchestra.run.smithi071.stdout: osd scrub max interval = 600 2023-05-24T20:58:10.895 INFO:teuthology.orchestra.run.smithi071.stdout: osd mclock profile = high_recovery_ops 2023-05-24T20:58:10.896 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:10.896 INFO:teuthology.orchestra.run.smithi071.stdout: osd recover clone overlap = true 2023-05-24T20:58:10.896 INFO:teuthology.orchestra.run.smithi071.stdout: osd recovery max chunk = 1048576 2023-05-24T20:58:10.896 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:10.896 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug shutdown = true 2023-05-24T20:58:10.896 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug op order = true 2023-05-24T20:58:10.896 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug verify stray on activate = true 2023-05-24T20:58:10.897 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:10.897 INFO:teuthology.orchestra.run.smithi071.stdout: osd open classes on start = true 2023-05-24T20:58:10.897 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug pg log writeout = true 2023-05-24T20:58:10.897 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:10.897 INFO:teuthology.orchestra.run.smithi071.stdout: osd deep scrub update digest min age = 30 2023-05-24T20:58:10.897 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:10.897 INFO:teuthology.orchestra.run.smithi071.stdout: osd map max advance = 10 2023-05-24T20:58:10.897 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:10.898 INFO:teuthology.orchestra.run.smithi071.stdout: journal zero on create = true 2023-05-24T20:58:10.898 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:10.898 INFO:teuthology.orchestra.run.smithi071.stdout: filestore ondisk finisher threads = 3 2023-05-24T20:58:10.898 INFO:teuthology.orchestra.run.smithi071.stdout: filestore apply finisher threads = 3 2023-05-24T20:58:10.898 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:10.898 INFO:teuthology.orchestra.run.smithi071.stdout: bdev debug aio = true 2023-05-24T20:58:10.898 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug misdirected ops = true 2023-05-24T20:58:10.898 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2023-05-24T20:58:10.899 INFO:teuthology.orchestra.run.smithi071.stdout: debug osd = 20 2023-05-24T20:58:10.899 INFO:teuthology.orchestra.run.smithi071.stdout: rocksdb delete range threshold = 0 2023-05-24T20:58:10.899 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:10.899 INFO:teuthology.orchestra.run.smithi071.stdout:[mgr] 2023-05-24T20:58:10.899 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2023-05-24T20:58:10.899 INFO:teuthology.orchestra.run.smithi071.stdout: debug mgr = 20 2023-05-24T20:58:10.899 INFO:teuthology.orchestra.run.smithi071.stdout: debug mon = 20 2023-05-24T20:58:10.899 INFO:teuthology.orchestra.run.smithi071.stdout: debug auth = 20 2023-05-24T20:58:10.900 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min pgs per osd = 4 2023-05-24T20:58:10.900 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min bytes per osd = 10 2023-05-24T20:58:10.900 INFO:teuthology.orchestra.run.smithi071.stdout: mgr/telemetry/nag = false 2023-05-24T20:58:10.900 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:10.900 INFO:teuthology.orchestra.run.smithi071.stdout:[mon] 2023-05-24T20:58:10.900 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2023-05-24T20:58:10.900 INFO:teuthology.orchestra.run.smithi071.stdout: debug mon = 20 2023-05-24T20:58:10.900 INFO:teuthology.orchestra.run.smithi071.stdout: debug paxos = 20 2023-05-24T20:58:10.901 INFO:teuthology.orchestra.run.smithi071.stdout: debug auth = 20 2023-05-24T20:58:10.901 INFO:teuthology.orchestra.run.smithi071.stdout: mon data avail warn = 5 2023-05-24T20:58:10.901 INFO:teuthology.orchestra.run.smithi071.stdout: mon mgr mkfs grace = 240 2023-05-24T20:58:10.901 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min pgs per osd = 4 2023-05-24T20:58:10.901 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd reporter subtree level = osd 2023-05-24T20:58:10.901 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd prime pg temp = true 2023-05-24T20:58:10.901 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min bytes per osd = 10 2023-05-24T20:58:10.901 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:10.902 INFO:teuthology.orchestra.run.smithi071.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-05-24T20:58:10.902 INFO:teuthology.orchestra.run.smithi071.stdout: auth mon ticket ttl = 660 # 11m 2023-05-24T20:58:10.902 INFO:teuthology.orchestra.run.smithi071.stdout: auth service ticket ttl = 240 # 4m 2023-05-24T20:58:10.902 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:10.902 INFO:teuthology.orchestra.run.smithi071.stdout: # don't complain about insecure global_id in the test suite 2023-05-24T20:58:10.902 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-05-24T20:58:10.902 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-05-24T20:58:10.903 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:10.903 INFO:teuthology.orchestra.run.smithi071.stdout: # 1m isn't quite enough 2023-05-24T20:58:10.903 INFO:teuthology.orchestra.run.smithi071.stdout: mon_down_mkfs_grace = 2m 2023-05-24T20:58:10.903 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:10.903 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_filestore_osds = false 2023-05-24T20:58:10.903 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:10.903 INFO:teuthology.orchestra.run.smithi071.stdout:[client] 2023-05-24T20:58:10.903 INFO:teuthology.orchestra.run.smithi071.stdout: rgw cache enabled = true 2023-05-24T20:58:10.904 INFO:teuthology.orchestra.run.smithi071.stdout: rgw enable ops log = true 2023-05-24T20:58:10.904 INFO:teuthology.orchestra.run.smithi071.stdout: rgw enable usage log = true 2023-05-24T20:58:10.904 INFO:teuthology.orchestra.run.smithi071.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-05-24T20:58:10.904 INFO:teuthology.orchestra.run.smithi071.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-05-24T20:58:10.904 INFO:teuthology.orchestra.run.smithi071.stdout: debug rgw = 20 2023-05-24T20:58:10.904 INFO:teuthology.orchestra.run.smithi071.stdout: debug rgw sync = 20 2023-05-24T20:58:10.904 INFO:teuthology.orchestra.run.smithi071.stdout: rgw crypt require ssl = False 2023-05-24T20:58:10.904 INFO:teuthology.orchestra.run.smithi071.stdout: rgw crypt s3 kms backend = testing 2023-05-24T20:58:10.905 INFO:teuthology.orchestra.run.smithi071.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-05-24T20:58:10.905 INFO:teuthology.orchestra.run.smithi071.stdout: rgw curl low speed time = 300 2023-05-24T20:58:10.905 INFO:teuthology.orchestra.run.smithi071.stdout: rgw data log num shards = 4 2023-05-24T20:58:10.905 INFO:teuthology.orchestra.run.smithi071.stdout: rgw md log max shards = 4 2023-05-24T20:58:10.905 INFO:teuthology.orchestra.run.smithi071.stdout: rgw sync data inject err probability = 0.1 2023-05-24T20:58:10.905 INFO:teuthology.orchestra.run.smithi071.stdout: rgw sync log trim interval = 0 2023-05-24T20:58:10.905 INFO:teuthology.orchestra.run.smithi071.stdout: rgw sync meta inject err probability = 0.1 2023-05-24T20:58:10.905 INFO:teuthology.orchestra.run.smithi071.stdout: rgw sync obj etag verify = True 2023-05-24T20:58:10.906 INFO:teuthology.orchestra.run.smithi071.stdout: setgroup = ceph 2023-05-24T20:58:10.906 INFO:teuthology.orchestra.run.smithi071.stdout: setuser = ceph 2023-05-24T20:58:10.906 INFO:teuthology.orchestra.run.smithi071.stdout:[mon.a] 2023-05-24T20:58:10.906 INFO:teuthology.orchestra.run.smithi071.stdout:[client.0] 2023-05-24T20:58:10.906 INFO:teuthology.orchestra.run.smithi071.stdout: rgw data notify interval msec = 0 2023-05-24T20:58:10.906 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.0] 2023-05-24T20:58:10.906 INFO:teuthology.orchestra.run.smithi071.stdout: osd_max_omap_entries_per_request = 10 2023-05-24T20:58:10.906 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.1] 2023-05-24T20:58:10.907 INFO:teuthology.orchestra.run.smithi071.stdout: osd_max_omap_entries_per_request = 1000 2023-05-24T20:58:10.907 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.2] 2023-05-24T20:58:10.907 INFO:teuthology.orchestra.run.smithi071.stdout: osd_max_omap_entries_per_request = 10000 2023-05-24T20:58:10.908 INFO:teuthology.orchestra.run.smithi157.stdout:[global] 2023-05-24T20:58:10.908 INFO:teuthology.orchestra.run.smithi157.stdout: chdir = "" 2023-05-24T20:58:10.908 INFO:teuthology.orchestra.run.smithi157.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-05-24T20:58:10.908 INFO:teuthology.orchestra.run.smithi157.stdout: auth supported = cephx 2023-05-24T20:58:10.908 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:10.908 INFO:teuthology.orchestra.run.smithi157.stdout: filestore xattr use omap = true 2023-05-24T20:58:10.909 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:10.909 INFO:teuthology.orchestra.run.smithi157.stdout: mon clock drift allowed = 1.000 2023-05-24T20:58:10.909 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:10.909 INFO:teuthology.orchestra.run.smithi157.stdout: osd crush chooseleaf type = 0 2023-05-24T20:58:10.909 INFO:teuthology.orchestra.run.smithi157.stdout: auth debug = true 2023-05-24T20:58:10.909 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:10.909 INFO:teuthology.orchestra.run.smithi157.stdout: ms die on old message = true 2023-05-24T20:58:10.909 INFO:teuthology.orchestra.run.smithi157.stdout: ms die on bug = true 2023-05-24T20:58:10.910 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:10.910 INFO:teuthology.orchestra.run.smithi157.stdout: mon max pg per osd = 10000 # >= luminous 2023-05-24T20:58:10.910 INFO:teuthology.orchestra.run.smithi157.stdout: mon pg warn max object skew = 0 2023-05-24T20:58:10.910 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:10.910 INFO:teuthology.orchestra.run.smithi157.stdout: # disable pg_autoscaler by default for new pools 2023-05-24T20:58:10.910 INFO:teuthology.orchestra.run.smithi157.stdout: osd_pool_default_pg_autoscale_mode = off 2023-05-24T20:58:10.910 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:10.910 INFO:teuthology.orchestra.run.smithi157.stdout: osd pool default size = 2 2023-05-24T20:58:10.911 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:10.911 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd allow primary affinity = true 2023-05-24T20:58:10.911 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd allow pg remap = true 2023-05-24T20:58:10.911 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on legacy crush tunables = false 2023-05-24T20:58:10.911 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on crush straw calc version zero = false 2023-05-24T20:58:10.911 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on no sortbitwise = false 2023-05-24T20:58:10.911 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on osd down out interval zero = false 2023-05-24T20:58:10.911 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on too few osds = false 2023-05-24T20:58:10.912 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-05-24T20:58:10.912 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_pool_no_redundancy = false 2023-05-24T20:58:10.912 INFO:teuthology.orchestra.run.smithi157.stdout: mon_allow_pool_size_one = true 2023-05-24T20:58:10.912 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:10.912 INFO:teuthology.orchestra.run.smithi157.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-05-24T20:58:10.912 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:10.912 INFO:teuthology.orchestra.run.smithi157.stdout: osd default data pool replay window = 5 2023-05-24T20:58:10.913 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:10.913 INFO:teuthology.orchestra.run.smithi157.stdout: mon allow pool delete = true 2023-05-24T20:58:10.913 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:10.913 INFO:teuthology.orchestra.run.smithi157.stdout: mon cluster log file level = debug 2023-05-24T20:58:10.913 INFO:teuthology.orchestra.run.smithi157.stdout: debug asserts on shutdown = true 2023-05-24T20:58:10.913 INFO:teuthology.orchestra.run.smithi157.stdout: mon health detail to clog = false 2023-05-24T20:58:10.913 INFO:teuthology.orchestra.run.smithi157.stdout: mon host = 172.21.15.71 2023-05-24T20:58:10.913 INFO:teuthology.orchestra.run.smithi157.stdout: fsid = 39018ae0-8be5-4c54-a564-aa238e12e56e 2023-05-24T20:58:10.914 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:10.914 INFO:teuthology.orchestra.run.smithi157.stdout:[osd] 2023-05-24T20:58:10.914 INFO:teuthology.orchestra.run.smithi157.stdout: osd journal size = 100 2023-05-24T20:58:10.914 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:10.914 INFO:teuthology.orchestra.run.smithi157.stdout: osd scrub load threshold = 5.0 2023-05-24T20:58:10.914 INFO:teuthology.orchestra.run.smithi157.stdout: osd scrub max interval = 600 2023-05-24T20:58:10.914 INFO:teuthology.orchestra.run.smithi157.stdout: osd mclock profile = high_recovery_ops 2023-05-24T20:58:10.914 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:10.915 INFO:teuthology.orchestra.run.smithi157.stdout: osd recover clone overlap = true 2023-05-24T20:58:10.915 INFO:teuthology.orchestra.run.smithi157.stdout: osd recovery max chunk = 1048576 2023-05-24T20:58:10.915 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:10.915 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug shutdown = true 2023-05-24T20:58:10.915 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug op order = true 2023-05-24T20:58:10.915 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug verify stray on activate = true 2023-05-24T20:58:10.916 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:10.916 INFO:teuthology.orchestra.run.smithi157.stdout: osd open classes on start = true 2023-05-24T20:58:10.916 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug pg log writeout = true 2023-05-24T20:58:10.916 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:10.916 INFO:teuthology.orchestra.run.smithi157.stdout: osd deep scrub update digest min age = 30 2023-05-24T20:58:10.916 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:10.917 INFO:teuthology.orchestra.run.smithi157.stdout: osd map max advance = 10 2023-05-24T20:58:10.917 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:10.917 INFO:teuthology.orchestra.run.smithi157.stdout: journal zero on create = true 2023-05-24T20:58:10.917 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:10.917 INFO:teuthology.orchestra.run.smithi157.stdout: filestore ondisk finisher threads = 3 2023-05-24T20:58:10.917 INFO:teuthology.orchestra.run.smithi157.stdout: filestore apply finisher threads = 3 2023-05-24T20:58:10.917 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:10.918 INFO:teuthology.orchestra.run.smithi157.stdout: bdev debug aio = true 2023-05-24T20:58:10.918 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug misdirected ops = true 2023-05-24T20:58:10.918 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2023-05-24T20:58:10.918 INFO:teuthology.orchestra.run.smithi157.stdout: debug osd = 20 2023-05-24T20:58:10.918 INFO:teuthology.orchestra.run.smithi157.stdout: rocksdb delete range threshold = 0 2023-05-24T20:58:10.918 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:10.918 INFO:teuthology.orchestra.run.smithi157.stdout:[mgr] 2023-05-24T20:58:10.918 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2023-05-24T20:58:10.919 INFO:teuthology.orchestra.run.smithi157.stdout: debug mgr = 20 2023-05-24T20:58:10.919 INFO:teuthology.orchestra.run.smithi157.stdout: debug mon = 20 2023-05-24T20:58:10.919 INFO:teuthology.orchestra.run.smithi157.stdout: debug auth = 20 2023-05-24T20:58:10.919 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min pgs per osd = 4 2023-05-24T20:58:10.919 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min bytes per osd = 10 2023-05-24T20:58:10.919 INFO:teuthology.orchestra.run.smithi157.stdout: mgr/telemetry/nag = false 2023-05-24T20:58:10.919 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:10.919 INFO:teuthology.orchestra.run.smithi157.stdout:[mon] 2023-05-24T20:58:10.920 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2023-05-24T20:58:10.920 INFO:teuthology.orchestra.run.smithi157.stdout: debug mon = 20 2023-05-24T20:58:10.920 INFO:teuthology.orchestra.run.smithi157.stdout: debug paxos = 20 2023-05-24T20:58:10.920 INFO:teuthology.orchestra.run.smithi157.stdout: debug auth = 20 2023-05-24T20:58:10.920 INFO:teuthology.orchestra.run.smithi157.stdout: mon data avail warn = 5 2023-05-24T20:58:10.920 INFO:teuthology.orchestra.run.smithi157.stdout: mon mgr mkfs grace = 240 2023-05-24T20:58:10.920 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min pgs per osd = 4 2023-05-24T20:58:10.921 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd reporter subtree level = osd 2023-05-24T20:58:10.921 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd prime pg temp = true 2023-05-24T20:58:10.921 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min bytes per osd = 10 2023-05-24T20:58:10.921 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:10.921 INFO:teuthology.orchestra.run.smithi157.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-05-24T20:58:10.921 INFO:teuthology.orchestra.run.smithi157.stdout: auth mon ticket ttl = 660 # 11m 2023-05-24T20:58:10.922 INFO:teuthology.orchestra.run.smithi157.stdout: auth service ticket ttl = 240 # 4m 2023-05-24T20:58:10.922 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:10.922 INFO:teuthology.orchestra.run.smithi157.stdout: # don't complain about insecure global_id in the test suite 2023-05-24T20:58:10.922 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-05-24T20:58:10.922 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-05-24T20:58:10.922 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:10.922 INFO:teuthology.orchestra.run.smithi157.stdout: # 1m isn't quite enough 2023-05-24T20:58:10.922 INFO:teuthology.orchestra.run.smithi157.stdout: mon_down_mkfs_grace = 2m 2023-05-24T20:58:10.923 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:10.923 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_filestore_osds = false 2023-05-24T20:58:10.923 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:10.923 INFO:teuthology.orchestra.run.smithi157.stdout:[client] 2023-05-24T20:58:10.923 INFO:teuthology.orchestra.run.smithi157.stdout: rgw cache enabled = true 2023-05-24T20:58:10.924 INFO:teuthology.orchestra.run.smithi157.stdout: rgw enable ops log = true 2023-05-24T20:58:10.924 INFO:teuthology.orchestra.run.smithi157.stdout: rgw enable usage log = true 2023-05-24T20:58:10.924 INFO:teuthology.orchestra.run.smithi157.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-05-24T20:58:10.924 INFO:teuthology.orchestra.run.smithi157.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-05-24T20:58:10.924 INFO:teuthology.orchestra.run.smithi157.stdout: debug rgw = 20 2023-05-24T20:58:10.925 INFO:teuthology.orchestra.run.smithi157.stdout: debug rgw sync = 20 2023-05-24T20:58:10.925 INFO:teuthology.orchestra.run.smithi157.stdout: rgw crypt require ssl = False 2023-05-24T20:58:10.925 INFO:teuthology.orchestra.run.smithi157.stdout: rgw crypt s3 kms backend = testing 2023-05-24T20:58:10.925 INFO:teuthology.orchestra.run.smithi157.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-05-24T20:58:10.925 INFO:teuthology.orchestra.run.smithi157.stdout: rgw curl low speed time = 300 2023-05-24T20:58:10.925 INFO:teuthology.orchestra.run.smithi157.stdout: rgw data log num shards = 4 2023-05-24T20:58:10.925 INFO:teuthology.orchestra.run.smithi157.stdout: rgw md log max shards = 4 2023-05-24T20:58:10.925 INFO:teuthology.orchestra.run.smithi157.stdout: rgw sync data inject err probability = 0.1 2023-05-24T20:58:10.926 INFO:teuthology.orchestra.run.smithi157.stdout: rgw sync log trim interval = 0 2023-05-24T20:58:10.926 INFO:teuthology.orchestra.run.smithi157.stdout: rgw sync meta inject err probability = 0.1 2023-05-24T20:58:10.926 INFO:teuthology.orchestra.run.smithi157.stdout: rgw sync obj etag verify = True 2023-05-24T20:58:10.926 INFO:teuthology.orchestra.run.smithi157.stdout: setgroup = ceph 2023-05-24T20:58:10.926 INFO:teuthology.orchestra.run.smithi157.stdout: setuser = ceph 2023-05-24T20:58:10.926 INFO:teuthology.orchestra.run.smithi157.stdout:[mon.a] 2023-05-24T20:58:10.926 INFO:teuthology.orchestra.run.smithi157.stdout:[client.0] 2023-05-24T20:58:10.926 INFO:teuthology.orchestra.run.smithi157.stdout: rgw data notify interval msec = 0 2023-05-24T20:58:10.927 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.0] 2023-05-24T20:58:10.927 INFO:teuthology.orchestra.run.smithi157.stdout: osd_max_omap_entries_per_request = 10 2023-05-24T20:58:10.927 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.1] 2023-05-24T20:58:10.927 INFO:teuthology.orchestra.run.smithi157.stdout: osd_max_omap_entries_per_request = 1000 2023-05-24T20:58:10.927 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.2] 2023-05-24T20:58:10.927 INFO:teuthology.orchestra.run.smithi157.stdout: osd_max_omap_entries_per_request = 10000 2023-05-24T20:58:10.929 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2023-05-24T20:58:10.929 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2023-05-24T20:58:11.030 INFO:tasks.ceph:Copying monmap to all nodes... 2023-05-24T20:58:11.031 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-05-24T20:58:11.031 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2023-05-24T20:58:11.049 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-05-24T20:58:11.049 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2023-05-24T20:58:11.109 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi071.front.sepia.ceph.com 2023-05-24T20:58:11.110 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-05-24T20:58:11.111 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/ceph/c1.keyring 2023-05-24T20:58:11.111 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0644 /etc/ceph/c1.keyring 2023-05-24T20:58:11.199 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-05-24T20:58:11.199 DEBUG:teuthology.orchestra.run.smithi071:> dd of=/home/ubuntu/cephtest/c1.monmap 2023-05-24T20:58:11.264 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi157.front.sepia.ceph.com 2023-05-24T20:58:11.265 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-05-24T20:58:11.266 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/ceph/c1.keyring 2023-05-24T20:58:11.266 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0644 /etc/ceph/c1.keyring 2023-05-24T20:58:11.313 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-05-24T20:58:11.313 DEBUG:teuthology.orchestra.run.smithi157:> dd of=/home/ubuntu/cephtest/c1.monmap 2023-05-24T20:58:11.378 INFO:tasks.ceph:Setting up mon nodes... 2023-05-24T20:58:11.378 INFO:tasks.ceph:Setting up mgr nodes... 2023-05-24T20:58:11.379 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2023-05-24T20:58:11.433 INFO:teuthology.orchestra.run.smithi071.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2023-05-24T20:58:11.435 INFO:tasks.ceph:Setting up mds nodes... 2023-05-24T20:58:11.435 INFO:tasks.ceph_client:Setting up client nodes... 2023-05-24T20:58:11.436 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2023-05-24T20:58:11.482 INFO:teuthology.orchestra.run.smithi071.stdout:creating /etc/ceph/c1.client.0.keyring 2023-05-24T20:58:11.495 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2023-05-24T20:58:11.545 INFO:teuthology.orchestra.run.smithi071.stdout:creating /etc/ceph/c1.client.1.keyring 2023-05-24T20:58:11.556 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-05-24T20:58:11.557 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi071.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-05-24T20:58:11.557 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2023-05-24T20:58:11.630 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-05-24T20:58:11.631 INFO:tasks.ceph:role: c1.osd.0 2023-05-24T20:58:11.631 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi071.front.sepia.ceph.com 2023-05-24T20:58:11.631 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-05-24T20:58:11.707 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-05-24T20:58:11.708 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2023-05-24T20:58:11.708 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-05-24T20:58:11.708 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-05-24T20:58:11.708 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-05-24T20:58:11.708 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2023-05-24T20:58:11.708 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-05-24T20:58:11.709 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-05-24T20:58:11.709 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-05-24T20:58:11.709 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-05-24T20:58:12.139 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2023-05-24T20:58:12.140 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2023-05-24T20:58:12.141 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2023-05-24T20:58:12.323 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2023-05-24T20:58:12.358 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2023-05-24T20:58:12.425 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-05-24T20:58:12.425 INFO:tasks.ceph:role: c1.osd.1 2023-05-24T20:58:12.426 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi071.front.sepia.ceph.com 2023-05-24T20:58:12.426 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-05-24T20:58:12.501 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-05-24T20:58:12.502 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2023-05-24T20:58:12.502 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-05-24T20:58:12.502 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-05-24T20:58:12.502 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-05-24T20:58:12.502 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2023-05-24T20:58:12.502 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-05-24T20:58:12.503 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-05-24T20:58:12.503 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-05-24T20:58:12.503 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-05-24T20:58:12.920 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2023-05-24T20:58:12.922 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2023-05-24T20:58:12.923 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2023-05-24T20:58:13.033 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2023-05-24T20:58:13.065 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2023-05-24T20:58:13.134 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-05-24T20:58:13.134 INFO:tasks.ceph:role: c1.osd.2 2023-05-24T20:58:13.134 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi071.front.sepia.ceph.com 2023-05-24T20:58:13.135 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-05-24T20:58:13.208 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-05-24T20:58:13.209 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2023-05-24T20:58:13.209 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-05-24T20:58:13.209 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-05-24T20:58:13.209 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-05-24T20:58:13.209 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2023-05-24T20:58:13.209 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-05-24T20:58:13.209 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-05-24T20:58:13.210 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-05-24T20:58:13.210 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-05-24T20:58:13.650 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2023-05-24T20:58:13.651 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2023-05-24T20:58:13.652 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2023-05-24T20:58:13.802 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2023-05-24T20:58:13.836 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2023-05-24T20:58:13.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:58:13.927+0000 7f027e2b7640 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2023-05-24T20:58:13.934 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:58:13.928+0000 7f027e2b7640 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2023-05-24T20:58:13.934 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:58:13.928+0000 7f027e2b7640 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2023-05-24T20:58:13.935 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:58:13.928+0000 7f027e2b7640 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2023-05-24T20:58:13.935 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:58:13.928+0000 7f027e2b7640 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2023-05-24T20:58:13.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:58:13.936+0000 7f027e2b7640 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2023-05-24T20:58:16.027 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2023-05-24T20:58:16.061 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2023-05-24T20:58:16.154 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:58:16.147+0000 7f4f0f97f640 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2023-05-24T20:58:16.154 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:58:16.148+0000 7f4f0f97f640 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2023-05-24T20:58:16.154 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:58:16.148+0000 7f4f0f97f640 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2023-05-24T20:58:16.154 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:58:16.148+0000 7f4f0f97f640 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2023-05-24T20:58:16.155 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:58:16.148+0000 7f4f0f97f640 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2023-05-24T20:58:16.155 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:58:16.148+0000 7f4f0f97f640 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2023-05-24T20:58:18.235 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2023-05-24T20:58:18.267 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2023-05-24T20:58:18.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:58:18.355+0000 7f5e799df640 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2023-05-24T20:58:18.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:58:18.355+0000 7f5e799df640 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2023-05-24T20:58:18.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:58:18.355+0000 7f5e799df640 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2023-05-24T20:58:18.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:58:18.355+0000 7f5e799df640 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2023-05-24T20:58:18.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:58:18.355+0000 7f5e799df640 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2023-05-24T20:58:18.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:58:18.356+0000 7f5e799df640 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2023-05-24T20:58:20.441 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2023-05-24T20:58:20.473 INFO:tasks.ceph:Reading keys from all nodes... 2023-05-24T20:58:20.474 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-05-24T20:58:20.474 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2023-05-24T20:58:20.545 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-05-24T20:58:20.545 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2023-05-24T20:58:20.615 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-05-24T20:58:20.615 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2023-05-24T20:58:20.684 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-05-24T20:58:20.685 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2023-05-24T20:58:20.753 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-05-24T20:58:20.754 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2023-05-24T20:58:20.813 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-05-24T20:58:20.813 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2023-05-24T20:58:20.872 INFO:tasks.ceph:Adding keys to all mons... 2023-05-24T20:58:20.872 DEBUG:teuthology.orchestra.run.smithi071:> sudo tee -a /etc/ceph/c1.keyring 2023-05-24T20:58:20.938 INFO:teuthology.orchestra.run.smithi071.stdout:[mgr.x] 2023-05-24T20:58:20.939 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQBjem5k5+x/GRAAf7lRrR41mHUT8MAuHpT6nA== 2023-05-24T20:58:20.939 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.0] 2023-05-24T20:58:20.939 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQBlem5ksUxXNxAAIexBG8kNdXZx8B31+gt4Vg== 2023-05-24T20:58:20.939 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.1] 2023-05-24T20:58:20.939 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQBoem5k/1baCBAAehGPsqZD9QKUo/buxGqYqw== 2023-05-24T20:58:20.939 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.2] 2023-05-24T20:58:20.940 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQBqem5kfcY1FRAAmo2zn+lXYG7sKcUrs6GoYQ== 2023-05-24T20:58:20.940 INFO:teuthology.orchestra.run.smithi071.stdout:[client.0] 2023-05-24T20:58:20.940 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQBjem5kzot2HBAAwdYRHYqYyTu8xEVE5OjLgg== 2023-05-24T20:58:20.940 INFO:teuthology.orchestra.run.smithi071.stdout:[client.1] 2023-05-24T20:58:20.940 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQBjem5ktswjIBAA7nbveSpPaSqVPnQWX4mRBg== 2023-05-24T20:58:20.941 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-05-24T20:58:21.037 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-05-24T20:58:21.091 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-05-24T20:58:21.145 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-05-24T20:58:21.198 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-05-24T20:58:21.255 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-05-24T20:58:21.312 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-05-24T20:58:21.312 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2023-05-24T20:58:21.340 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2023-05-24T20:58:21.440 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2023-05-24T20:58:21.468 DEBUG:teuthology.orchestra.run.smithi071:> rm -- /home/ubuntu/cephtest/c1.monmap 2023-05-24T20:58:21.527 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2023-05-24T20:58:21.528 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2023-05-24T20:58:21.528 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2023-05-24T20:58:21.569 INFO:tasks.ceph.c1.mon.a:Started 2023-05-24T20:58:21.569 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2023-05-24T20:58:21.569 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2023-05-24T20:58:21.570 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2023-05-24T20:58:21.571 INFO:tasks.ceph.c1.mgr.x:Started 2023-05-24T20:58:21.572 INFO:tasks.ceph:Setting crush tunables to default 2023-05-24T20:58:21.572 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster c1 osd crush tunables default 2023-05-24T20:58:21.782 INFO:teuthology.orchestra.run.smithi071.stderr:adjusted tunables profile to default 2023-05-24T20:58:21.791 INFO:tasks.ceph:check_enable_crimson: False 2023-05-24T20:58:21.792 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2023-05-24T20:58:21.792 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-05-24T20:58:21.793 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2023-05-24T20:58:21.819 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-05-24T20:58:21.820 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2023-05-24T20:58:21.887 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-05-24T20:58:21.887 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2023-05-24T20:58:21.953 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster c1 osd new b60e4ed9-696b-4651-8355-687a32c8ec0e 0 2023-05-24T20:58:21.984 INFO:tasks.ceph.c1.mgr.x.smithi071.stderr:2023-05-24T20:58:21.978+0000 7fa508f55200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-24T20:58:22.173 INFO:tasks.ceph.c1.mgr.x.smithi071.stderr:2023-05-24T20:58:22.167+0000 7fa508f55200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-24T20:58:22.224 INFO:teuthology.orchestra.run.smithi071.stdout:0 2023-05-24T20:58:22.237 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster c1 osd new af9d3031-9be7-4a21-835d-bc93ccbb6a28 1 2023-05-24T20:58:22.480 INFO:teuthology.orchestra.run.smithi071.stdout:1 2023-05-24T20:58:22.492 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster c1 osd new 2f460fc4-d82c-4cc4-a9d8-e01dd0ba8eb1 2 2023-05-24T20:58:22.543 INFO:tasks.ceph.c1.mgr.x.smithi071.stderr:2023-05-24T20:58:22.537+0000 7fa508f55200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-24T20:58:22.705 INFO:teuthology.orchestra.run.smithi071.stdout:2 2023-05-24T20:58:22.717 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2023-05-24T20:58:22.718 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2023-05-24T20:58:22.720 INFO:tasks.ceph.c1.osd.0:Started 2023-05-24T20:58:22.720 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2023-05-24T20:58:22.721 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2023-05-24T20:58:22.722 INFO:tasks.ceph.c1.osd.1:Started 2023-05-24T20:58:22.723 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2023-05-24T20:58:22.723 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2023-05-24T20:58:22.725 INFO:tasks.ceph.c1.osd.2:Started 2023-05-24T20:58:22.725 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2023-05-24T20:58:22.936 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:22.936 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":5,"fsid":"39018ae0-8be5-4c54-a564-aa238e12e56e","created":"2023-05-24T20:58:21.687786+0000","modified":"2023-05-24T20:58:22.697341+0000","last_up_change":"0.000000","last_in_change":"2023-05-24T20:58:22.697341+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b60e4ed9-696b-4651-8355-687a32c8ec0e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"af9d3031-9be7-4a21-835d-bc93ccbb6a28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2f460fc4-d82c-4cc4-a9d8-e01dd0ba8eb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-24T20:58:22.948 INFO:tasks.ceph.ceph_manager.c1:[] 2023-05-24T20:58:22.949 INFO:tasks.ceph:Waiting for OSDs to come up 2023-05-24T20:58:23.072 INFO:tasks.ceph.c1.mgr.x.smithi071.stderr:2023-05-24T20:58:23.066+0000 7fa508f55200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-05-24T20:58:23.251 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2023-05-24T20:58:23.265 INFO:tasks.ceph.c1.mgr.x.smithi071.stderr:2023-05-24T20:58:23.259+0000 7fa508f55200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-05-24T20:58:23.328 INFO:tasks.ceph.c1.osd.0.smithi071.stderr:2023-05-24T20:58:23.322+0000 7f2d29e18640 -1 Falling back to public interface 2023-05-24T20:58:23.335 INFO:tasks.ceph.c1.osd.2.smithi071.stderr:2023-05-24T20:58:23.329+0000 7fb136567640 -1 Falling back to public interface 2023-05-24T20:58:23.347 INFO:tasks.ceph.c1.osd.1.smithi071.stderr:2023-05-24T20:58:23.342+0000 7fa0b92ff640 -1 Falling back to public interface 2023-05-24T20:58:23.447 INFO:teuthology.misc.health.smithi071.stdout: 2023-05-24T20:58:23.447 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":5,"fsid":"39018ae0-8be5-4c54-a564-aa238e12e56e","created":"2023-05-24T20:58:21.687786+0000","modified":"2023-05-24T20:58:22.697341+0000","last_up_change":"0.000000","last_in_change":"2023-05-24T20:58:22.697341+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b60e4ed9-696b-4651-8355-687a32c8ec0e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"af9d3031-9be7-4a21-835d-bc93ccbb6a28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2f460fc4-d82c-4cc4-a9d8-e01dd0ba8eb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-24T20:58:23.458 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-05-24T20:58:23.645 INFO:tasks.ceph.c1.mgr.x.smithi071.stderr:2023-05-24T20:58:23.639+0000 7fa508f55200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-24T20:58:24.314 INFO:tasks.ceph.c1.mgr.x.smithi071.stderr:2023-05-24T20:58:24.308+0000 7fa508f55200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-24T20:58:24.386 INFO:tasks.ceph.c1.mgr.x.smithi071.stderr:2023-05-24T20:58:24.380+0000 7fa508f55200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-24T20:58:24.520 INFO:tasks.ceph.c1.mgr.x.smithi071.stderr:2023-05-24T20:58:24.514+0000 7fa508f55200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-24T20:58:24.589 INFO:tasks.ceph.c1.mgr.x.smithi071.stderr:2023-05-24T20:58:24.583+0000 7fa508f55200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-24T20:58:24.769 INFO:tasks.ceph.c1.mgr.x.smithi071.stderr:2023-05-24T20:58:24.763+0000 7fa508f55200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-24T20:58:24.846 INFO:tasks.ceph.c1.mgr.x.smithi071.stderr:2023-05-24T20:58:24.840+0000 7fa508f55200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-24T20:58:24.923 INFO:tasks.ceph.c1.mgr.x.smithi071.stderr:2023-05-24T20:58:24.917+0000 7fa508f55200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-24T20:58:25.020 INFO:tasks.ceph.c1.mgr.x.smithi071.stderr:2023-05-24T20:58:25.014+0000 7fa508f55200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-24T20:58:25.199 INFO:tasks.ceph.c1.mgr.x.smithi071.stderr:2023-05-24T20:58:25.193+0000 7fa508f55200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-24T20:58:25.281 INFO:tasks.ceph.c1.mgr.x.smithi071.stderr:2023-05-24T20:58:25.275+0000 7fa508f55200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-24T20:58:25.569 INFO:tasks.ceph.c1.mgr.x.smithi071.stderr:2023-05-24T20:58:25.563+0000 7fa508f55200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-24T20:58:25.640 INFO:tasks.ceph.c1.mgr.x.smithi071.stderr:2023-05-24T20:58:25.634+0000 7fa508f55200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-24T20:58:26.023 INFO:tasks.ceph.c1.osd.0.smithi071.stderr:2023-05-24T20:58:26.018+0000 7f2d29e18640 -1 osd.0 0 log_to_monitors true 2023-05-24T20:58:26.057 INFO:tasks.ceph.c1.osd.2.smithi071.stderr:2023-05-24T20:58:26.052+0000 7fb136567640 -1 osd.2 0 log_to_monitors true 2023-05-24T20:58:26.079 INFO:tasks.ceph.c1.osd.1.smithi071.stderr:2023-05-24T20:58:26.073+0000 7fa0b92ff640 -1 osd.1 0 log_to_monitors true 2023-05-24T20:58:26.160 INFO:tasks.ceph.c1.mgr.x.smithi071.stderr:2023-05-24T20:58:26.154+0000 7fa508f55200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-24T20:58:26.472 INFO:tasks.ceph.c1.mgr.x.smithi071.stderr:2023-05-24T20:58:26.466+0000 7fa508f55200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-24T20:58:26.614 INFO:tasks.ceph.c1.mgr.x.smithi071.stderr:2023-05-24T20:58:26.608+0000 7fa508f55200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-24T20:58:26.968 INFO:tasks.ceph.c1.mgr.x.smithi071.stderr:2023-05-24T20:58:26.962+0000 7fa508f55200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-24T20:58:27.121 INFO:tasks.ceph.c1.mgr.x.smithi071.stderr:2023-05-24T20:58:27.115+0000 7fa508f55200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-24T20:58:27.293 INFO:tasks.ceph.c1.mgr.x.smithi071.stderr:2023-05-24T20:58:27.287+0000 7fa508f55200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-24T20:58:27.628 INFO:tasks.ceph.c1.mgr.x.smithi071.stderr:2023-05-24T20:58:27.622+0000 7fa508f55200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-24T20:58:27.945 INFO:tasks.ceph.c1.osd.0.smithi071.stderr:2023-05-24T20:58:27.939+0000 7f2d1e1b9700 -1 osd.0 0 waiting for initial osdmap 2023-05-24T20:58:27.957 INFO:tasks.ceph.c1.osd.0.smithi071.stderr:2023-05-24T20:58:27.951+0000 7f2d15d93700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-05-24T20:58:27.966 INFO:tasks.ceph.c1.osd.2.smithi071.stderr:2023-05-24T20:58:27.960+0000 7fb12a908700 -1 osd.2 0 waiting for initial osdmap 2023-05-24T20:58:27.975 INFO:tasks.ceph.c1.osd.2.smithi071.stderr:2023-05-24T20:58:27.969+0000 7fb1224e2700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-05-24T20:58:28.572 INFO:tasks.ceph.c1.osd.1.smithi071.stderr:2023-05-24T20:58:28.566+0000 7fa0ad6a0700 -1 osd.1 0 waiting for initial osdmap 2023-05-24T20:58:28.582 INFO:tasks.ceph.c1.osd.1.smithi071.stderr:2023-05-24T20:58:28.576+0000 7fa0a527a700 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-05-24T20:58:28.684 INFO:tasks.ceph.c1.mgr.x.smithi071.stderr:2023-05-24T20:58:28.679+0000 7fa4ca371700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-05-24T20:58:29.761 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2023-05-24T20:58:30.077 INFO:teuthology.misc.health.smithi071.stdout: 2023-05-24T20:58:30.078 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":9,"fsid":"39018ae0-8be5-4c54-a564-aa238e12e56e","created":"2023-05-24T20:58:21.687786+0000","modified":"2023-05-24T20:58:29.567883+0000","last_up_change":"2023-05-24T20:58:29.567883+0000","last_in_change":"2023-05-24T20:58:22.697341+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b60e4ed9-696b-4651-8355-687a32c8ec0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":104692},{"type":"v1","addr":"172.21.15.71:6801","nonce":104692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":104692},{"type":"v1","addr":"172.21.15.71:6803","nonce":104692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":104692},{"type":"v1","addr":"172.21.15.71:6807","nonce":104692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":104692},{"type":"v1","addr":"172.21.15.71:6805","nonce":104692}]},"public_addr":"172.21.15.71:6801/104692","cluster_addr":"172.21.15.71:6803/104692","heartbeat_back_addr":"172.21.15.71:6807/104692","heartbeat_front_addr":"172.21.15.71:6805/104692","state":["exists","up"]},{"osd":1,"uuid":"af9d3031-9be7-4a21-835d-bc93ccbb6a28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":104704},{"type":"v1","addr":"172.21.15.71:6817","nonce":104704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":104704},{"type":"v1","addr":"172.21.15.71:6819","nonce":104704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":104704},{"type":"v1","addr":"172.21.15.71:6823","nonce":104704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":104704},{"type":"v1","addr":"172.21.15.71:6821","nonce":104704}]},"public_addr":"172.21.15.71:6817/104704","cluster_addr":"172.21.15.71:6819/104704","heartbeat_back_addr":"172.21.15.71:6823/104704","heartbeat_front_addr":"172.21.15.71:6821/104704","state":["exists","up"]},{"osd":2,"uuid":"2f460fc4-d82c-4cc4-a9d8-e01dd0ba8eb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":104693},{"type":"v1","addr":"172.21.15.71:6809","nonce":104693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":104693},{"type":"v1","addr":"172.21.15.71:6811","nonce":104693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":104693},{"type":"v1","addr":"172.21.15.71:6815","nonce":104693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":104693},{"type":"v1","addr":"172.21.15.71:6813","nonce":104693}]},"public_addr":"172.21.15.71:6809/104693","cluster_addr":"172.21.15.71:6811/104693","heartbeat_back_addr":"172.21.15.71:6815/104693","heartbeat_front_addr":"172.21.15.71:6813/104693","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-24T20:58:27.027478+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-24T20:58:27.041491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-24T20:58:30.090 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-05-24T20:58:30.090 INFO:tasks.ceph:Creating RBD pool 2023-05-24T20:58:30.091 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster c1 osd pool create rbd 8 2023-05-24T20:58:30.583 INFO:teuthology.orchestra.run.smithi071.stderr:pool 'rbd' created 2023-05-24T20:58:30.594 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-05-24T20:58:31.588 INFO:teuthology.orchestra.run.smithi071.stderr:enabled application 'rbd' on pool 'rbd' 2023-05-24T20:58:31.603 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2023-05-24T20:58:31.604 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2023-05-24T20:58:31.604 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2023-05-24T20:58:31.605 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2023-05-24T20:58:31.605 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-05-24T20:58:32.016 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:32.029 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":4,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":104280},{"type":"v1","addr":"172.21.15.71:6825","nonce":104280}]},"active_addr":"172.21.15.71:6825/104280","active_change":"2023-05-24T20:58:27.636388+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.71:0","nonce":4126075082}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.71:0","nonce":793653148}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.71:0","nonce":3985579128}]}]} 2023-05-24T20:58:32.031 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2023-05-24T20:58:32.032 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2023-05-24T20:58:32.032 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2023-05-24T20:58:32.350 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:32.351 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":11,"fsid":"39018ae0-8be5-4c54-a564-aa238e12e56e","created":"2023-05-24T20:58:21.687786+0000","modified":"2023-05-24T20:58:31.571489+0000","last_up_change":"2023-05-24T20:58:29.567883+0000","last_in_change":"2023-05-24T20:58:22.697341+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-24T20:58:29.717376+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-24T20:58:30.420731+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b60e4ed9-696b-4651-8355-687a32c8ec0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":104692},{"type":"v1","addr":"172.21.15.71:6801","nonce":104692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":104692},{"type":"v1","addr":"172.21.15.71:6803","nonce":104692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":104692},{"type":"v1","addr":"172.21.15.71:6807","nonce":104692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":104692},{"type":"v1","addr":"172.21.15.71:6805","nonce":104692}]},"public_addr":"172.21.15.71:6801/104692","cluster_addr":"172.21.15.71:6803/104692","heartbeat_back_addr":"172.21.15.71:6807/104692","heartbeat_front_addr":"172.21.15.71:6805/104692","state":["exists","up"]},{"osd":1,"uuid":"af9d3031-9be7-4a21-835d-bc93ccbb6a28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":104704},{"type":"v1","addr":"172.21.15.71:6817","nonce":104704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":104704},{"type":"v1","addr":"172.21.15.71:6819","nonce":104704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":104704},{"type":"v1","addr":"172.21.15.71:6823","nonce":104704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":104704},{"type":"v1","addr":"172.21.15.71:6821","nonce":104704}]},"public_addr":"172.21.15.71:6817/104704","cluster_addr":"172.21.15.71:6819/104704","heartbeat_back_addr":"172.21.15.71:6823/104704","heartbeat_front_addr":"172.21.15.71:6821/104704","state":["exists","up"]},{"osd":2,"uuid":"2f460fc4-d82c-4cc4-a9d8-e01dd0ba8eb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":104693},{"type":"v1","addr":"172.21.15.71:6809","nonce":104693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":104693},{"type":"v1","addr":"172.21.15.71:6811","nonce":104693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":104693},{"type":"v1","addr":"172.21.15.71:6815","nonce":104693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":104693},{"type":"v1","addr":"172.21.15.71:6813","nonce":104693}]},"public_addr":"172.21.15.71:6809/104693","cluster_addr":"172.21.15.71:6811/104693","heartbeat_back_addr":"172.21.15.71:6815/104693","heartbeat_front_addr":"172.21.15.71:6813/104693","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-24T20:58:27.027478+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-24T20:58:27.038967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-24T20:58:27.041491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-24T20:58:32.368 INFO:tasks.ceph.ceph_manager.c1:all up! 2023-05-24T20:58:32.369 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2023-05-24T20:58:32.708 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:32.708 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":12,"fsid":"39018ae0-8be5-4c54-a564-aa238e12e56e","created":"2023-05-24T20:58:21.687786+0000","modified":"2023-05-24T20:58:32.575640+0000","last_up_change":"2023-05-24T20:58:29.567883+0000","last_in_change":"2023-05-24T20:58:22.697341+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-24T20:58:29.717376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-24T20:58:30.420731+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b60e4ed9-696b-4651-8355-687a32c8ec0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":104692},{"type":"v1","addr":"172.21.15.71:6801","nonce":104692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":104692},{"type":"v1","addr":"172.21.15.71:6803","nonce":104692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":104692},{"type":"v1","addr":"172.21.15.71:6807","nonce":104692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":104692},{"type":"v1","addr":"172.21.15.71:6805","nonce":104692}]},"public_addr":"172.21.15.71:6801/104692","cluster_addr":"172.21.15.71:6803/104692","heartbeat_back_addr":"172.21.15.71:6807/104692","heartbeat_front_addr":"172.21.15.71:6805/104692","state":["exists","up"]},{"osd":1,"uuid":"af9d3031-9be7-4a21-835d-bc93ccbb6a28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":104704},{"type":"v1","addr":"172.21.15.71:6817","nonce":104704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":104704},{"type":"v1","addr":"172.21.15.71:6819","nonce":104704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":104704},{"type":"v1","addr":"172.21.15.71:6823","nonce":104704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":104704},{"type":"v1","addr":"172.21.15.71:6821","nonce":104704}]},"public_addr":"172.21.15.71:6817/104704","cluster_addr":"172.21.15.71:6819/104704","heartbeat_back_addr":"172.21.15.71:6823/104704","heartbeat_front_addr":"172.21.15.71:6821/104704","state":["exists","up"]},{"osd":2,"uuid":"2f460fc4-d82c-4cc4-a9d8-e01dd0ba8eb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":104693},{"type":"v1","addr":"172.21.15.71:6809","nonce":104693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":104693},{"type":"v1","addr":"172.21.15.71:6811","nonce":104693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":104693},{"type":"v1","addr":"172.21.15.71:6815","nonce":104693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":104693},{"type":"v1","addr":"172.21.15.71:6813","nonce":104693}]},"public_addr":"172.21.15.71:6809/104693","cluster_addr":"172.21.15.71:6811/104693","heartbeat_back_addr":"172.21.15.71:6815/104693","heartbeat_front_addr":"172.21.15.71:6813/104693","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-24T20:58:27.027478+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-24T20:58:27.038967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-24T20:58:27.041491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-24T20:58:32.721 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2023-05-24T20:58:32.722 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2023-05-24T20:58:32.722 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2023-05-24T20:58:32.889 INFO:teuthology.orchestra.run.smithi071.stdout:34359738370 2023-05-24T20:58:32.890 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2023-05-24T20:58:32.890 INFO:teuthology.orchestra.run.smithi071.stdout:38654705667 2023-05-24T20:58:32.891 INFO:teuthology.orchestra.run.smithi071.stdout:34359738370 2023-05-24T20:58:32.891 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2023-05-24T20:58:32.892 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2023-05-24T20:58:33.229 INFO:teuthology.orchestra.run.smithi071.stdout:38654705666 2023-05-24T20:58:33.230 INFO:teuthology.orchestra.run.smithi071.stdout:34359738369 2023-05-24T20:58:33.237 INFO:teuthology.orchestra.run.smithi071.stdout:34359738369 2023-05-24T20:58:33.245 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705667 got 38654705666 for osd.1 2023-05-24T20:58:33.246 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738369 for osd.0 2023-05-24T20:58:33.252 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738369 for osd.2 2023-05-24T20:58:34.247 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2023-05-24T20:58:34.248 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2023-05-24T20:58:34.253 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2023-05-24T20:58:34.622 INFO:teuthology.orchestra.run.smithi071.stdout:38654705667 2023-05-24T20:58:34.631 INFO:teuthology.orchestra.run.smithi071.stdout:34359738370 2023-05-24T20:58:34.636 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705667 got 38654705667 for osd.1 2023-05-24T20:58:34.636 DEBUG:teuthology.parallel:result is None 2023-05-24T20:58:34.639 INFO:teuthology.orchestra.run.smithi071.stdout:34359738370 2023-05-24T20:58:34.646 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738370 for osd.0 2023-05-24T20:58:34.646 DEBUG:teuthology.parallel:result is None 2023-05-24T20:58:34.653 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738370 for osd.2 2023-05-24T20:58:34.653 DEBUG:teuthology.parallel:result is None 2023-05-24T20:58:34.653 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2023-05-24T20:58:34.654 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2023-05-24T20:58:35.006 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:35.008 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-05-24T20:58:35.019 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2023-05-24T20:58:33.677063+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":25652,"kb_used_data":1448,"kb_used_omap":0,"kb_used_meta":23488,"kb_avail":314547148,"statfs":{"total":322122547200,"available":322096279552,"internally_reserved":0,"allocated":1482752,"data_stored":1256546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":24051712},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-24T20:58:31.580576+0000","last_change":"2023-05-24T20:58:31.579248+0000","last_active":"2023-05-24T20:58:31.580576+0000","last_peered":"2023-05-24T20:58:31.580576+0000","last_clean":"2023-05-24T20:58:31.580576+0000","last_became_active":"2023-05-24T20:58:31.579033+0000","last_became_peered":"2023-05-24T20:58:31.579033+0000","last_unstale":"2023-05-24T20:58:31.580576+0000","last_undegraded":"2023-05-24T20:58:31.580576+0000","last_fullsized":"2023-05-24T20:58:31.580576+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-24T20:58:30.569255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-24T20:58:30.569255+0000","last_clean_scrub_stamp":"2023-05-24T20:58:30.569255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-26T00:08:40.469794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-24T20:58:31.580881+0000","last_change":"2023-05-24T20:58:31.579123+0000","last_active":"2023-05-24T20:58:31.580881+0000","last_peered":"2023-05-24T20:58:31.580881+0000","last_clean":"2023-05-24T20:58:31.580881+0000","last_became_active":"2023-05-24T20:58:31.578920+0000","last_became_peered":"2023-05-24T20:58:31.578920+0000","last_unstale":"2023-05-24T20:58:31.580881+0000","last_undegraded":"2023-05-24T20:58:31.580881+0000","last_fullsized":"2023-05-24T20:58:31.580881+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-24T20:58:30.569255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-24T20:58:30.569255+0000","last_clean_scrub_stamp":"2023-05-24T20:58:30.569255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-26T01:44:04.083190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-24T20:58:31.581631+0000","last_change":"2023-05-24T20:58:31.580187+0000","last_active":"2023-05-24T20:58:31.581631+0000","last_peered":"2023-05-24T20:58:31.581631+0000","last_clean":"2023-05-24T20:58:31.581631+0000","last_became_active":"2023-05-24T20:58:31.579808+0000","last_became_peered":"2023-05-24T20:58:31.579808+0000","last_unstale":"2023-05-24T20:58:31.581631+0000","last_undegraded":"2023-05-24T20:58:31.581631+0000","last_fullsized":"2023-05-24T20:58:31.581631+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-24T20:58:30.569255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-24T20:58:30.569255+0000","last_clean_scrub_stamp":"2023-05-24T20:58:30.569255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-26T01:58:46.582769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-05-24T20:58:32.579270+0000","last_change":"2023-05-24T20:58:31.581014+0000","last_active":"2023-05-24T20:58:32.579270+0000","last_peered":"2023-05-24T20:58:32.579270+0000","last_clean":"2023-05-24T20:58:32.579270+0000","last_became_active":"2023-05-24T20:58:31.580128+0000","last_became_peered":"2023-05-24T20:58:31.580128+0000","last_unstale":"2023-05-24T20:58:32.579270+0000","last_undegraded":"2023-05-24T20:58:32.579270+0000","last_fullsized":"2023-05-24T20:58:32.579270+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-24T20:58:30.569255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-24T20:58:30.569255+0000","last_clean_scrub_stamp":"2023-05-24T20:58:30.569255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-26T04:14:56.649180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":136,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-24T20:58:32.001428+0000","last_change":"2023-05-24T20:58:31.580068+0000","last_active":"2023-05-24T20:58:32.001428+0000","last_peered":"2023-05-24T20:58:32.001428+0000","last_clean":"2023-05-24T20:58:32.001428+0000","last_became_active":"2023-05-24T20:58:31.579803+0000","last_became_peered":"2023-05-24T20:58:31.579803+0000","last_unstale":"2023-05-24T20:58:32.001428+0000","last_undegraded":"2023-05-24T20:58:32.001428+0000","last_fullsized":"2023-05-24T20:58:32.001428+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-24T20:58:30.569255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-24T20:58:30.569255+0000","last_clean_scrub_stamp":"2023-05-24T20:58:30.569255+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-25T23:58:15.319735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-24T20:58:31.581617+0000","last_change":"2023-05-24T20:58:31.580148+0000","last_active":"2023-05-24T20:58:31.581617+0000","last_peered":"2023-05-24T20:58:31.581617+0000","last_clean":"2023-05-24T20:58:31.581617+0000","last_became_active":"2023-05-24T20:58:31.579883+0000","last_became_peered":"2023-05-24T20:58:31.579883+0000","last_unstale":"2023-05-24T20:58:31.581617+0000","last_undegraded":"2023-05-24T20:58:31.581617+0000","last_fullsized":"2023-05-24T20:58:31.581617+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-24T20:58:30.569255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-24T20:58:30.569255+0000","last_clean_scrub_stamp":"2023-05-24T20:58:30.569255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-25T23:11:32.413191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-05-24T20:58:32.579279+0000","last_change":"2023-05-24T20:58:31.580882+0000","last_active":"2023-05-24T20:58:32.579279+0000","last_peered":"2023-05-24T20:58:32.579279+0000","last_clean":"2023-05-24T20:58:32.579279+0000","last_became_active":"2023-05-24T20:58:31.580111+0000","last_became_peered":"2023-05-24T20:58:31.580111+0000","last_unstale":"2023-05-24T20:58:32.579279+0000","last_undegraded":"2023-05-24T20:58:32.579279+0000","last_fullsized":"2023-05-24T20:58:32.579279+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-24T20:58:30.569255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-24T20:58:30.569255+0000","last_clean_scrub_stamp":"2023-05-24T20:58:30.569255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-26T06:28:38.040819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-24T20:58:31.582294+0000","last_change":"2023-05-24T20:58:31.580742+0000","last_active":"2023-05-24T20:58:31.582294+0000","last_peered":"2023-05-24T20:58:31.582294+0000","last_clean":"2023-05-24T20:58:31.582294+0000","last_became_active":"2023-05-24T20:58:31.580450+0000","last_became_peered":"2023-05-24T20:58:31.580450+0000","last_unstale":"2023-05-24T20:58:31.582294+0000","last_undegraded":"2023-05-24T20:58:31.582294+0000","last_fullsized":"2023-05-24T20:58:31.582294+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-24T20:58:30.569255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-24T20:58:30.569255+0000","last_clean_scrub_stamp":"2023-05-24T20:58:30.569255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-26T06:28:38.040819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-24T20:58:31.580510+0000","last_change":"2023-05-24T20:58:31.579172+0000","last_active":"2023-05-24T20:58:31.580510+0000","last_peered":"2023-05-24T20:58:31.580510+0000","last_clean":"2023-05-24T20:58:31.580510+0000","last_became_active":"2023-05-24T20:58:31.578902+0000","last_became_peered":"2023-05-24T20:58:31.578902+0000","last_unstale":"2023-05-24T20:58:31.580510+0000","last_undegraded":"2023-05-24T20:58:31.580510+0000","last_fullsized":"2023-05-24T20:58:31.580510+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-24T20:58:30.569255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-24T20:58:30.569255+0000","last_clean_scrub_stamp":"2023-05-24T20:58:30.569255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-26T08:27:23.095939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705667,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8872,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":8000,"kb_avail":104848728,"statfs":{"total":107374182400,"available":107365097472,"internally_reserved":0,"allocated":692224,"data_stored":615638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8192000},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27200000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":9124,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848476,"statfs":{"total":107374182400,"available":107364839424,"internally_reserved":0,"allocated":692224,"data_stored":615638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]}]},{"osd":2,"up_from":8,"seq":34359738370,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7656,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":7552,"kb_avail":104849944,"statfs":{"total":107374182400,"available":107366342656,"internally_reserved":0,"allocated":98304,"data_stored":25270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7733248},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.215}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.255}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-05-24T20:58:35.022 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2023-05-24T20:58:35.331 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:35.333 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-05-24T20:58:35.344 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2023-05-24T20:58:33.677063+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":25652,"kb_used_data":1448,"kb_used_omap":0,"kb_used_meta":23488,"kb_avail":314547148,"statfs":{"total":322122547200,"available":322096279552,"internally_reserved":0,"allocated":1482752,"data_stored":1256546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":24051712},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-24T20:58:31.580576+0000","last_change":"2023-05-24T20:58:31.579248+0000","last_active":"2023-05-24T20:58:31.580576+0000","last_peered":"2023-05-24T20:58:31.580576+0000","last_clean":"2023-05-24T20:58:31.580576+0000","last_became_active":"2023-05-24T20:58:31.579033+0000","last_became_peered":"2023-05-24T20:58:31.579033+0000","last_unstale":"2023-05-24T20:58:31.580576+0000","last_undegraded":"2023-05-24T20:58:31.580576+0000","last_fullsized":"2023-05-24T20:58:31.580576+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-24T20:58:30.569255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-24T20:58:30.569255+0000","last_clean_scrub_stamp":"2023-05-24T20:58:30.569255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-26T00:08:40.469794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-24T20:58:31.580881+0000","last_change":"2023-05-24T20:58:31.579123+0000","last_active":"2023-05-24T20:58:31.580881+0000","last_peered":"2023-05-24T20:58:31.580881+0000","last_clean":"2023-05-24T20:58:31.580881+0000","last_became_active":"2023-05-24T20:58:31.578920+0000","last_became_peered":"2023-05-24T20:58:31.578920+0000","last_unstale":"2023-05-24T20:58:31.580881+0000","last_undegraded":"2023-05-24T20:58:31.580881+0000","last_fullsized":"2023-05-24T20:58:31.580881+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-24T20:58:30.569255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-24T20:58:30.569255+0000","last_clean_scrub_stamp":"2023-05-24T20:58:30.569255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-26T01:44:04.083190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-24T20:58:31.581631+0000","last_change":"2023-05-24T20:58:31.580187+0000","last_active":"2023-05-24T20:58:31.581631+0000","last_peered":"2023-05-24T20:58:31.581631+0000","last_clean":"2023-05-24T20:58:31.581631+0000","last_became_active":"2023-05-24T20:58:31.579808+0000","last_became_peered":"2023-05-24T20:58:31.579808+0000","last_unstale":"2023-05-24T20:58:31.581631+0000","last_undegraded":"2023-05-24T20:58:31.581631+0000","last_fullsized":"2023-05-24T20:58:31.581631+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-24T20:58:30.569255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-24T20:58:30.569255+0000","last_clean_scrub_stamp":"2023-05-24T20:58:30.569255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-26T01:58:46.582769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-05-24T20:58:32.579270+0000","last_change":"2023-05-24T20:58:31.581014+0000","last_active":"2023-05-24T20:58:32.579270+0000","last_peered":"2023-05-24T20:58:32.579270+0000","last_clean":"2023-05-24T20:58:32.579270+0000","last_became_active":"2023-05-24T20:58:31.580128+0000","last_became_peered":"2023-05-24T20:58:31.580128+0000","last_unstale":"2023-05-24T20:58:32.579270+0000","last_undegraded":"2023-05-24T20:58:32.579270+0000","last_fullsized":"2023-05-24T20:58:32.579270+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-24T20:58:30.569255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-24T20:58:30.569255+0000","last_clean_scrub_stamp":"2023-05-24T20:58:30.569255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-26T04:14:56.649180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":136,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-24T20:58:32.001428+0000","last_change":"2023-05-24T20:58:31.580068+0000","last_active":"2023-05-24T20:58:32.001428+0000","last_peered":"2023-05-24T20:58:32.001428+0000","last_clean":"2023-05-24T20:58:32.001428+0000","last_became_active":"2023-05-24T20:58:31.579803+0000","last_became_peered":"2023-05-24T20:58:31.579803+0000","last_unstale":"2023-05-24T20:58:32.001428+0000","last_undegraded":"2023-05-24T20:58:32.001428+0000","last_fullsized":"2023-05-24T20:58:32.001428+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-24T20:58:30.569255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-24T20:58:30.569255+0000","last_clean_scrub_stamp":"2023-05-24T20:58:30.569255+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-25T23:58:15.319735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-24T20:58:31.581617+0000","last_change":"2023-05-24T20:58:31.580148+0000","last_active":"2023-05-24T20:58:31.581617+0000","last_peered":"2023-05-24T20:58:31.581617+0000","last_clean":"2023-05-24T20:58:31.581617+0000","last_became_active":"2023-05-24T20:58:31.579883+0000","last_became_peered":"2023-05-24T20:58:31.579883+0000","last_unstale":"2023-05-24T20:58:31.581617+0000","last_undegraded":"2023-05-24T20:58:31.581617+0000","last_fullsized":"2023-05-24T20:58:31.581617+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-24T20:58:30.569255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-24T20:58:30.569255+0000","last_clean_scrub_stamp":"2023-05-24T20:58:30.569255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-25T23:11:32.413191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-05-24T20:58:32.579279+0000","last_change":"2023-05-24T20:58:31.580882+0000","last_active":"2023-05-24T20:58:32.579279+0000","last_peered":"2023-05-24T20:58:32.579279+0000","last_clean":"2023-05-24T20:58:32.579279+0000","last_became_active":"2023-05-24T20:58:31.580111+0000","last_became_peered":"2023-05-24T20:58:31.580111+0000","last_unstale":"2023-05-24T20:58:32.579279+0000","last_undegraded":"2023-05-24T20:58:32.579279+0000","last_fullsized":"2023-05-24T20:58:32.579279+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-24T20:58:30.569255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-24T20:58:30.569255+0000","last_clean_scrub_stamp":"2023-05-24T20:58:30.569255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-26T06:28:38.040819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-24T20:58:31.582294+0000","last_change":"2023-05-24T20:58:31.580742+0000","last_active":"2023-05-24T20:58:31.582294+0000","last_peered":"2023-05-24T20:58:31.582294+0000","last_clean":"2023-05-24T20:58:31.582294+0000","last_became_active":"2023-05-24T20:58:31.580450+0000","last_became_peered":"2023-05-24T20:58:31.580450+0000","last_unstale":"2023-05-24T20:58:31.582294+0000","last_undegraded":"2023-05-24T20:58:31.582294+0000","last_fullsized":"2023-05-24T20:58:31.582294+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-24T20:58:30.569255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-24T20:58:30.569255+0000","last_clean_scrub_stamp":"2023-05-24T20:58:30.569255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-26T06:28:38.040819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-24T20:58:31.580510+0000","last_change":"2023-05-24T20:58:31.579172+0000","last_active":"2023-05-24T20:58:31.580510+0000","last_peered":"2023-05-24T20:58:31.580510+0000","last_clean":"2023-05-24T20:58:31.580510+0000","last_became_active":"2023-05-24T20:58:31.578902+0000","last_became_peered":"2023-05-24T20:58:31.578902+0000","last_unstale":"2023-05-24T20:58:31.580510+0000","last_undegraded":"2023-05-24T20:58:31.580510+0000","last_fullsized":"2023-05-24T20:58:31.580510+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-24T20:58:30.569255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-24T20:58:30.569255+0000","last_clean_scrub_stamp":"2023-05-24T20:58:30.569255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-26T08:27:23.095939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705667,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8872,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":8000,"kb_avail":104848728,"statfs":{"total":107374182400,"available":107365097472,"internally_reserved":0,"allocated":692224,"data_stored":615638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8192000},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27200000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":9124,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848476,"statfs":{"total":107374182400,"available":107364839424,"internally_reserved":0,"allocated":692224,"data_stored":615638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]}]},{"osd":2,"up_from":8,"seq":34359738370,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7656,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":7552,"kb_avail":104849944,"statfs":{"total":107374182400,"available":107366342656,"internally_reserved":0,"allocated":98304,"data_stored":25270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7733248},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.215}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.255}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-05-24T20:58:35.345 INFO:tasks.ceph.ceph_manager.c1:clean! 2023-05-24T20:58:35.345 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2023-05-24T20:58:35.345 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2023-05-24T20:58:35.346 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2023-05-24T20:58:35.700 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:35.701 INFO:teuthology.orchestra.run.smithi071.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-05-24T20:58:35.714 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2023-05-24T20:58:35.714 INFO:teuthology.run_tasks:Running task ceph... 2023-05-24T20:58:35.725 INFO:tasks.ceph:Creating ceph cluster c2... 2023-05-24T20:58:35.726 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-05-24T20:58:35.726 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-05-24_20:32:11-rgw-wip-yuri-testing-2023-05-24-0738-reef-distro-default-smithi/7285505', 'branch': 'wip-yuri-testing-2023-05-24-0738-reef', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8} tasks/test_multi}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7285505', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-05-24_20:32:11-rgw-wip-yuri-testing-2023-05-24-0738-reef-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri-testing-2023-05-24-0738-reef'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '3395903b291ea37c38b39a05e6d9aa4f40306c80', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '3395903b291ea37c38b39a05e6d9aa4f40306c80'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a1'}, {'endpoints': ['c1.client.1'], 'name': 'a2'}]}, {'enabled_features': ['resharding'], 'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'b', 'zones': [{'endpoints': ['c2.client.0'], 'is_default': True, 'is_master': True, 'name': 'b1'}, {'endpoints': ['c2.client.1'], 'name': 'b2'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-yuri-testing-2023-05-24-0738-reef', 'sha1': '3395903b291ea37c38b39a05e6d9aa4f40306c80'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'seed': 8560, 'sha1': '3395903b291ea37c38b39a05e6d9aa4f40306c80', 'sleep_before_teardown': 0, 'subset': '111/12000', 'suite': 'rgw', 'suite_branch': 'wip-yuri-testing-2023-05-24-0738-reef', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '3395903b291ea37c38b39a05e6d9aa4f40306c80', 'targets': {'smithi071.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA5CM/4hsfiqXE11jZAHgpOo2FbW4ZOzUXKXNJNz9fWWeDtJTYStjEV6efAd9vMM91OzJdMX3iMjxJTS8Y5i2uY=', 'smithi157.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO+lmSy90RyAYeFfSD6aXgxRUJPCkLZkHkQxsxqLXA3D4XXntKSNvOhxkzpRSCW+Uo3pl5PdjBfPz+AOeUYtCwU='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '3395903b291ea37c38b39a05e6d9aa4f40306c80', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '3395903b291ea37c38b39a05e6d9aa4f40306c80', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6fb4f0c910de3902d27909a642e8d50d731dd28c', 'timestamp': '2023-05-24_20:32:11', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.23459'} 2023-05-24T20:58:35.727 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2023-05-24T20:58:35.729 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2023-05-24T20:58:35.751 DEBUG:teuthology.orchestra.run.smithi071:> sudo install -d -m0777 -- /var/run/ceph 2023-05-24T20:58:35.787 DEBUG:teuthology.orchestra.run.smithi157:> sudo install -d -m0777 -- /var/run/ceph 2023-05-24T20:58:35.824 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-05-24T20:58:35.825 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/scratch_devs of=/dev/stdout 2023-05-24T20:58:35.886 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-05-24T20:58:35.886 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_1 2023-05-24T20:58:35.946 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-05-24T20:58:35.946 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-05-24T20:58:35.946 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 6h/6d Inode: 363375 Links: 1 2023-05-24T20:58:35.947 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-05-24T20:58:35.947 INFO:teuthology.orchestra.run.smithi157.stdout:Context: system_u:object_r:device_t:s0 2023-05-24T20:58:35.947 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2023-05-24 20:55:57.512272105 +0000 2023-05-24T20:58:35.947 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2023-05-24 20:52:20.086598397 +0000 2023-05-24T20:58:35.948 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2023-05-24 20:52:20.086598397 +0000 2023-05-24T20:58:35.948 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: - 2023-05-24T20:58:35.948 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-05-24T20:58:36.013 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2023-05-24T20:58:36.014 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2023-05-24T20:58:36.014 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 0.000127177 s, 4.0 MB/s 2023-05-24T20:58:36.015 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-05-24T20:58:36.074 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_2 2023-05-24T20:58:36.135 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-05-24T20:58:36.135 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-05-24T20:58:36.136 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 6h/6d Inode: 372765 Links: 1 2023-05-24T20:58:36.136 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-05-24T20:58:36.136 INFO:teuthology.orchestra.run.smithi157.stdout:Context: system_u:object_r:device_t:s0 2023-05-24T20:58:36.136 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2023-05-24 20:55:57.512272105 +0000 2023-05-24T20:58:36.137 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2023-05-24 20:52:20.597588706 +0000 2023-05-24T20:58:36.137 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2023-05-24 20:52:20.597588706 +0000 2023-05-24T20:58:36.137 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: - 2023-05-24T20:58:36.138 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-05-24T20:58:36.202 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2023-05-24T20:58:36.203 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2023-05-24T20:58:36.203 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 0.000120177 s, 4.3 MB/s 2023-05-24T20:58:36.204 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-05-24T20:58:36.263 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_3 2023-05-24T20:58:36.322 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-05-24T20:58:36.323 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-05-24T20:58:36.323 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 6h/6d Inode: 368021 Links: 1 2023-05-24T20:58:36.323 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-05-24T20:58:36.323 INFO:teuthology.orchestra.run.smithi157.stdout:Context: system_u:object_r:device_t:s0 2023-05-24T20:58:36.323 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2023-05-24 20:55:57.512272105 +0000 2023-05-24T20:58:36.324 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2023-05-24 20:52:21.145578312 +0000 2023-05-24T20:58:36.324 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2023-05-24 20:52:21.145578312 +0000 2023-05-24T20:58:36.324 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: - 2023-05-24T20:58:36.325 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-05-24T20:58:36.388 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2023-05-24T20:58:36.389 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2023-05-24T20:58:36.389 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 9.4296e-05 s, 5.4 MB/s 2023-05-24T20:58:36.390 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-05-24T20:58:36.449 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_4 2023-05-24T20:58:36.507 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-05-24T20:58:36.507 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-05-24T20:58:36.507 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 6h/6d Inode: 372888 Links: 1 2023-05-24T20:58:36.508 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-05-24T20:58:36.508 INFO:teuthology.orchestra.run.smithi157.stdout:Context: system_u:object_r:device_t:s0 2023-05-24T20:58:36.508 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2023-05-24 20:55:57.512272105 +0000 2023-05-24T20:58:36.508 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2023-05-24 20:52:21.670568354 +0000 2023-05-24T20:58:36.508 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2023-05-24 20:52:21.670568354 +0000 2023-05-24T20:58:36.508 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: - 2023-05-24T20:58:36.509 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-05-24T20:58:36.573 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2023-05-24T20:58:36.574 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2023-05-24T20:58:36.574 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 0.000109524 s, 4.7 MB/s 2023-05-24T20:58:36.576 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-05-24T20:58:36.635 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-05-24T20:58:36.636 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi157.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-05-24T20:58:36.636 INFO:tasks.ceph:Generating config... 2023-05-24T20:58:36.638 INFO:tasks.ceph:[client] debug rgw = 20 2023-05-24T20:58:36.639 INFO:tasks.ceph:[client] debug rgw sync = 20 2023-05-24T20:58:36.639 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2023-05-24T20:58:36.639 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2023-05-24T20:58:36.639 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-05-24T20:58:36.639 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2023-05-24T20:58:36.640 INFO:tasks.ceph:[client] rgw data log num shards = 4 2023-05-24T20:58:36.640 INFO:tasks.ceph:[client] rgw md log max shards = 4 2023-05-24T20:58:36.640 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2023-05-24T20:58:36.640 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2023-05-24T20:58:36.640 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2023-05-24T20:58:36.641 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2023-05-24T20:58:36.641 INFO:tasks.ceph:[client] setgroup = ceph 2023-05-24T20:58:36.641 INFO:tasks.ceph:[client] setuser = ceph 2023-05-24T20:58:36.641 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2023-05-24T20:58:36.641 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-05-24T20:58:36.641 INFO:tasks.ceph:[mgr] debug ms = 1 2023-05-24T20:58:36.642 INFO:tasks.ceph:[mon] debug mon = 20 2023-05-24T20:58:36.642 INFO:tasks.ceph:[mon] debug ms = 1 2023-05-24T20:58:36.642 INFO:tasks.ceph:[mon] debug paxos = 20 2023-05-24T20:58:36.642 INFO:tasks.ceph:[osd] debug ms = 1 2023-05-24T20:58:36.642 INFO:tasks.ceph:[osd] debug osd = 20 2023-05-24T20:58:36.643 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2023-05-24T20:58:36.643 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2023-05-24T20:58:36.643 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2023-05-24T20:58:36.643 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2023-05-24T20:58:36.643 INFO:tasks.ceph:Setting up c2.mon.a... 2023-05-24T20:58:36.644 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2023-05-24T20:58:36.726 INFO:teuthology.orchestra.run.smithi157.stdout:creating /etc/ceph/c2.keyring 2023-05-24T20:58:36.729 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2023-05-24T20:58:36.774 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0644 /etc/ceph/c2.keyring 2023-05-24T20:58:36.802 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.157')] 2023-05-24T20:58:36.802 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.157'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2023-05-24T20:58:36.803 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-05-24T20:58:36.804 DEBUG:teuthology.orchestra.run.smithi157:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-05-24T20:58:36.859 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.157 --print /home/ubuntu/cephtest/c2.monmap 2023-05-24T20:58:36.943 INFO:teuthology.orchestra.run.smithi157.stderr:ignoring --setuser ceph since I am not root 2023-05-24T20:58:36.944 INFO:teuthology.orchestra.run.smithi157.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T20:58:36.944 INFO:teuthology.orchestra.run.smithi157.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2023-05-24T20:58:36.945 INFO:teuthology.orchestra.run.smithi157.stdout:monmaptool: generated fsid 946dce5b-d643-4d68-bc45-86cf88c78170 2023-05-24T20:58:36.945 INFO:teuthology.orchestra.run.smithi157.stdout:setting min_mon_release = pacific 2023-05-24T20:58:36.945 INFO:teuthology.orchestra.run.smithi157.stdout:epoch 0 2023-05-24T20:58:36.945 INFO:teuthology.orchestra.run.smithi157.stdout:fsid 946dce5b-d643-4d68-bc45-86cf88c78170 2023-05-24T20:58:36.945 INFO:teuthology.orchestra.run.smithi157.stdout:last_changed 2023-05-24T20:58:36.938942+0000 2023-05-24T20:58:36.945 INFO:teuthology.orchestra.run.smithi157.stdout:created 2023-05-24T20:58:36.938942+0000 2023-05-24T20:58:36.946 INFO:teuthology.orchestra.run.smithi157.stdout:min_mon_release 16 (pacific) 2023-05-24T20:58:36.946 INFO:teuthology.orchestra.run.smithi157.stdout:election_strategy: 1 2023-05-24T20:58:36.946 INFO:teuthology.orchestra.run.smithi157.stdout:0: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.a 2023-05-24T20:58:36.946 INFO:teuthology.orchestra.run.smithi157.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2023-05-24T20:58:36.947 DEBUG:teuthology.orchestra.run.smithi157:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-05-24T20:58:36.965 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID 946dce5b-d643-4d68-bc45-86cf88c78170... 2023-05-24T20:58:36.966 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2023-05-24T20:58:36.968 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2023-05-24T20:58:37.020 INFO:teuthology.orchestra.run.smithi071.stdout:[global] 2023-05-24T20:58:37.021 INFO:teuthology.orchestra.run.smithi071.stdout: chdir = "" 2023-05-24T20:58:37.021 INFO:teuthology.orchestra.run.smithi071.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-05-24T20:58:37.021 INFO:teuthology.orchestra.run.smithi071.stdout: auth supported = cephx 2023-05-24T20:58:37.021 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:37.021 INFO:teuthology.orchestra.run.smithi071.stdout: filestore xattr use omap = true 2023-05-24T20:58:37.021 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:37.022 INFO:teuthology.orchestra.run.smithi071.stdout: mon clock drift allowed = 1.000 2023-05-24T20:58:37.022 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:37.022 INFO:teuthology.orchestra.run.smithi071.stdout: osd crush chooseleaf type = 0 2023-05-24T20:58:37.022 INFO:teuthology.orchestra.run.smithi071.stdout: auth debug = true 2023-05-24T20:58:37.022 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:37.023 INFO:teuthology.orchestra.run.smithi071.stdout: ms die on old message = true 2023-05-24T20:58:37.023 INFO:teuthology.orchestra.run.smithi071.stdout: ms die on bug = true 2023-05-24T20:58:37.023 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:37.024 INFO:teuthology.orchestra.run.smithi071.stdout: mon max pg per osd = 10000 # >= luminous 2023-05-24T20:58:37.024 INFO:teuthology.orchestra.run.smithi071.stdout: mon pg warn max object skew = 0 2023-05-24T20:58:37.024 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:37.024 INFO:teuthology.orchestra.run.smithi071.stdout: # disable pg_autoscaler by default for new pools 2023-05-24T20:58:37.025 INFO:teuthology.orchestra.run.smithi071.stdout: osd_pool_default_pg_autoscale_mode = off 2023-05-24T20:58:37.025 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:37.025 INFO:teuthology.orchestra.run.smithi071.stdout: osd pool default size = 2 2023-05-24T20:58:37.025 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:37.026 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd allow primary affinity = true 2023-05-24T20:58:37.026 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd allow pg remap = true 2023-05-24T20:58:37.026 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on legacy crush tunables = false 2023-05-24T20:58:37.027 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on crush straw calc version zero = false 2023-05-24T20:58:37.027 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on no sortbitwise = false 2023-05-24T20:58:37.027 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on osd down out interval zero = false 2023-05-24T20:58:37.027 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on too few osds = false 2023-05-24T20:58:37.028 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-05-24T20:58:37.028 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_pool_no_redundancy = false 2023-05-24T20:58:37.029 INFO:teuthology.orchestra.run.smithi071.stdout: mon_allow_pool_size_one = true 2023-05-24T20:58:37.029 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:37.029 INFO:teuthology.orchestra.run.smithi071.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-05-24T20:58:37.030 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:37.030 INFO:teuthology.orchestra.run.smithi071.stdout: osd default data pool replay window = 5 2023-05-24T20:58:37.031 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:37.032 INFO:teuthology.orchestra.run.smithi071.stdout: mon allow pool delete = true 2023-05-24T20:58:37.032 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:37.032 INFO:teuthology.orchestra.run.smithi071.stdout: mon cluster log file level = debug 2023-05-24T20:58:37.032 INFO:teuthology.orchestra.run.smithi071.stdout: debug asserts on shutdown = true 2023-05-24T20:58:37.032 INFO:teuthology.orchestra.run.smithi071.stdout: mon health detail to clog = false 2023-05-24T20:58:37.032 INFO:teuthology.orchestra.run.smithi071.stdout: mon host = 172.21.15.157 2023-05-24T20:58:37.033 INFO:teuthology.orchestra.run.smithi071.stdout: fsid = 946dce5b-d643-4d68-bc45-86cf88c78170 2023-05-24T20:58:37.033 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:37.033 INFO:teuthology.orchestra.run.smithi071.stdout:[osd] 2023-05-24T20:58:37.033 INFO:teuthology.orchestra.run.smithi071.stdout: osd journal size = 100 2023-05-24T20:58:37.033 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:37.033 INFO:teuthology.orchestra.run.smithi071.stdout: osd scrub load threshold = 5.0 2023-05-24T20:58:37.033 INFO:teuthology.orchestra.run.smithi071.stdout: osd scrub max interval = 600 2023-05-24T20:58:37.033 INFO:teuthology.orchestra.run.smithi071.stdout: osd mclock profile = high_recovery_ops 2023-05-24T20:58:37.034 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:37.034 INFO:teuthology.orchestra.run.smithi071.stdout: osd recover clone overlap = true 2023-05-24T20:58:37.034 INFO:teuthology.orchestra.run.smithi071.stdout: osd recovery max chunk = 1048576 2023-05-24T20:58:37.034 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:37.034 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug shutdown = true 2023-05-24T20:58:37.034 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug op order = true 2023-05-24T20:58:37.034 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug verify stray on activate = true 2023-05-24T20:58:37.035 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:37.035 INFO:teuthology.orchestra.run.smithi071.stdout: osd open classes on start = true 2023-05-24T20:58:37.035 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug pg log writeout = true 2023-05-24T20:58:37.035 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:37.035 INFO:teuthology.orchestra.run.smithi071.stdout: osd deep scrub update digest min age = 30 2023-05-24T20:58:37.035 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:37.035 INFO:teuthology.orchestra.run.smithi071.stdout: osd map max advance = 10 2023-05-24T20:58:37.035 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:37.036 INFO:teuthology.orchestra.run.smithi071.stdout: journal zero on create = true 2023-05-24T20:58:37.036 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:37.036 INFO:teuthology.orchestra.run.smithi071.stdout: filestore ondisk finisher threads = 3 2023-05-24T20:58:37.036 INFO:teuthology.orchestra.run.smithi071.stdout: filestore apply finisher threads = 3 2023-05-24T20:58:37.036 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:37.036 INFO:teuthology.orchestra.run.smithi071.stdout: bdev debug aio = true 2023-05-24T20:58:37.036 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug misdirected ops = true 2023-05-24T20:58:37.037 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2023-05-24T20:58:37.037 INFO:teuthology.orchestra.run.smithi071.stdout: debug osd = 20 2023-05-24T20:58:37.037 INFO:teuthology.orchestra.run.smithi071.stdout: rocksdb delete range threshold = 0 2023-05-24T20:58:37.037 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:37.038 INFO:teuthology.orchestra.run.smithi071.stdout:[mgr] 2023-05-24T20:58:37.038 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2023-05-24T20:58:37.038 INFO:teuthology.orchestra.run.smithi071.stdout: debug mgr = 20 2023-05-24T20:58:37.038 INFO:teuthology.orchestra.run.smithi071.stdout: debug mon = 20 2023-05-24T20:58:37.038 INFO:teuthology.orchestra.run.smithi071.stdout: debug auth = 20 2023-05-24T20:58:37.038 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min pgs per osd = 4 2023-05-24T20:58:37.038 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min bytes per osd = 10 2023-05-24T20:58:37.038 INFO:teuthology.orchestra.run.smithi071.stdout: mgr/telemetry/nag = false 2023-05-24T20:58:37.039 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:37.039 INFO:teuthology.orchestra.run.smithi071.stdout:[mon] 2023-05-24T20:58:37.039 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2023-05-24T20:58:37.039 INFO:teuthology.orchestra.run.smithi071.stdout: debug mon = 20 2023-05-24T20:58:37.039 INFO:teuthology.orchestra.run.smithi071.stdout: debug paxos = 20 2023-05-24T20:58:37.039 INFO:teuthology.orchestra.run.smithi071.stdout: debug auth = 20 2023-05-24T20:58:37.039 INFO:teuthology.orchestra.run.smithi071.stdout: mon data avail warn = 5 2023-05-24T20:58:37.040 INFO:teuthology.orchestra.run.smithi071.stdout: mon mgr mkfs grace = 240 2023-05-24T20:58:37.040 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min pgs per osd = 4 2023-05-24T20:58:37.040 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd reporter subtree level = osd 2023-05-24T20:58:37.040 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd prime pg temp = true 2023-05-24T20:58:37.040 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min bytes per osd = 10 2023-05-24T20:58:37.040 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:37.040 INFO:teuthology.orchestra.run.smithi071.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-05-24T20:58:37.041 INFO:teuthology.orchestra.run.smithi071.stdout: auth mon ticket ttl = 660 # 11m 2023-05-24T20:58:37.041 INFO:teuthology.orchestra.run.smithi071.stdout: auth service ticket ttl = 240 # 4m 2023-05-24T20:58:37.041 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:37.041 INFO:teuthology.orchestra.run.smithi071.stdout: # don't complain about insecure global_id in the test suite 2023-05-24T20:58:37.041 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-05-24T20:58:37.042 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-05-24T20:58:37.042 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:37.042 INFO:teuthology.orchestra.run.smithi071.stdout: # 1m isn't quite enough 2023-05-24T20:58:37.042 INFO:teuthology.orchestra.run.smithi071.stdout: mon_down_mkfs_grace = 2m 2023-05-24T20:58:37.042 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:37.042 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_filestore_osds = false 2023-05-24T20:58:37.043 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:58:37.043 INFO:teuthology.orchestra.run.smithi071.stdout:[client] 2023-05-24T20:58:37.043 INFO:teuthology.orchestra.run.smithi071.stdout: rgw cache enabled = true 2023-05-24T20:58:37.043 INFO:teuthology.orchestra.run.smithi071.stdout: rgw enable ops log = true 2023-05-24T20:58:37.044 INFO:teuthology.orchestra.run.smithi071.stdout: rgw enable usage log = true 2023-05-24T20:58:37.044 INFO:teuthology.orchestra.run.smithi071.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-05-24T20:58:37.044 INFO:teuthology.orchestra.run.smithi071.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-05-24T20:58:37.045 INFO:teuthology.orchestra.run.smithi071.stdout: debug rgw = 20 2023-05-24T20:58:37.045 INFO:teuthology.orchestra.run.smithi071.stdout: debug rgw sync = 20 2023-05-24T20:58:37.045 INFO:teuthology.orchestra.run.smithi071.stdout: rgw crypt require ssl = False 2023-05-24T20:58:37.045 INFO:teuthology.orchestra.run.smithi071.stdout: rgw crypt s3 kms backend = testing 2023-05-24T20:58:37.045 INFO:teuthology.orchestra.run.smithi071.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-05-24T20:58:37.045 INFO:teuthology.orchestra.run.smithi071.stdout: rgw curl low speed time = 300 2023-05-24T20:58:37.045 INFO:teuthology.orchestra.run.smithi071.stdout: rgw data log num shards = 4 2023-05-24T20:58:37.045 INFO:teuthology.orchestra.run.smithi071.stdout: rgw md log max shards = 4 2023-05-24T20:58:37.046 INFO:teuthology.orchestra.run.smithi071.stdout: rgw sync data inject err probability = 0.1 2023-05-24T20:58:37.046 INFO:teuthology.orchestra.run.smithi071.stdout: rgw sync log trim interval = 0 2023-05-24T20:58:37.046 INFO:teuthology.orchestra.run.smithi071.stdout: rgw sync meta inject err probability = 0.1 2023-05-24T20:58:37.046 INFO:teuthology.orchestra.run.smithi071.stdout: rgw sync obj etag verify = True 2023-05-24T20:58:37.046 INFO:teuthology.orchestra.run.smithi071.stdout: setgroup = ceph 2023-05-24T20:58:37.046 INFO:teuthology.orchestra.run.smithi071.stdout: setuser = ceph 2023-05-24T20:58:37.046 INFO:teuthology.orchestra.run.smithi071.stdout:[mon.a] 2023-05-24T20:58:37.047 INFO:teuthology.orchestra.run.smithi071.stdout:[client.0] 2023-05-24T20:58:37.047 INFO:teuthology.orchestra.run.smithi071.stdout: rgw data notify interval msec = 0 2023-05-24T20:58:37.047 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.0] 2023-05-24T20:58:37.047 INFO:teuthology.orchestra.run.smithi071.stdout: osd_max_omap_entries_per_request = 10 2023-05-24T20:58:37.047 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.1] 2023-05-24T20:58:37.047 INFO:teuthology.orchestra.run.smithi071.stdout: osd_max_omap_entries_per_request = 1000 2023-05-24T20:58:37.047 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.2] 2023-05-24T20:58:37.047 INFO:teuthology.orchestra.run.smithi071.stdout: osd_max_omap_entries_per_request = 10000 2023-05-24T20:58:37.049 INFO:teuthology.orchestra.run.smithi157.stdout:[global] 2023-05-24T20:58:37.050 INFO:teuthology.orchestra.run.smithi157.stdout: chdir = "" 2023-05-24T20:58:37.050 INFO:teuthology.orchestra.run.smithi157.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-05-24T20:58:37.050 INFO:teuthology.orchestra.run.smithi157.stdout: auth supported = cephx 2023-05-24T20:58:37.050 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:37.050 INFO:teuthology.orchestra.run.smithi157.stdout: filestore xattr use omap = true 2023-05-24T20:58:37.050 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:37.050 INFO:teuthology.orchestra.run.smithi157.stdout: mon clock drift allowed = 1.000 2023-05-24T20:58:37.051 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:37.051 INFO:teuthology.orchestra.run.smithi157.stdout: osd crush chooseleaf type = 0 2023-05-24T20:58:37.051 INFO:teuthology.orchestra.run.smithi157.stdout: auth debug = true 2023-05-24T20:58:37.052 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:37.052 INFO:teuthology.orchestra.run.smithi157.stdout: ms die on old message = true 2023-05-24T20:58:37.052 INFO:teuthology.orchestra.run.smithi157.stdout: ms die on bug = true 2023-05-24T20:58:37.053 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:37.053 INFO:teuthology.orchestra.run.smithi157.stdout: mon max pg per osd = 10000 # >= luminous 2023-05-24T20:58:37.053 INFO:teuthology.orchestra.run.smithi157.stdout: mon pg warn max object skew = 0 2023-05-24T20:58:37.053 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:37.054 INFO:teuthology.orchestra.run.smithi157.stdout: # disable pg_autoscaler by default for new pools 2023-05-24T20:58:37.054 INFO:teuthology.orchestra.run.smithi157.stdout: osd_pool_default_pg_autoscale_mode = off 2023-05-24T20:58:37.054 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:37.055 INFO:teuthology.orchestra.run.smithi157.stdout: osd pool default size = 2 2023-05-24T20:58:37.055 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:37.055 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd allow primary affinity = true 2023-05-24T20:58:37.055 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd allow pg remap = true 2023-05-24T20:58:37.056 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on legacy crush tunables = false 2023-05-24T20:58:37.056 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on crush straw calc version zero = false 2023-05-24T20:58:37.056 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on no sortbitwise = false 2023-05-24T20:58:37.056 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on osd down out interval zero = false 2023-05-24T20:58:37.057 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on too few osds = false 2023-05-24T20:58:37.057 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-05-24T20:58:37.057 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_pool_no_redundancy = false 2023-05-24T20:58:37.057 INFO:teuthology.orchestra.run.smithi157.stdout: mon_allow_pool_size_one = true 2023-05-24T20:58:37.057 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:37.057 INFO:teuthology.orchestra.run.smithi157.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-05-24T20:58:37.057 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:37.058 INFO:teuthology.orchestra.run.smithi157.stdout: osd default data pool replay window = 5 2023-05-24T20:58:37.058 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:37.058 INFO:teuthology.orchestra.run.smithi157.stdout: mon allow pool delete = true 2023-05-24T20:58:37.058 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:37.058 INFO:teuthology.orchestra.run.smithi157.stdout: mon cluster log file level = debug 2023-05-24T20:58:37.058 INFO:teuthology.orchestra.run.smithi157.stdout: debug asserts on shutdown = true 2023-05-24T20:58:37.058 INFO:teuthology.orchestra.run.smithi157.stdout: mon health detail to clog = false 2023-05-24T20:58:37.058 INFO:teuthology.orchestra.run.smithi157.stdout: mon host = 172.21.15.157 2023-05-24T20:58:37.059 INFO:teuthology.orchestra.run.smithi157.stdout: fsid = 946dce5b-d643-4d68-bc45-86cf88c78170 2023-05-24T20:58:37.059 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:37.059 INFO:teuthology.orchestra.run.smithi157.stdout:[osd] 2023-05-24T20:58:37.059 INFO:teuthology.orchestra.run.smithi157.stdout: osd journal size = 100 2023-05-24T20:58:37.059 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:37.059 INFO:teuthology.orchestra.run.smithi157.stdout: osd scrub load threshold = 5.0 2023-05-24T20:58:37.059 INFO:teuthology.orchestra.run.smithi157.stdout: osd scrub max interval = 600 2023-05-24T20:58:37.060 INFO:teuthology.orchestra.run.smithi157.stdout: osd mclock profile = high_recovery_ops 2023-05-24T20:58:37.060 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:37.060 INFO:teuthology.orchestra.run.smithi157.stdout: osd recover clone overlap = true 2023-05-24T20:58:37.060 INFO:teuthology.orchestra.run.smithi157.stdout: osd recovery max chunk = 1048576 2023-05-24T20:58:37.060 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:37.060 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug shutdown = true 2023-05-24T20:58:37.060 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug op order = true 2023-05-24T20:58:37.061 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug verify stray on activate = true 2023-05-24T20:58:37.061 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:37.061 INFO:teuthology.orchestra.run.smithi157.stdout: osd open classes on start = true 2023-05-24T20:58:37.061 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug pg log writeout = true 2023-05-24T20:58:37.061 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:37.061 INFO:teuthology.orchestra.run.smithi157.stdout: osd deep scrub update digest min age = 30 2023-05-24T20:58:37.062 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:37.062 INFO:teuthology.orchestra.run.smithi157.stdout: osd map max advance = 10 2023-05-24T20:58:37.062 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:37.062 INFO:teuthology.orchestra.run.smithi157.stdout: journal zero on create = true 2023-05-24T20:58:37.062 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:37.062 INFO:teuthology.orchestra.run.smithi157.stdout: filestore ondisk finisher threads = 3 2023-05-24T20:58:37.062 INFO:teuthology.orchestra.run.smithi157.stdout: filestore apply finisher threads = 3 2023-05-24T20:58:37.063 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:37.063 INFO:teuthology.orchestra.run.smithi157.stdout: bdev debug aio = true 2023-05-24T20:58:37.063 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug misdirected ops = true 2023-05-24T20:58:37.063 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2023-05-24T20:58:37.063 INFO:teuthology.orchestra.run.smithi157.stdout: debug osd = 20 2023-05-24T20:58:37.063 INFO:teuthology.orchestra.run.smithi157.stdout: rocksdb delete range threshold = 0 2023-05-24T20:58:37.063 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:37.064 INFO:teuthology.orchestra.run.smithi157.stdout:[mgr] 2023-05-24T20:58:37.064 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2023-05-24T20:58:37.064 INFO:teuthology.orchestra.run.smithi157.stdout: debug mgr = 20 2023-05-24T20:58:37.064 INFO:teuthology.orchestra.run.smithi157.stdout: debug mon = 20 2023-05-24T20:58:37.064 INFO:teuthology.orchestra.run.smithi157.stdout: debug auth = 20 2023-05-24T20:58:37.064 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min pgs per osd = 4 2023-05-24T20:58:37.064 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min bytes per osd = 10 2023-05-24T20:58:37.065 INFO:teuthology.orchestra.run.smithi157.stdout: mgr/telemetry/nag = false 2023-05-24T20:58:37.065 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:37.065 INFO:teuthology.orchestra.run.smithi157.stdout:[mon] 2023-05-24T20:58:37.065 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2023-05-24T20:58:37.065 INFO:teuthology.orchestra.run.smithi157.stdout: debug mon = 20 2023-05-24T20:58:37.065 INFO:teuthology.orchestra.run.smithi157.stdout: debug paxos = 20 2023-05-24T20:58:37.065 INFO:teuthology.orchestra.run.smithi157.stdout: debug auth = 20 2023-05-24T20:58:37.066 INFO:teuthology.orchestra.run.smithi157.stdout: mon data avail warn = 5 2023-05-24T20:58:37.066 INFO:teuthology.orchestra.run.smithi157.stdout: mon mgr mkfs grace = 240 2023-05-24T20:58:37.066 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min pgs per osd = 4 2023-05-24T20:58:37.066 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd reporter subtree level = osd 2023-05-24T20:58:37.066 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd prime pg temp = true 2023-05-24T20:58:37.066 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min bytes per osd = 10 2023-05-24T20:58:37.066 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:37.066 INFO:teuthology.orchestra.run.smithi157.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-05-24T20:58:37.067 INFO:teuthology.orchestra.run.smithi157.stdout: auth mon ticket ttl = 660 # 11m 2023-05-24T20:58:37.067 INFO:teuthology.orchestra.run.smithi157.stdout: auth service ticket ttl = 240 # 4m 2023-05-24T20:58:37.067 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:37.067 INFO:teuthology.orchestra.run.smithi157.stdout: # don't complain about insecure global_id in the test suite 2023-05-24T20:58:37.067 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-05-24T20:58:37.067 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-05-24T20:58:37.067 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:37.068 INFO:teuthology.orchestra.run.smithi157.stdout: # 1m isn't quite enough 2023-05-24T20:58:37.068 INFO:teuthology.orchestra.run.smithi157.stdout: mon_down_mkfs_grace = 2m 2023-05-24T20:58:37.068 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:37.068 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_filestore_osds = false 2023-05-24T20:58:37.068 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:37.068 INFO:teuthology.orchestra.run.smithi157.stdout:[client] 2023-05-24T20:58:37.068 INFO:teuthology.orchestra.run.smithi157.stdout: rgw cache enabled = true 2023-05-24T20:58:37.069 INFO:teuthology.orchestra.run.smithi157.stdout: rgw enable ops log = true 2023-05-24T20:58:37.069 INFO:teuthology.orchestra.run.smithi157.stdout: rgw enable usage log = true 2023-05-24T20:58:37.069 INFO:teuthology.orchestra.run.smithi157.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-05-24T20:58:37.069 INFO:teuthology.orchestra.run.smithi157.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-05-24T20:58:37.069 INFO:teuthology.orchestra.run.smithi157.stdout: debug rgw = 20 2023-05-24T20:58:37.069 INFO:teuthology.orchestra.run.smithi157.stdout: debug rgw sync = 20 2023-05-24T20:58:37.070 INFO:teuthology.orchestra.run.smithi157.stdout: rgw crypt require ssl = False 2023-05-24T20:58:37.070 INFO:teuthology.orchestra.run.smithi157.stdout: rgw crypt s3 kms backend = testing 2023-05-24T20:58:37.070 INFO:teuthology.orchestra.run.smithi157.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-05-24T20:58:37.070 INFO:teuthology.orchestra.run.smithi157.stdout: rgw curl low speed time = 300 2023-05-24T20:58:37.070 INFO:teuthology.orchestra.run.smithi157.stdout: rgw data log num shards = 4 2023-05-24T20:58:37.070 INFO:teuthology.orchestra.run.smithi157.stdout: rgw md log max shards = 4 2023-05-24T20:58:37.071 INFO:teuthology.orchestra.run.smithi157.stdout: rgw sync data inject err probability = 0.1 2023-05-24T20:58:37.071 INFO:teuthology.orchestra.run.smithi157.stdout: rgw sync log trim interval = 0 2023-05-24T20:58:37.071 INFO:teuthology.orchestra.run.smithi157.stdout: rgw sync meta inject err probability = 0.1 2023-05-24T20:58:37.071 INFO:teuthology.orchestra.run.smithi157.stdout: rgw sync obj etag verify = True 2023-05-24T20:58:37.071 INFO:teuthology.orchestra.run.smithi157.stdout: setgroup = ceph 2023-05-24T20:58:37.071 INFO:teuthology.orchestra.run.smithi157.stdout: setuser = ceph 2023-05-24T20:58:37.071 INFO:teuthology.orchestra.run.smithi157.stdout:[mon.a] 2023-05-24T20:58:37.072 INFO:teuthology.orchestra.run.smithi157.stdout:[client.0] 2023-05-24T20:58:37.072 INFO:teuthology.orchestra.run.smithi157.stdout: rgw data notify interval msec = 0 2023-05-24T20:58:37.072 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.0] 2023-05-24T20:58:37.072 INFO:teuthology.orchestra.run.smithi157.stdout: osd_max_omap_entries_per_request = 10 2023-05-24T20:58:37.072 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.1] 2023-05-24T20:58:37.072 INFO:teuthology.orchestra.run.smithi157.stdout: osd_max_omap_entries_per_request = 1000 2023-05-24T20:58:37.072 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.2] 2023-05-24T20:58:37.073 INFO:teuthology.orchestra.run.smithi157.stdout: osd_max_omap_entries_per_request = 10000 2023-05-24T20:58:37.073 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2023-05-24T20:58:37.074 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2023-05-24T20:58:37.171 INFO:tasks.ceph:Copying monmap to all nodes... 2023-05-24T20:58:37.172 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-05-24T20:58:37.172 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2023-05-24T20:58:37.190 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-05-24T20:58:37.191 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2023-05-24T20:58:37.249 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi071.front.sepia.ceph.com 2023-05-24T20:58:37.250 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-05-24T20:58:37.250 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/ceph/c2.keyring 2023-05-24T20:58:37.250 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0644 /etc/ceph/c2.keyring 2023-05-24T20:58:37.295 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-05-24T20:58:37.295 DEBUG:teuthology.orchestra.run.smithi071:> dd of=/home/ubuntu/cephtest/c2.monmap 2023-05-24T20:58:37.355 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi157.front.sepia.ceph.com 2023-05-24T20:58:37.356 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-05-24T20:58:37.356 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/ceph/c2.keyring 2023-05-24T20:58:37.357 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0644 /etc/ceph/c2.keyring 2023-05-24T20:58:37.394 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-05-24T20:58:37.395 DEBUG:teuthology.orchestra.run.smithi157:> dd of=/home/ubuntu/cephtest/c2.monmap 2023-05-24T20:58:37.452 INFO:tasks.ceph:Setting up mon nodes... 2023-05-24T20:58:37.453 INFO:tasks.ceph:Setting up mgr nodes... 2023-05-24T20:58:37.453 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2023-05-24T20:58:37.546 INFO:teuthology.orchestra.run.smithi157.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2023-05-24T20:58:37.549 INFO:tasks.ceph:Setting up mds nodes... 2023-05-24T20:58:37.549 INFO:tasks.ceph_client:Setting up client nodes... 2023-05-24T20:58:37.549 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2023-05-24T20:58:37.593 INFO:teuthology.orchestra.run.smithi157.stdout:creating /etc/ceph/c2.client.0.keyring 2023-05-24T20:58:37.605 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2023-05-24T20:58:37.649 INFO:teuthology.orchestra.run.smithi157.stdout:creating /etc/ceph/c2.client.1.keyring 2023-05-24T20:58:37.662 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-05-24T20:58:37.662 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi071.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi157.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-05-24T20:58:37.663 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2023-05-24T20:58:37.729 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-05-24T20:58:37.729 INFO:tasks.ceph:role: c2.osd.0 2023-05-24T20:58:37.730 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi157.front.sepia.ceph.com 2023-05-24T20:58:37.730 DEBUG:teuthology.orchestra.run.smithi157:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-05-24T20:58:37.797 INFO:teuthology.orchestra.run.smithi157.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-05-24T20:58:37.797 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 attr=2, projid32bit=1 2023-05-24T20:58:37.797 INFO:teuthology.orchestra.run.smithi157.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-05-24T20:58:37.797 INFO:teuthology.orchestra.run.smithi157.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-05-24T20:58:37.797 INFO:teuthology.orchestra.run.smithi157.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-05-24T20:58:37.798 INFO:teuthology.orchestra.run.smithi157.stdout: = sunit=0 swidth=0 blks 2023-05-24T20:58:37.798 INFO:teuthology.orchestra.run.smithi157.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-05-24T20:58:37.798 INFO:teuthology.orchestra.run.smithi157.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-05-24T20:58:37.798 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-05-24T20:58:37.798 INFO:teuthology.orchestra.run.smithi157.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-05-24T20:58:37.820 INFO:teuthology.orchestra.run.smithi157.stdout:Discarding blocks...Done. 2023-05-24T20:58:37.821 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi157.front.sepia.ceph.com -o noatime 2023-05-24T20:58:37.822 DEBUG:teuthology.orchestra.run.smithi157:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2023-05-24T20:58:37.903 DEBUG:teuthology.orchestra.run.smithi157:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2023-05-24T20:58:37.976 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2023-05-24T20:58:38.043 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-05-24T20:58:38.043 INFO:tasks.ceph:role: c2.osd.1 2023-05-24T20:58:38.043 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi157.front.sepia.ceph.com 2023-05-24T20:58:38.044 DEBUG:teuthology.orchestra.run.smithi157:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-05-24T20:58:38.110 INFO:teuthology.orchestra.run.smithi157.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-05-24T20:58:38.111 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 attr=2, projid32bit=1 2023-05-24T20:58:38.111 INFO:teuthology.orchestra.run.smithi157.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-05-24T20:58:38.111 INFO:teuthology.orchestra.run.smithi157.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-05-24T20:58:38.111 INFO:teuthology.orchestra.run.smithi157.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-05-24T20:58:38.111 INFO:teuthology.orchestra.run.smithi157.stdout: = sunit=0 swidth=0 blks 2023-05-24T20:58:38.111 INFO:teuthology.orchestra.run.smithi157.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-05-24T20:58:38.111 INFO:teuthology.orchestra.run.smithi157.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-05-24T20:58:38.112 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-05-24T20:58:38.112 INFO:teuthology.orchestra.run.smithi157.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-05-24T20:58:38.133 INFO:teuthology.orchestra.run.smithi157.stdout:Discarding blocks...Done. 2023-05-24T20:58:38.134 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi157.front.sepia.ceph.com -o noatime 2023-05-24T20:58:38.135 DEBUG:teuthology.orchestra.run.smithi157:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2023-05-24T20:58:38.214 DEBUG:teuthology.orchestra.run.smithi157:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2023-05-24T20:58:38.285 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2023-05-24T20:58:38.353 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-05-24T20:58:38.354 INFO:tasks.ceph:role: c2.osd.2 2023-05-24T20:58:38.354 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi157.front.sepia.ceph.com 2023-05-24T20:58:38.354 DEBUG:teuthology.orchestra.run.smithi157:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-05-24T20:58:38.420 INFO:teuthology.orchestra.run.smithi157.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-05-24T20:58:38.421 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 attr=2, projid32bit=1 2023-05-24T20:58:38.421 INFO:teuthology.orchestra.run.smithi157.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-05-24T20:58:38.421 INFO:teuthology.orchestra.run.smithi157.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-05-24T20:58:38.421 INFO:teuthology.orchestra.run.smithi157.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-05-24T20:58:38.421 INFO:teuthology.orchestra.run.smithi157.stdout: = sunit=0 swidth=0 blks 2023-05-24T20:58:38.421 INFO:teuthology.orchestra.run.smithi157.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-05-24T20:58:38.422 INFO:teuthology.orchestra.run.smithi157.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-05-24T20:58:38.422 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-05-24T20:58:38.422 INFO:teuthology.orchestra.run.smithi157.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-05-24T20:58:38.443 INFO:teuthology.orchestra.run.smithi157.stdout:Discarding blocks...Done. 2023-05-24T20:58:38.444 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi157.front.sepia.ceph.com -o noatime 2023-05-24T20:58:38.445 DEBUG:teuthology.orchestra.run.smithi157:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2023-05-24T20:58:38.525 DEBUG:teuthology.orchestra.run.smithi157:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2023-05-24T20:58:38.595 DEBUG:teuthology.orchestra.run.smithi157:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2023-05-24T20:58:38.708 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:58:38.702+0000 7faa23882640 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2023-05-24T20:58:38.709 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:58:38.702+0000 7faa23882640 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2023-05-24T20:58:38.709 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:58:38.702+0000 7faa23882640 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2023-05-24T20:58:38.709 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:58:38.702+0000 7faa23882640 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2023-05-24T20:58:38.710 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:58:38.702+0000 7faa23882640 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2023-05-24T20:58:38.710 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:58:38.702+0000 7faa23882640 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2023-05-24T20:58:40.779 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2023-05-24T20:58:40.813 DEBUG:teuthology.orchestra.run.smithi157:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2023-05-24T20:58:40.906 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:58:40.899+0000 7ff6e700b640 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2023-05-24T20:58:40.906 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:58:40.900+0000 7ff6e700b640 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2023-05-24T20:58:40.906 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:58:40.900+0000 7ff6e700b640 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2023-05-24T20:58:40.906 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:58:40.900+0000 7ff6e700b640 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2023-05-24T20:58:40.906 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:58:40.900+0000 7ff6e700b640 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2023-05-24T20:58:40.906 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:58:40.900+0000 7ff6e700b640 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2023-05-24T20:58:42.980 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2023-05-24T20:58:43.013 DEBUG:teuthology.orchestra.run.smithi157:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2023-05-24T20:58:43.105 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:58:43.099+0000 7fdbfb405640 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2023-05-24T20:58:43.105 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:58:43.099+0000 7fdbfb405640 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2023-05-24T20:58:43.106 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:58:43.099+0000 7fdbfb405640 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2023-05-24T20:58:43.106 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:58:43.099+0000 7fdbfb405640 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2023-05-24T20:58:43.106 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:58:43.099+0000 7fdbfb405640 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2023-05-24T20:58:43.106 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:58:43.099+0000 7fdbfb405640 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2023-05-24T20:58:45.175 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2023-05-24T20:58:45.209 INFO:tasks.ceph:Reading keys from all nodes... 2023-05-24T20:58:45.209 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-05-24T20:58:45.209 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2023-05-24T20:58:45.280 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-05-24T20:58:45.280 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2023-05-24T20:58:45.349 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-05-24T20:58:45.350 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2023-05-24T20:58:45.419 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-05-24T20:58:45.419 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2023-05-24T20:58:45.488 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-05-24T20:58:45.488 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2023-05-24T20:58:45.547 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-05-24T20:58:45.547 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2023-05-24T20:58:45.605 INFO:tasks.ceph:Adding keys to all mons... 2023-05-24T20:58:45.606 DEBUG:teuthology.orchestra.run.smithi157:> sudo tee -a /etc/ceph/c2.keyring 2023-05-24T20:58:45.672 INFO:teuthology.orchestra.run.smithi157.stdout:[mgr.x] 2023-05-24T20:58:45.672 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQB9em5kAYhDIBAAvvg42xXnyE15L720mHQmPw== 2023-05-24T20:58:45.673 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.0] 2023-05-24T20:58:45.673 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQB+em5k+mziKRAAEcU0UN2qoFO0HElCJdkx9A== 2023-05-24T20:58:45.673 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.1] 2023-05-24T20:58:45.673 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQCAem5kZIuqNRAAS7ww9njSuBFFQF1SNddn5w== 2023-05-24T20:58:45.673 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.2] 2023-05-24T20:58:45.673 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQCDem5k7jfyBRAA9YVfBqJRZaMw15DBQK3euw== 2023-05-24T20:58:45.673 INFO:teuthology.orchestra.run.smithi157.stdout:[client.0] 2023-05-24T20:58:45.674 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQB9em5keJwPIxAABZnnuGvb8oqu3K83EaAa6A== 2023-05-24T20:58:45.674 INFO:teuthology.orchestra.run.smithi157.stdout:[client.1] 2023-05-24T20:58:45.674 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQB9em5k4hFjJhAAQh/aMeh6mcyvscDIwNNbYQ== 2023-05-24T20:58:45.675 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-05-24T20:58:45.769 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-05-24T20:58:45.823 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-05-24T20:58:45.878 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-05-24T20:58:45.932 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-05-24T20:58:45.989 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-05-24T20:58:46.046 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-05-24T20:58:46.047 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2023-05-24T20:58:46.074 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2023-05-24T20:58:46.174 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2023-05-24T20:58:46.203 DEBUG:teuthology.orchestra.run.smithi157:> rm -- /home/ubuntu/cephtest/c2.monmap 2023-05-24T20:58:46.265 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2023-05-24T20:58:46.266 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2023-05-24T20:58:46.266 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2023-05-24T20:58:46.308 INFO:tasks.ceph.c2.mon.a:Started 2023-05-24T20:58:46.308 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2023-05-24T20:58:46.309 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2023-05-24T20:58:46.309 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2023-05-24T20:58:46.311 INFO:tasks.ceph.c2.mgr.x:Started 2023-05-24T20:58:46.311 INFO:tasks.ceph:Setting crush tunables to default 2023-05-24T20:58:46.312 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster c2 osd crush tunables default 2023-05-24T20:58:46.519 INFO:teuthology.orchestra.run.smithi157.stderr:adjusted tunables profile to default 2023-05-24T20:58:46.529 INFO:tasks.ceph:check_enable_crimson: False 2023-05-24T20:58:46.530 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2023-05-24T20:58:46.530 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-05-24T20:58:46.531 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2023-05-24T20:58:46.558 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-05-24T20:58:46.558 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2023-05-24T20:58:46.625 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-05-24T20:58:46.625 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2023-05-24T20:58:46.691 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster c2 osd new 44f3e715-b430-4e7d-bb59-24afe3d6e9cc 0 2023-05-24T20:58:46.722 INFO:tasks.ceph.c2.mgr.x.smithi157.stderr:2023-05-24T20:58:46.716+0000 7f6a2a76c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-24T20:58:46.899 INFO:tasks.ceph.c2.mgr.x.smithi157.stderr:2023-05-24T20:58:46.893+0000 7f6a2a76c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-24T20:58:46.964 INFO:teuthology.orchestra.run.smithi157.stdout:0 2023-05-24T20:58:46.977 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster c2 osd new 1e623fb0-b0b3-4df6-9079-e2c9654d11fb 1 2023-05-24T20:58:47.191 INFO:teuthology.orchestra.run.smithi157.stdout:1 2023-05-24T20:58:47.204 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster c2 osd new 5acf236b-29ec-46ab-a632-10e4551b0322 2 2023-05-24T20:58:47.271 INFO:tasks.ceph.c2.mgr.x.smithi157.stderr:2023-05-24T20:58:47.265+0000 7f6a2a76c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-24T20:58:47.419 INFO:teuthology.orchestra.run.smithi157.stdout:2 2023-05-24T20:58:47.433 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2023-05-24T20:58:47.434 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2023-05-24T20:58:47.436 INFO:tasks.ceph.c2.osd.0:Started 2023-05-24T20:58:47.437 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2023-05-24T20:58:47.437 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2023-05-24T20:58:47.438 INFO:tasks.ceph.c2.osd.1:Started 2023-05-24T20:58:47.439 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2023-05-24T20:58:47.439 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2023-05-24T20:58:47.441 INFO:tasks.ceph.c2.osd.2:Started 2023-05-24T20:58:47.442 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2023-05-24T20:58:47.673 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:47.673 INFO:teuthology.orchestra.run.smithi157.stdout:{"epoch":5,"fsid":"946dce5b-d643-4d68-bc45-86cf88c78170","created":"2023-05-24T20:58:46.420452+0000","modified":"2023-05-24T20:58:47.413150+0000","last_up_change":"0.000000","last_in_change":"2023-05-24T20:58:47.413150+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"44f3e715-b430-4e7d-bb59-24afe3d6e9cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1e623fb0-b0b3-4df6-9079-e2c9654d11fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5acf236b-29ec-46ab-a632-10e4551b0322","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-24T20:58:47.685 INFO:tasks.ceph.ceph_manager.c2:[] 2023-05-24T20:58:47.686 INFO:tasks.ceph:Waiting for OSDs to come up 2023-05-24T20:58:47.800 INFO:tasks.ceph.c2.mgr.x.smithi157.stderr:2023-05-24T20:58:47.794+0000 7f6a2a76c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-05-24T20:58:47.988 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2023-05-24T20:58:47.994 INFO:tasks.ceph.c2.mgr.x.smithi157.stderr:2023-05-24T20:58:47.989+0000 7f6a2a76c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-05-24T20:58:48.037 INFO:tasks.ceph.c2.osd.0.smithi157.stderr:2023-05-24T20:58:48.031+0000 7f3a97cc4640 -1 Falling back to public interface 2023-05-24T20:58:48.046 INFO:tasks.ceph.c2.osd.1.smithi157.stderr:2023-05-24T20:58:48.040+0000 7f3f7107b640 -1 Falling back to public interface 2023-05-24T20:58:48.064 INFO:tasks.ceph.c2.osd.2.smithi157.stderr:2023-05-24T20:58:48.058+0000 7f2982694640 -1 Falling back to public interface 2023-05-24T20:58:48.181 INFO:teuthology.misc.health.smithi157.stdout: 2023-05-24T20:58:48.182 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":5,"fsid":"946dce5b-d643-4d68-bc45-86cf88c78170","created":"2023-05-24T20:58:46.420452+0000","modified":"2023-05-24T20:58:47.413150+0000","last_up_change":"0.000000","last_in_change":"2023-05-24T20:58:47.413150+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"44f3e715-b430-4e7d-bb59-24afe3d6e9cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1e623fb0-b0b3-4df6-9079-e2c9654d11fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5acf236b-29ec-46ab-a632-10e4551b0322","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-24T20:58:48.193 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-05-24T20:58:48.365 INFO:tasks.ceph.c2.mgr.x.smithi157.stderr:2023-05-24T20:58:48.359+0000 7f6a2a76c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-24T20:58:49.023 INFO:tasks.ceph.c2.mgr.x.smithi157.stderr:2023-05-24T20:58:49.017+0000 7f6a2a76c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-24T20:58:49.095 INFO:tasks.ceph.c2.mgr.x.smithi157.stderr:2023-05-24T20:58:49.090+0000 7f6a2a76c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-24T20:58:49.231 INFO:tasks.ceph.c2.mgr.x.smithi157.stderr:2023-05-24T20:58:49.225+0000 7f6a2a76c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-24T20:58:49.302 INFO:tasks.ceph.c2.mgr.x.smithi157.stderr:2023-05-24T20:58:49.295+0000 7f6a2a76c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-24T20:58:49.493 INFO:tasks.ceph.c2.mgr.x.smithi157.stderr:2023-05-24T20:58:49.487+0000 7f6a2a76c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-24T20:58:49.571 INFO:tasks.ceph.c2.mgr.x.smithi157.stderr:2023-05-24T20:58:49.566+0000 7f6a2a76c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-24T20:58:49.650 INFO:tasks.ceph.c2.mgr.x.smithi157.stderr:2023-05-24T20:58:49.644+0000 7f6a2a76c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-24T20:58:49.745 INFO:tasks.ceph.c2.mgr.x.smithi157.stderr:2023-05-24T20:58:49.739+0000 7f6a2a76c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-24T20:58:49.926 INFO:tasks.ceph.c2.mgr.x.smithi157.stderr:2023-05-24T20:58:49.921+0000 7f6a2a76c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-24T20:58:50.010 INFO:tasks.ceph.c2.mgr.x.smithi157.stderr:2023-05-24T20:58:50.004+0000 7f6a2a76c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-24T20:58:50.318 INFO:tasks.ceph.c2.mgr.x.smithi157.stderr:2023-05-24T20:58:50.312+0000 7f6a2a76c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-24T20:58:50.390 INFO:tasks.ceph.c2.mgr.x.smithi157.stderr:2023-05-24T20:58:50.384+0000 7f6a2a76c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-24T20:58:50.744 INFO:tasks.ceph.c2.osd.0.smithi157.stderr:2023-05-24T20:58:50.738+0000 7f3a97cc4640 -1 osd.0 0 log_to_monitors true 2023-05-24T20:58:50.783 INFO:tasks.ceph.c2.osd.1.smithi157.stderr:2023-05-24T20:58:50.777+0000 7f3f7107b640 -1 osd.1 0 log_to_monitors true 2023-05-24T20:58:50.793 INFO:tasks.ceph.c2.osd.2.smithi157.stderr:2023-05-24T20:58:50.787+0000 7f2982694640 -1 osd.2 0 log_to_monitors true 2023-05-24T20:58:50.915 INFO:tasks.ceph.c2.mgr.x.smithi157.stderr:2023-05-24T20:58:50.909+0000 7f6a2a76c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-24T20:58:51.230 INFO:tasks.ceph.c2.mgr.x.smithi157.stderr:2023-05-24T20:58:51.225+0000 7f6a2a76c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-24T20:58:51.372 INFO:tasks.ceph.c2.mgr.x.smithi157.stderr:2023-05-24T20:58:51.366+0000 7f6a2a76c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-24T20:58:51.728 INFO:tasks.ceph.c2.mgr.x.smithi157.stderr:2023-05-24T20:58:51.722+0000 7f6a2a76c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-24T20:58:51.896 INFO:tasks.ceph.c2.mgr.x.smithi157.stderr:2023-05-24T20:58:51.890+0000 7f6a2a76c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-24T20:58:52.107 INFO:tasks.ceph.c2.mgr.x.smithi157.stderr:2023-05-24T20:58:52.101+0000 7f6a2a76c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-24T20:58:52.469 INFO:tasks.ceph.c2.mgr.x.smithi157.stderr:2023-05-24T20:58:52.463+0000 7f6a2a76c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-24T20:58:52.515 INFO:tasks.ceph.c2.osd.2.smithi157.stderr:2023-05-24T20:58:52.508+0000 7f2976a35700 -1 osd.2 0 waiting for initial osdmap 2023-05-24T20:58:52.518 INFO:tasks.ceph.c2.osd.1.smithi157.stderr:2023-05-24T20:58:52.513+0000 7f3f6541c700 -1 osd.1 0 waiting for initial osdmap 2023-05-24T20:58:52.528 INFO:tasks.ceph.c2.osd.1.smithi157.stderr:2023-05-24T20:58:52.522+0000 7f3f5cff6700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-05-24T20:58:52.529 INFO:tasks.ceph.c2.osd.2.smithi157.stderr:2023-05-24T20:58:52.523+0000 7f296e60f700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-05-24T20:58:52.540 INFO:tasks.ceph.c2.osd.0.smithi157.stderr:2023-05-24T20:58:52.534+0000 7f3a8c065700 -1 osd.0 0 waiting for initial osdmap 2023-05-24T20:58:52.549 INFO:tasks.ceph.c2.osd.0.smithi157.stderr:2023-05-24T20:58:52.543+0000 7f3a83c3f700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-05-24T20:58:53.489 INFO:tasks.ceph.c2.mgr.x.smithi157.stderr:2023-05-24T20:58:53.484+0000 7f69ebb88700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-05-24T20:58:54.497 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2023-05-24T20:58:54.820 INFO:teuthology.misc.health.smithi157.stdout: 2023-05-24T20:58:54.820 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":9,"fsid":"946dce5b-d643-4d68-bc45-86cf88c78170","created":"2023-05-24T20:58:46.420452+0000","modified":"2023-05-24T20:58:54.516189+0000","last_up_change":"2023-05-24T20:58:53.515484+0000","last_in_change":"2023-05-24T20:58:47.413150+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"44f3e715-b430-4e7d-bb59-24afe3d6e9cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":104794},{"type":"v1","addr":"172.21.15.157:6801","nonce":104794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":104794},{"type":"v1","addr":"172.21.15.157:6803","nonce":104794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":104794},{"type":"v1","addr":"172.21.15.157:6807","nonce":104794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":104794},{"type":"v1","addr":"172.21.15.157:6805","nonce":104794}]},"public_addr":"172.21.15.157:6801/104794","cluster_addr":"172.21.15.157:6803/104794","heartbeat_back_addr":"172.21.15.157:6807/104794","heartbeat_front_addr":"172.21.15.157:6805/104794","state":["exists","up"]},{"osd":1,"uuid":"1e623fb0-b0b3-4df6-9079-e2c9654d11fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":104795},{"type":"v1","addr":"172.21.15.157:6809","nonce":104795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":104795},{"type":"v1","addr":"172.21.15.157:6811","nonce":104795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":104795},{"type":"v1","addr":"172.21.15.157:6815","nonce":104795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":104795},{"type":"v1","addr":"172.21.15.157:6813","nonce":104795}]},"public_addr":"172.21.15.157:6809/104795","cluster_addr":"172.21.15.157:6811/104795","heartbeat_back_addr":"172.21.15.157:6815/104795","heartbeat_front_addr":"172.21.15.157:6813/104795","state":["exists","up"]},{"osd":2,"uuid":"5acf236b-29ec-46ab-a632-10e4551b0322","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":104808},{"type":"v1","addr":"172.21.15.157:6817","nonce":104808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":104808},{"type":"v1","addr":"172.21.15.157:6819","nonce":104808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":104808},{"type":"v1","addr":"172.21.15.157:6823","nonce":104808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":104808},{"type":"v1","addr":"172.21.15.157:6821","nonce":104808}]},"public_addr":"172.21.15.157:6817/104808","cluster_addr":"172.21.15.157:6819/104808","heartbeat_back_addr":"172.21.15.157:6823/104808","heartbeat_front_addr":"172.21.15.157:6821/104808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-24T20:58:51.730481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-24T20:58:52.514508+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-24T20:58:52.508726+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-24T20:58:54.833 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-05-24T20:58:54.833 INFO:tasks.ceph:Creating RBD pool 2023-05-24T20:58:54.833 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster c2 osd pool create rbd 8 2023-05-24T20:58:55.531 INFO:teuthology.orchestra.run.smithi157.stderr:pool 'rbd' created 2023-05-24T20:58:55.542 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-05-24T20:58:56.536 INFO:teuthology.orchestra.run.smithi157.stderr:enabled application 'rbd' on pool 'rbd' 2023-05-24T20:58:56.551 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2023-05-24T20:58:56.551 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2023-05-24T20:58:56.552 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2023-05-24T20:58:56.552 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2023-05-24T20:58:56.552 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-05-24T20:58:56.966 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:56.979 INFO:teuthology.orchestra.run.smithi157.stdout:{"epoch":4,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":104382},{"type":"v1","addr":"172.21.15.157:6825","nonce":104382}]},"active_addr":"172.21.15.157:6825/104382","active_change":"2023-05-24T20:58:52.476346+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.157:0","nonce":2208544809}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.157:0","nonce":1009115938}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.157:0","nonce":2699965254}]}]} 2023-05-24T20:58:56.982 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2023-05-24T20:58:56.982 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2023-05-24T20:58:56.983 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2023-05-24T20:58:57.300 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:57.300 INFO:teuthology.orchestra.run.smithi157.stdout:{"epoch":11,"fsid":"946dce5b-d643-4d68-bc45-86cf88c78170","created":"2023-05-24T20:58:46.420452+0000","modified":"2023-05-24T20:58:56.519443+0000","last_up_change":"2023-05-24T20:58:53.515484+0000","last_in_change":"2023-05-24T20:58:47.413150+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-24T20:58:54.536536+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-24T20:58:55.158151+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"44f3e715-b430-4e7d-bb59-24afe3d6e9cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":104794},{"type":"v1","addr":"172.21.15.157:6801","nonce":104794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":104794},{"type":"v1","addr":"172.21.15.157:6803","nonce":104794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":104794},{"type":"v1","addr":"172.21.15.157:6807","nonce":104794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":104794},{"type":"v1","addr":"172.21.15.157:6805","nonce":104794}]},"public_addr":"172.21.15.157:6801/104794","cluster_addr":"172.21.15.157:6803/104794","heartbeat_back_addr":"172.21.15.157:6807/104794","heartbeat_front_addr":"172.21.15.157:6805/104794","state":["exists","up"]},{"osd":1,"uuid":"1e623fb0-b0b3-4df6-9079-e2c9654d11fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":104795},{"type":"v1","addr":"172.21.15.157:6809","nonce":104795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":104795},{"type":"v1","addr":"172.21.15.157:6811","nonce":104795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":104795},{"type":"v1","addr":"172.21.15.157:6815","nonce":104795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":104795},{"type":"v1","addr":"172.21.15.157:6813","nonce":104795}]},"public_addr":"172.21.15.157:6809/104795","cluster_addr":"172.21.15.157:6811/104795","heartbeat_back_addr":"172.21.15.157:6815/104795","heartbeat_front_addr":"172.21.15.157:6813/104795","state":["exists","up"]},{"osd":2,"uuid":"5acf236b-29ec-46ab-a632-10e4551b0322","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":104808},{"type":"v1","addr":"172.21.15.157:6817","nonce":104808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":104808},{"type":"v1","addr":"172.21.15.157:6819","nonce":104808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":104808},{"type":"v1","addr":"172.21.15.157:6823","nonce":104808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":104808},{"type":"v1","addr":"172.21.15.157:6821","nonce":104808}]},"public_addr":"172.21.15.157:6817/104808","cluster_addr":"172.21.15.157:6819/104808","heartbeat_back_addr":"172.21.15.157:6823/104808","heartbeat_front_addr":"172.21.15.157:6821/104808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-24T20:58:51.730481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-24T20:58:52.514508+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-24T20:58:52.508726+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-24T20:58:57.313 INFO:tasks.ceph.ceph_manager.c2:all up! 2023-05-24T20:58:57.314 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2023-05-24T20:58:57.633 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:58:57.633 INFO:teuthology.orchestra.run.smithi157.stdout:{"epoch":12,"fsid":"946dce5b-d643-4d68-bc45-86cf88c78170","created":"2023-05-24T20:58:46.420452+0000","modified":"2023-05-24T20:58:57.523105+0000","last_up_change":"2023-05-24T20:58:53.515484+0000","last_in_change":"2023-05-24T20:58:47.413150+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-24T20:58:54.536536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-24T20:58:55.158151+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"44f3e715-b430-4e7d-bb59-24afe3d6e9cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":104794},{"type":"v1","addr":"172.21.15.157:6801","nonce":104794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":104794},{"type":"v1","addr":"172.21.15.157:6803","nonce":104794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":104794},{"type":"v1","addr":"172.21.15.157:6807","nonce":104794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":104794},{"type":"v1","addr":"172.21.15.157:6805","nonce":104794}]},"public_addr":"172.21.15.157:6801/104794","cluster_addr":"172.21.15.157:6803/104794","heartbeat_back_addr":"172.21.15.157:6807/104794","heartbeat_front_addr":"172.21.15.157:6805/104794","state":["exists","up"]},{"osd":1,"uuid":"1e623fb0-b0b3-4df6-9079-e2c9654d11fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":104795},{"type":"v1","addr":"172.21.15.157:6809","nonce":104795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":104795},{"type":"v1","addr":"172.21.15.157:6811","nonce":104795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":104795},{"type":"v1","addr":"172.21.15.157:6815","nonce":104795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":104795},{"type":"v1","addr":"172.21.15.157:6813","nonce":104795}]},"public_addr":"172.21.15.157:6809/104795","cluster_addr":"172.21.15.157:6811/104795","heartbeat_back_addr":"172.21.15.157:6815/104795","heartbeat_front_addr":"172.21.15.157:6813/104795","state":["exists","up"]},{"osd":2,"uuid":"5acf236b-29ec-46ab-a632-10e4551b0322","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":104808},{"type":"v1","addr":"172.21.15.157:6817","nonce":104808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":104808},{"type":"v1","addr":"172.21.15.157:6819","nonce":104808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":104808},{"type":"v1","addr":"172.21.15.157:6823","nonce":104808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":104808},{"type":"v1","addr":"172.21.15.157:6821","nonce":104808}]},"public_addr":"172.21.15.157:6817/104808","cluster_addr":"172.21.15.157:6819/104808","heartbeat_back_addr":"172.21.15.157:6823/104808","heartbeat_front_addr":"172.21.15.157:6821/104808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-24T20:58:51.730481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-24T20:58:52.514508+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-24T20:58:52.508726+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-24T20:58:57.647 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2023-05-24T20:58:57.648 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2023-05-24T20:58:57.649 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2023-05-24T20:58:57.816 INFO:teuthology.orchestra.run.smithi157.stdout:34359738370 2023-05-24T20:58:57.817 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2023-05-24T20:58:57.822 INFO:teuthology.orchestra.run.smithi157.stdout:34359738370 2023-05-24T20:58:57.823 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2023-05-24T20:58:57.829 INFO:teuthology.orchestra.run.smithi157.stdout:34359738371 2023-05-24T20:58:57.830 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2023-05-24T20:58:58.155 INFO:teuthology.orchestra.run.smithi157.stdout:34359738369 2023-05-24T20:58:58.171 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738369 for osd.2 2023-05-24T20:58:58.204 INFO:teuthology.orchestra.run.smithi157.stdout:34359738369 2023-05-24T20:58:58.210 INFO:teuthology.orchestra.run.smithi157.stdout:34359738370 2023-05-24T20:58:58.220 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738369 for osd.0 2023-05-24T20:58:58.225 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738370 for osd.1 2023-05-24T20:58:59.172 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2023-05-24T20:58:59.221 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2023-05-24T20:58:59.227 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2023-05-24T20:58:59.524 INFO:teuthology.orchestra.run.smithi157.stdout:34359738370 2023-05-24T20:58:59.542 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738370 for osd.2 2023-05-24T20:58:59.543 DEBUG:teuthology.parallel:result is None 2023-05-24T20:58:59.607 INFO:teuthology.orchestra.run.smithi157.stdout:34359738370 2023-05-24T20:58:59.621 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738370 for osd.0 2023-05-24T20:58:59.622 DEBUG:teuthology.parallel:result is None 2023-05-24T20:58:59.640 INFO:teuthology.orchestra.run.smithi157.stdout:34359738371 2023-05-24T20:58:59.654 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.1 2023-05-24T20:58:59.654 DEBUG:teuthology.parallel:result is None 2023-05-24T20:58:59.655 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2023-05-24T20:58:59.655 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2023-05-24T20:59:00.005 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:59:00.008 INFO:teuthology.orchestra.run.smithi157.stderr:dumped all 2023-05-24T20:59:00.019 INFO:teuthology.orchestra.run.smithi157.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2023-05-24T20:58:58.484192+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":25600,"kb_used_data":1456,"kb_used_omap":0,"kb_used_meta":23488,"kb_avail":314547200,"statfs":{"total":322122547200,"available":322096332800,"internally_reserved":0,"allocated":1490944,"data_stored":1259527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":24051712},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-24T20:58:56.529249+0000","last_change":"2023-05-24T20:58:56.528334+0000","last_active":"2023-05-24T20:58:56.529249+0000","last_peered":"2023-05-24T20:58:56.529249+0000","last_clean":"2023-05-24T20:58:56.529249+0000","last_became_active":"2023-05-24T20:58:56.528058+0000","last_became_peered":"2023-05-24T20:58:56.528058+0000","last_unstale":"2023-05-24T20:58:56.529249+0000","last_undegraded":"2023-05-24T20:58:56.529249+0000","last_fullsized":"2023-05-24T20:58:56.529249+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-24T20:58:55.517371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-24T20:58:55.517371+0000","last_clean_scrub_stamp":"2023-05-24T20:58:55.517371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-26T07:04:21.480704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-24T20:58:56.531404+0000","last_change":"2023-05-24T20:58:56.530374+0000","last_active":"2023-05-24T20:58:56.531404+0000","last_peered":"2023-05-24T20:58:56.531404+0000","last_clean":"2023-05-24T20:58:56.531404+0000","last_became_active":"2023-05-24T20:58:56.530093+0000","last_became_peered":"2023-05-24T20:58:56.530093+0000","last_unstale":"2023-05-24T20:58:56.531404+0000","last_undegraded":"2023-05-24T20:58:56.531404+0000","last_fullsized":"2023-05-24T20:58:56.531404+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-24T20:58:55.517371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-24T20:58:55.517371+0000","last_clean_scrub_stamp":"2023-05-24T20:58:55.517371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-26T07:00:36.590815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-24T20:58:56.531453+0000","last_change":"2023-05-24T20:58:56.530428+0000","last_active":"2023-05-24T20:58:56.531453+0000","last_peered":"2023-05-24T20:58:56.531453+0000","last_clean":"2023-05-24T20:58:56.531453+0000","last_became_active":"2023-05-24T20:58:56.530153+0000","last_became_peered":"2023-05-24T20:58:56.530153+0000","last_unstale":"2023-05-24T20:58:56.531453+0000","last_undegraded":"2023-05-24T20:58:56.531453+0000","last_fullsized":"2023-05-24T20:58:56.531453+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-24T20:58:55.517371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-24T20:58:55.517371+0000","last_clean_scrub_stamp":"2023-05-24T20:58:55.517371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-26T02:25:21.506900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-05-24T20:58:57.525023+0000","last_change":"2023-05-24T20:58:56.527994+0000","last_active":"2023-05-24T20:58:57.525023+0000","last_peered":"2023-05-24T20:58:57.525023+0000","last_clean":"2023-05-24T20:58:57.525023+0000","last_became_active":"2023-05-24T20:58:56.527163+0000","last_became_peered":"2023-05-24T20:58:56.527163+0000","last_unstale":"2023-05-24T20:58:57.525023+0000","last_undegraded":"2023-05-24T20:58:57.525023+0000","last_fullsized":"2023-05-24T20:58:57.525023+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-24T20:58:55.517371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-24T20:58:55.517371+0000","last_clean_scrub_stamp":"2023-05-24T20:58:55.517371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-25T21:47:32.867310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":136,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-24T20:58:56.928068+0000","last_change":"2023-05-24T20:58:56.525641+0000","last_active":"2023-05-24T20:58:56.928068+0000","last_peered":"2023-05-24T20:58:56.928068+0000","last_clean":"2023-05-24T20:58:56.928068+0000","last_became_active":"2023-05-24T20:58:56.525380+0000","last_became_peered":"2023-05-24T20:58:56.525380+0000","last_unstale":"2023-05-24T20:58:56.928068+0000","last_undegraded":"2023-05-24T20:58:56.928068+0000","last_fullsized":"2023-05-24T20:58:56.928068+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-24T20:58:55.517371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-24T20:58:55.517371+0000","last_clean_scrub_stamp":"2023-05-24T20:58:55.517371+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-26T05:28:39.173807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-24T20:58:56.528508+0000","last_change":"2023-05-24T20:58:56.527277+0000","last_active":"2023-05-24T20:58:56.528508+0000","last_peered":"2023-05-24T20:58:56.528508+0000","last_clean":"2023-05-24T20:58:56.528508+0000","last_became_active":"2023-05-24T20:58:56.527037+0000","last_became_peered":"2023-05-24T20:58:56.527037+0000","last_unstale":"2023-05-24T20:58:56.528508+0000","last_undegraded":"2023-05-24T20:58:56.528508+0000","last_fullsized":"2023-05-24T20:58:56.528508+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-24T20:58:55.517371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-24T20:58:55.517371+0000","last_clean_scrub_stamp":"2023-05-24T20:58:55.517371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-25T21:30:21.357555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-05-24T20:58:57.525038+0000","last_change":"2023-05-24T20:58:56.527435+0000","last_active":"2023-05-24T20:58:57.525038+0000","last_peered":"2023-05-24T20:58:57.525038+0000","last_clean":"2023-05-24T20:58:57.525038+0000","last_became_active":"2023-05-24T20:58:56.527172+0000","last_became_peered":"2023-05-24T20:58:56.527172+0000","last_unstale":"2023-05-24T20:58:57.525038+0000","last_undegraded":"2023-05-24T20:58:57.525038+0000","last_fullsized":"2023-05-24T20:58:57.525038+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-24T20:58:55.517371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-24T20:58:55.517371+0000","last_clean_scrub_stamp":"2023-05-24T20:58:55.517371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-26T08:20:07.947376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-24T20:58:56.530324+0000","last_change":"2023-05-24T20:58:56.529465+0000","last_active":"2023-05-24T20:58:56.530324+0000","last_peered":"2023-05-24T20:58:56.530324+0000","last_clean":"2023-05-24T20:58:56.530324+0000","last_became_active":"2023-05-24T20:58:56.529186+0000","last_became_peered":"2023-05-24T20:58:56.529186+0000","last_unstale":"2023-05-24T20:58:56.530324+0000","last_undegraded":"2023-05-24T20:58:56.530324+0000","last_fullsized":"2023-05-24T20:58:56.530324+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-24T20:58:55.517371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-24T20:58:55.517371+0000","last_clean_scrub_stamp":"2023-05-24T20:58:55.517371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-26T07:52:00.025028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-24T20:58:56.529275+0000","last_change":"2023-05-24T20:58:56.528307+0000","last_active":"2023-05-24T20:58:56.529275+0000","last_peered":"2023-05-24T20:58:56.529275+0000","last_clean":"2023-05-24T20:58:56.529275+0000","last_became_active":"2023-05-24T20:58:56.528040+0000","last_became_peered":"2023-05-24T20:58:56.528040+0000","last_unstale":"2023-05-24T20:58:56.529275+0000","last_undegraded":"2023-05-24T20:58:56.529275+0000","last_fullsized":"2023-05-24T20:58:56.529275+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-24T20:58:55.517371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-24T20:58:55.517371+0000","last_clean_scrub_stamp":"2023-05-24T20:58:55.517371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-26T08:43:43.651265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7728,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":7616,"kb_avail":104849872,"statfs":{"total":107374182400,"available":107366268928,"internally_reserved":0,"allocated":106496,"data_stored":28829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7798784},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738370,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8936,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848664,"statfs":{"total":107374182400,"available":107365031936,"internally_reserved":0,"allocated":692224,"data_stored":615349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32400000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8936,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848664,"statfs":{"total":107374182400,"available":107365031936,"internally_reserved":0,"allocated":692224,"data_stored":615349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-05-24T20:59:00.021 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2023-05-24T20:59:00.334 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:59:00.336 INFO:teuthology.orchestra.run.smithi157.stderr:dumped all 2023-05-24T20:59:00.348 INFO:teuthology.orchestra.run.smithi157.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2023-05-24T20:58:58.484192+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":25600,"kb_used_data":1456,"kb_used_omap":0,"kb_used_meta":23488,"kb_avail":314547200,"statfs":{"total":322122547200,"available":322096332800,"internally_reserved":0,"allocated":1490944,"data_stored":1259527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":24051712},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-24T20:58:56.529249+0000","last_change":"2023-05-24T20:58:56.528334+0000","last_active":"2023-05-24T20:58:56.529249+0000","last_peered":"2023-05-24T20:58:56.529249+0000","last_clean":"2023-05-24T20:58:56.529249+0000","last_became_active":"2023-05-24T20:58:56.528058+0000","last_became_peered":"2023-05-24T20:58:56.528058+0000","last_unstale":"2023-05-24T20:58:56.529249+0000","last_undegraded":"2023-05-24T20:58:56.529249+0000","last_fullsized":"2023-05-24T20:58:56.529249+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-24T20:58:55.517371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-24T20:58:55.517371+0000","last_clean_scrub_stamp":"2023-05-24T20:58:55.517371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-26T07:04:21.480704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-24T20:58:56.531404+0000","last_change":"2023-05-24T20:58:56.530374+0000","last_active":"2023-05-24T20:58:56.531404+0000","last_peered":"2023-05-24T20:58:56.531404+0000","last_clean":"2023-05-24T20:58:56.531404+0000","last_became_active":"2023-05-24T20:58:56.530093+0000","last_became_peered":"2023-05-24T20:58:56.530093+0000","last_unstale":"2023-05-24T20:58:56.531404+0000","last_undegraded":"2023-05-24T20:58:56.531404+0000","last_fullsized":"2023-05-24T20:58:56.531404+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-24T20:58:55.517371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-24T20:58:55.517371+0000","last_clean_scrub_stamp":"2023-05-24T20:58:55.517371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-26T07:00:36.590815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-24T20:58:56.531453+0000","last_change":"2023-05-24T20:58:56.530428+0000","last_active":"2023-05-24T20:58:56.531453+0000","last_peered":"2023-05-24T20:58:56.531453+0000","last_clean":"2023-05-24T20:58:56.531453+0000","last_became_active":"2023-05-24T20:58:56.530153+0000","last_became_peered":"2023-05-24T20:58:56.530153+0000","last_unstale":"2023-05-24T20:58:56.531453+0000","last_undegraded":"2023-05-24T20:58:56.531453+0000","last_fullsized":"2023-05-24T20:58:56.531453+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-24T20:58:55.517371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-24T20:58:55.517371+0000","last_clean_scrub_stamp":"2023-05-24T20:58:55.517371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-26T02:25:21.506900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-05-24T20:58:57.525023+0000","last_change":"2023-05-24T20:58:56.527994+0000","last_active":"2023-05-24T20:58:57.525023+0000","last_peered":"2023-05-24T20:58:57.525023+0000","last_clean":"2023-05-24T20:58:57.525023+0000","last_became_active":"2023-05-24T20:58:56.527163+0000","last_became_peered":"2023-05-24T20:58:56.527163+0000","last_unstale":"2023-05-24T20:58:57.525023+0000","last_undegraded":"2023-05-24T20:58:57.525023+0000","last_fullsized":"2023-05-24T20:58:57.525023+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-24T20:58:55.517371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-24T20:58:55.517371+0000","last_clean_scrub_stamp":"2023-05-24T20:58:55.517371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-25T21:47:32.867310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":136,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-24T20:58:56.928068+0000","last_change":"2023-05-24T20:58:56.525641+0000","last_active":"2023-05-24T20:58:56.928068+0000","last_peered":"2023-05-24T20:58:56.928068+0000","last_clean":"2023-05-24T20:58:56.928068+0000","last_became_active":"2023-05-24T20:58:56.525380+0000","last_became_peered":"2023-05-24T20:58:56.525380+0000","last_unstale":"2023-05-24T20:58:56.928068+0000","last_undegraded":"2023-05-24T20:58:56.928068+0000","last_fullsized":"2023-05-24T20:58:56.928068+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-24T20:58:55.517371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-24T20:58:55.517371+0000","last_clean_scrub_stamp":"2023-05-24T20:58:55.517371+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-26T05:28:39.173807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-24T20:58:56.528508+0000","last_change":"2023-05-24T20:58:56.527277+0000","last_active":"2023-05-24T20:58:56.528508+0000","last_peered":"2023-05-24T20:58:56.528508+0000","last_clean":"2023-05-24T20:58:56.528508+0000","last_became_active":"2023-05-24T20:58:56.527037+0000","last_became_peered":"2023-05-24T20:58:56.527037+0000","last_unstale":"2023-05-24T20:58:56.528508+0000","last_undegraded":"2023-05-24T20:58:56.528508+0000","last_fullsized":"2023-05-24T20:58:56.528508+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-24T20:58:55.517371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-24T20:58:55.517371+0000","last_clean_scrub_stamp":"2023-05-24T20:58:55.517371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-25T21:30:21.357555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-05-24T20:58:57.525038+0000","last_change":"2023-05-24T20:58:56.527435+0000","last_active":"2023-05-24T20:58:57.525038+0000","last_peered":"2023-05-24T20:58:57.525038+0000","last_clean":"2023-05-24T20:58:57.525038+0000","last_became_active":"2023-05-24T20:58:56.527172+0000","last_became_peered":"2023-05-24T20:58:56.527172+0000","last_unstale":"2023-05-24T20:58:57.525038+0000","last_undegraded":"2023-05-24T20:58:57.525038+0000","last_fullsized":"2023-05-24T20:58:57.525038+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-24T20:58:55.517371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-24T20:58:55.517371+0000","last_clean_scrub_stamp":"2023-05-24T20:58:55.517371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-26T08:20:07.947376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-24T20:58:56.530324+0000","last_change":"2023-05-24T20:58:56.529465+0000","last_active":"2023-05-24T20:58:56.530324+0000","last_peered":"2023-05-24T20:58:56.530324+0000","last_clean":"2023-05-24T20:58:56.530324+0000","last_became_active":"2023-05-24T20:58:56.529186+0000","last_became_peered":"2023-05-24T20:58:56.529186+0000","last_unstale":"2023-05-24T20:58:56.530324+0000","last_undegraded":"2023-05-24T20:58:56.530324+0000","last_fullsized":"2023-05-24T20:58:56.530324+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-24T20:58:55.517371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-24T20:58:55.517371+0000","last_clean_scrub_stamp":"2023-05-24T20:58:55.517371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-26T07:52:00.025028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-24T20:58:56.529275+0000","last_change":"2023-05-24T20:58:56.528307+0000","last_active":"2023-05-24T20:58:56.529275+0000","last_peered":"2023-05-24T20:58:56.529275+0000","last_clean":"2023-05-24T20:58:56.529275+0000","last_became_active":"2023-05-24T20:58:56.528040+0000","last_became_peered":"2023-05-24T20:58:56.528040+0000","last_unstale":"2023-05-24T20:58:56.529275+0000","last_undegraded":"2023-05-24T20:58:56.529275+0000","last_fullsized":"2023-05-24T20:58:56.529275+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-24T20:58:55.517371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-24T20:58:55.517371+0000","last_clean_scrub_stamp":"2023-05-24T20:58:55.517371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-26T08:43:43.651265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7728,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":7616,"kb_avail":104849872,"statfs":{"total":107374182400,"available":107366268928,"internally_reserved":0,"allocated":106496,"data_stored":28829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7798784},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738370,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8936,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848664,"statfs":{"total":107374182400,"available":107365031936,"internally_reserved":0,"allocated":692224,"data_stored":615349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32400000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8936,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848664,"statfs":{"total":107374182400,"available":107365031936,"internally_reserved":0,"allocated":692224,"data_stored":615349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-05-24T20:59:00.349 INFO:tasks.ceph.ceph_manager.c2:clean! 2023-05-24T20:59:00.350 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2023-05-24T20:59:00.350 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2023-05-24T20:59:00.350 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2023-05-24T20:59:00.708 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-05-24T20:59:00.708 INFO:teuthology.orchestra.run.smithi157.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-05-24T20:59:00.722 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2023-05-24T20:59:00.722 INFO:teuthology.run_tasks:Running task rgw... 2023-05-24T20:59:00.736 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2023-05-24T20:59:00.737 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2023-05-24T20:59:00.737 INFO:tasks.rgw:Creating data pools 2023-05-24T20:59:00.738 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2023-05-24T20:59:00.738 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2023-05-24T20:59:01.758 INFO:teuthology.orchestra.run.smithi071.stderr:pool 'default.rgw.buckets.data' created 2023-05-24T20:59:01.774 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2023-05-24T20:59:02.779 INFO:teuthology.orchestra.run.smithi071.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-05-24T20:59:02.794 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2023-05-24T20:59:03.771 INFO:teuthology.orchestra.run.smithi071.stderr:pool 'default.rgw.buckets.index' created 2023-05-24T20:59:03.784 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2023-05-24T20:59:04.789 INFO:teuthology.orchestra.run.smithi071.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-05-24T20:59:04.801 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2023-05-24T20:59:04.802 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2023-05-24T20:59:05.122 INFO:teuthology.orchestra.run.smithi071.stderr:pool 'default.rgw.buckets.data' already exists 2023-05-24T20:59:05.133 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2023-05-24T20:59:05.767 INFO:teuthology.orchestra.run.smithi071.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-05-24T20:59:05.778 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2023-05-24T20:59:06.100 INFO:teuthology.orchestra.run.smithi071.stderr:pool 'default.rgw.buckets.index' already exists 2023-05-24T20:59:06.111 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2023-05-24T20:59:06.773 INFO:teuthology.orchestra.run.smithi071.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-05-24T20:59:06.785 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2023-05-24T20:59:06.786 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2023-05-24T20:59:07.414 INFO:teuthology.orchestra.run.smithi157.stderr:pool 'default.rgw.buckets.data' created 2023-05-24T20:59:07.427 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2023-05-24T20:59:08.442 INFO:teuthology.orchestra.run.smithi157.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-05-24T20:59:08.450 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2023-05-24T20:59:09.420 INFO:teuthology.orchestra.run.smithi157.stderr:pool 'default.rgw.buckets.index' created 2023-05-24T20:59:09.436 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2023-05-24T20:59:10.430 INFO:teuthology.orchestra.run.smithi157.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-05-24T20:59:10.449 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2023-05-24T20:59:10.450 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2023-05-24T20:59:10.774 INFO:teuthology.orchestra.run.smithi157.stderr:pool 'default.rgw.buckets.data' already exists 2023-05-24T20:59:10.785 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2023-05-24T20:59:11.421 INFO:teuthology.orchestra.run.smithi157.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-05-24T20:59:11.433 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2023-05-24T20:59:11.753 INFO:teuthology.orchestra.run.smithi157.stderr:pool 'default.rgw.buckets.index' already exists 2023-05-24T20:59:11.764 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2023-05-24T20:59:12.422 INFO:teuthology.orchestra.run.smithi157.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-05-24T20:59:12.434 DEBUG:tasks.rgw:Pools created 2023-05-24T20:59:12.434 INFO:tasks.rgw:Configuring compression type = random 2023-05-24T20:59:12.435 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2023-05-24T20:59:12.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2023-05-24T20:59:12.435 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2023-05-24T20:59:12.560 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T20:59:12.560 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T20:59:12.570 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:12.564+0000 7fd5a4899780 20 rados->read ofs=0 len=0 2023-05-24T20:59:13.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:13.686+0000 7fd5a4899780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-24T20:59:13.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:13.686+0000 7fd5a4899780 20 realm 2023-05-24T20:59:13.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:13.686+0000 7fd5a4899780 20 rados->read ofs=0 len=0 2023-05-24T20:59:13.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:13.687+0000 7fd5a4899780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-24T20:59:13.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:13.687+0000 7fd5a4899780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-05-24T20:59:13.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:13.687+0000 7fd5a4899780 20 rados->read ofs=0 len=0 2023-05-24T20:59:13.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:13.687+0000 7fd5a4899780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-24T20:59:13.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:13.687+0000 7fd5a4899780 20 rados->read ofs=0 len=0 2023-05-24T20:59:13.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:13.687+0000 7fd5a4899780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-24T20:59:13.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:13.687+0000 7fd5a4899780 20 rados->read ofs=0 len=0 2023-05-24T20:59:13.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:13.687+0000 7fd5a4899780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-24T20:59:13.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:13.687+0000 7fd5a4899780 20 rados->read ofs=0 len=0 2023-05-24T20:59:13.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:13.688+0000 7fd5a4899780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-24T20:59:13.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:13.688+0000 7fd5a4899780 10 Creating default zonegroup 2023-05-24T20:59:13.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:13.688+0000 7fd5a4899780 10 couldn't find old data placement pools config, setting up new ones for the zone 2023-05-24T20:59:13.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:13.697+0000 7fd5a4899780 10 WARNING: driver->list_zones() returned r=-2 2023-05-24T20:59:13.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:13.697+0000 7fd5a4899780 20 rados->read ofs=0 len=0 2023-05-24T20:59:13.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:13.697+0000 7fd5a4899780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-24T20:59:13.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:13.701+0000 7fd5a4899780 20 rados->read ofs=0 len=0 2023-05-24T20:59:13.708 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:13.701+0000 7fd5a4899780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-24T20:59:13.708 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:13.701+0000 7fd5a4899780 10 could not read realm id: (2) No such file or directory 2023-05-24T20:59:13.708 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:13.701+0000 7fd5a4899780 10 WARNING: failed to set zone as default, r=-22 2023-05-24T20:59:13.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:13.701+0000 7fd5a4899780 20 rados->read ofs=0 len=0 2023-05-24T20:59:13.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:13.701+0000 7fd5a4899780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-24T20:59:13.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:13.706+0000 7fd5a4899780 20 rados->read ofs=0 len=0 2023-05-24T20:59:13.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:13.707+0000 7fd5a4899780 20 rados_obj.operate() r=0 bl.length=874 2023-05-24T20:59:13.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:13.707+0000 7fd5a4899780 20 rados->read ofs=0 len=0 2023-05-24T20:59:13.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:13.710+0000 7fd5a4899780 20 rados_obj.operate() r=0 bl.length=398 2023-05-24T20:59:13.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:13.710+0000 7fd5a4899780 10 Using default name default 2023-05-24T20:59:13.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:13.710+0000 7fd5a4899780 20 rados->read ofs=0 len=0 2023-05-24T20:59:13.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:13.711+0000 7fd5a4899780 20 rados_obj.operate() r=0 bl.length=46 2023-05-24T20:59:13.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:13.711+0000 7fd5a4899780 20 rados->read ofs=0 len=0 2023-05-24T20:59:13.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:13.711+0000 7fd5a4899780 20 rados_obj.operate() r=0 bl.length=874 2023-05-24T20:59:13.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:13.711+0000 7fd5a4899780 20 zone default found 2023-05-24T20:59:13.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:13.711+0000 7fd5a4899780 4 Realm: () 2023-05-24T20:59:13.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:13.711+0000 7fd5a4899780 4 ZoneGroup: default (021b038f-1add-4dfa-98be-7c0e2a548b4c) 2023-05-24T20:59:13.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:13.711+0000 7fd5a4899780 4 Zone: default (d8e90b5c-9231-4987-9ad9-41ae93a71a62) 2023-05-24T20:59:13.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:13.711+0000 7fd5a4899780 10 cannot find current period zonegroup using local zonegroup configuration 2023-05-24T20:59:13.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:13.711+0000 7fd5a4899780 20 zonegroup default 2023-05-24T20:59:13.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:13.711+0000 7fd5a4899780 20 rados->read ofs=0 len=0 2023-05-24T20:59:13.719 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:13.711+0000 7fd5a4899780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-24T20:59:13.719 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:13.711+0000 7fd5a4899780 20 rados->read ofs=0 len=0 2023-05-24T20:59:15.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:15.694+0000 7fd5a4899780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-24T20:59:15.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:15.695+0000 7fd5a4899780 20 rados->read ofs=0 len=0 2023-05-24T20:59:15.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:15.696+0000 7fd5a4899780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-24T20:59:15.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:15.696+0000 7fd5a4899780 20 started sync module instance, tier type = 2023-05-24T20:59:15.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:15.696+0000 7fd5a4899780 20 started zone id=d8e90b5c-9231-4987-9ad9-41ae93a71a62 (name=default) with tier type = 2023-05-24T20:59:17.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:17.734+0000 7fd5a4899780 20 add_watcher() i=0 2023-05-24T20:59:17.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:17.734+0000 7fd5a4899780 20 add_watcher() i=1 2023-05-24T20:59:17.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:17.734+0000 7fd5a4899780 20 add_watcher() i=2 2023-05-24T20:59:17.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:17.734+0000 7fd5a4899780 20 add_watcher() i=3 2023-05-24T20:59:17.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:17.734+0000 7fd5a4899780 20 add_watcher() i=4 2023-05-24T20:59:17.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:17.734+0000 7fd5a4899780 20 add_watcher() i=5 2023-05-24T20:59:17.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:17.734+0000 7fd5a4899780 20 add_watcher() i=6 2023-05-24T20:59:17.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:17.734+0000 7fd5a4899780 20 add_watcher() i=7 2023-05-24T20:59:17.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:17.734+0000 7fd5a4899780 2 all 8 watchers are set, enabling cache 2023-05-24T20:59:17.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:17.735+0000 7fd5a4899780 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2023-05-24T20:59:17.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:17.735+0000 7fd5a4899780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2023-05-24T20:59:17.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:17.735+0000 7fd5a4899780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-05-24T20:59:17.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:17.735+0000 7fd5a4899780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2023-05-24T20:59:17.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:17.735+0000 7fd5a4899780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2023-05-24T20:59:17.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:17.735+0000 7fd5a4899780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-05-24T20:59:17.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:17.735+0000 7fd5a4899780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2023-05-24T20:59:17.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:17.735+0000 7fd5a4899780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2023-05-24T20:59:17.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:17.735+0000 7fd5a4899780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-05-24T20:59:17.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:17.735+0000 7fd5a4899780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2023-05-24T20:59:17.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:17.735+0000 7fd5a4899780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2023-05-24T20:59:17.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:17.735+0000 7fd5a4899780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-05-24T20:59:17.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:17.736+0000 7fd5a4899780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2023-05-24T20:59:17.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:17.736+0000 7fd5a4899780 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1210 entering 2023-05-24T20:59:17.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:17.737+0000 7fd5a4899780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-05-24T20:59:17.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:17.739+0000 7fd589815700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-05-24T20:59:17.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:17.739+0000 7fd5a4899780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-05-24T20:59:17.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:17.739+0000 7fd5a4899780 20 check_secure_mon_conn(): mode 1 is insecure 2023-05-24T20:59:20.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.707+0000 7fd5a4899780 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2023-05-24T20:59:20.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.707+0000 7fd5a4899780 5 note: GC not initialized 2023-05-24T20:59:20.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.708+0000 7fd5a4899780 20 init_complete bucket index max shards: 11 2023-05-24T20:59:20.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.709+0000 7fd5a4899780 10 rgw notify: Started notification manager with: 1 workers 2023-05-24T20:59:20.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.709+0000 7fd583008700 20 rgw notify: INFO: next queues processing will happen at: Wed May 24 20:59:51 2023 2023-05-24T20:59:20.716 INFO:teuthology.orchestra.run.smithi071.stderr: 2023-05-24T20:59:20.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.709+0000 7fd5827e6700 20 reqs_thread_entry: start 2023-05-24T20:59:20.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.710+0000 7fd58500c700 20 reqs_thread_entry: start 2023-05-24T20:59:20.739 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.733+0000 7fd589815700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-05-24T20:59:20.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.737+0000 7fd5a4899780 20 remove_watcher() i=0 2023-05-24T20:59:20.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.737+0000 7fd5a4899780 2 removed watcher, disabling cache 2023-05-24T20:59:20.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.740+0000 7fd5a4899780 20 remove_watcher() i=1 2023-05-24T20:59:20.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.741+0000 7fd5a4899780 20 remove_watcher() i=2 2023-05-24T20:59:20.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.743+0000 7fd5a4899780 20 remove_watcher() i=3 2023-05-24T20:59:20.750 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.744+0000 7fd5a4899780 20 remove_watcher() i=4 2023-05-24T20:59:20.751 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.745+0000 7fd5a4899780 20 remove_watcher() i=5 2023-05-24T20:59:20.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.748+0000 7fd5a4899780 20 remove_watcher() i=6 2023-05-24T20:59:20.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.749+0000 7fd5a4899780 20 remove_watcher() i=7 2023-05-24T20:59:20.758 INFO:teuthology.orchestra.run.smithi071.stdout:[] 2023-05-24T20:59:20.759 DEBUG:tasks.util.rgw: json result: [] 2023-05-24T20:59:20.760 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-05-24T20:59:20.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-05-24T20:59:20.760 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-05-24T20:59:20.815 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T20:59:20.815 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T20:59:20.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.818+0000 7f33b2a2f780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-05-24T20:59:20.825 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.818+0000 7f33b2a2f780 20 check_secure_mon_conn(): mode 1 is insecure 2023-05-24T20:59:20.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.822+0000 7f33941a4700 20 reqs_thread_entry: start 2023-05-24T20:59:20.834 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T20:59:20.834 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "d8e90b5c-9231-4987-9ad9-41ae93a71a62", 2023-05-24T20:59:20.834 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "default", 2023-05-24T20:59:20.834 INFO:teuthology.orchestra.run.smithi071.stdout: "domain_root": "default.rgw.meta:root", 2023-05-24T20:59:20.834 INFO:teuthology.orchestra.run.smithi071.stdout: "control_pool": "default.rgw.control", 2023-05-24T20:59:20.834 INFO:teuthology.orchestra.run.smithi071.stdout: "gc_pool": "default.rgw.log:gc", 2023-05-24T20:59:20.835 INFO:teuthology.orchestra.run.smithi071.stdout: "lc_pool": "default.rgw.log:lc", 2023-05-24T20:59:20.835 INFO:teuthology.orchestra.run.smithi071.stdout: "log_pool": "default.rgw.log", 2023-05-24T20:59:20.835 INFO:teuthology.orchestra.run.smithi071.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-05-24T20:59:20.835 INFO:teuthology.orchestra.run.smithi071.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-05-24T20:59:20.835 INFO:teuthology.orchestra.run.smithi071.stdout: "roles_pool": "default.rgw.meta:roles", 2023-05-24T20:59:20.835 INFO:teuthology.orchestra.run.smithi071.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-05-24T20:59:20.835 INFO:teuthology.orchestra.run.smithi071.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-05-24T20:59:20.835 INFO:teuthology.orchestra.run.smithi071.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-05-24T20:59:20.836 INFO:teuthology.orchestra.run.smithi071.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-05-24T20:59:20.836 INFO:teuthology.orchestra.run.smithi071.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-05-24T20:59:20.836 INFO:teuthology.orchestra.run.smithi071.stdout: "otp_pool": "default.rgw.otp", 2023-05-24T20:59:20.836 INFO:teuthology.orchestra.run.smithi071.stdout: "system_key": { 2023-05-24T20:59:20.836 INFO:teuthology.orchestra.run.smithi071.stdout: "access_key": "", 2023-05-24T20:59:20.836 INFO:teuthology.orchestra.run.smithi071.stdout: "secret_key": "" 2023-05-24T20:59:20.836 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T20:59:20.836 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_pools": [ 2023-05-24T20:59:20.837 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T20:59:20.837 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "default-placement", 2023-05-24T20:59:20.837 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T20:59:20.837 INFO:teuthology.orchestra.run.smithi071.stdout: "index_pool": "default.rgw.buckets.index", 2023-05-24T20:59:20.837 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": { 2023-05-24T20:59:20.837 INFO:teuthology.orchestra.run.smithi071.stdout: "STANDARD": { 2023-05-24T20:59:20.837 INFO:teuthology.orchestra.run.smithi071.stdout: "data_pool": "default.rgw.buckets.data", 2023-05-24T20:59:20.838 INFO:teuthology.orchestra.run.smithi071.stdout: "compression_type": "random" 2023-05-24T20:59:20.838 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T20:59:20.838 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T20:59:20.838 INFO:teuthology.orchestra.run.smithi071.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-05-24T20:59:20.839 INFO:teuthology.orchestra.run.smithi071.stdout: "index_type": 0, 2023-05-24T20:59:20.839 INFO:teuthology.orchestra.run.smithi071.stdout: "inline_data": true 2023-05-24T20:59:20.839 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T20:59:20.839 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T20:59:20.839 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T20:59:20.839 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "", 2023-05-24T20:59:20.839 INFO:teuthology.orchestra.run.smithi071.stdout: "notif_pool": "default.rgw.log:notif" 2023-05-24T20:59:20.839 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T20:59:20.840 DEBUG:tasks.util.rgw: json result: {'id': 'd8e90b5c-9231-4987-9ad9-41ae93a71a62', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-05-24T20:59:20.841 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2023-05-24T20:59:20.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2023-05-24T20:59:20.841 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2023-05-24T20:59:20.886 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T20:59:20.886 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T20:59:20.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.888+0000 7f61d1b05780 20 rados->read ofs=0 len=0 2023-05-24T20:59:20.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.889+0000 7f61d1b05780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-24T20:59:20.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.889+0000 7f61d1b05780 20 realm 2023-05-24T20:59:20.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.889+0000 7f61d1b05780 20 rados->read ofs=0 len=0 2023-05-24T20:59:20.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.889+0000 7f61d1b05780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-24T20:59:20.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.889+0000 7f61d1b05780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-05-24T20:59:20.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.889+0000 7f61d1b05780 20 rados->read ofs=0 len=0 2023-05-24T20:59:20.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.889+0000 7f61d1b05780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-24T20:59:20.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.889+0000 7f61d1b05780 20 rados->read ofs=0 len=0 2023-05-24T20:59:20.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.889+0000 7f61d1b05780 20 rados_obj.operate() r=0 bl.length=46 2023-05-24T20:59:20.897 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.889+0000 7f61d1b05780 20 rados->read ofs=0 len=0 2023-05-24T20:59:20.897 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.890+0000 7f61d1b05780 20 rados_obj.operate() r=0 bl.length=890 2023-05-24T20:59:20.898 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.890+0000 7f61d1b05780 20 searching for the correct realm 2023-05-24T20:59:20.903 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.897+0000 7f61d1b05780 20 RGWRados::pool_iterate: got zonegroup_info.021b038f-1add-4dfa-98be-7c0e2a548b4c 2023-05-24T20:59:20.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.897+0000 7f61d1b05780 20 RGWRados::pool_iterate: got zone_info.d8e90b5c-9231-4987-9ad9-41ae93a71a62 2023-05-24T20:59:20.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.897+0000 7f61d1b05780 20 RGWRados::pool_iterate: got zone_names.default 2023-05-24T20:59:20.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.897+0000 7f61d1b05780 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-05-24T20:59:20.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.897+0000 7f61d1b05780 20 rados->read ofs=0 len=0 2023-05-24T20:59:20.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.897+0000 7f61d1b05780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-24T20:59:20.905 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.897+0000 7f61d1b05780 20 rados->read ofs=0 len=0 2023-05-24T20:59:20.905 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.897+0000 7f61d1b05780 20 rados_obj.operate() r=0 bl.length=46 2023-05-24T20:59:20.905 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.897+0000 7f61d1b05780 20 rados->read ofs=0 len=0 2023-05-24T20:59:20.906 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.897+0000 7f61d1b05780 20 rados_obj.operate() r=0 bl.length=398 2023-05-24T20:59:20.907 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.897+0000 7f61d1b05780 20 zone default found 2023-05-24T20:59:20.907 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.897+0000 7f61d1b05780 4 Realm: () 2023-05-24T20:59:20.907 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.897+0000 7f61d1b05780 4 ZoneGroup: default (021b038f-1add-4dfa-98be-7c0e2a548b4c) 2023-05-24T20:59:20.907 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.897+0000 7f61d1b05780 4 Zone: default (d8e90b5c-9231-4987-9ad9-41ae93a71a62) 2023-05-24T20:59:20.907 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.897+0000 7f61d1b05780 10 cannot find current period zonegroup using local zonegroup configuration 2023-05-24T20:59:20.907 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.897+0000 7f61d1b05780 20 zonegroup default 2023-05-24T20:59:20.907 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.897+0000 7f61d1b05780 20 rados->read ofs=0 len=0 2023-05-24T20:59:20.908 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.898+0000 7f61d1b05780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-24T20:59:20.908 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.898+0000 7f61d1b05780 20 rados->read ofs=0 len=0 2023-05-24T20:59:20.908 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.898+0000 7f61d1b05780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-24T20:59:20.908 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.898+0000 7f61d1b05780 20 rados->read ofs=0 len=0 2023-05-24T20:59:20.908 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.898+0000 7f61d1b05780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-24T20:59:20.908 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.898+0000 7f61d1b05780 20 started sync module instance, tier type = 2023-05-24T20:59:20.908 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.898+0000 7f61d1b05780 20 started zone id=d8e90b5c-9231-4987-9ad9-41ae93a71a62 (name=default) with tier type = 2023-05-24T20:59:20.921 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.915+0000 7f61d1b05780 20 add_watcher() i=0 2023-05-24T20:59:20.921 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.915+0000 7f61d1b05780 20 add_watcher() i=1 2023-05-24T20:59:20.921 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.915+0000 7f61d1b05780 20 add_watcher() i=2 2023-05-24T20:59:20.922 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.915+0000 7f61d1b05780 20 add_watcher() i=3 2023-05-24T20:59:20.922 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.915+0000 7f61d1b05780 20 add_watcher() i=4 2023-05-24T20:59:20.922 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.915+0000 7f61d1b05780 20 add_watcher() i=5 2023-05-24T20:59:20.922 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.915+0000 7f61d1b05780 20 add_watcher() i=6 2023-05-24T20:59:20.922 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.916+0000 7f61d1b05780 20 add_watcher() i=7 2023-05-24T20:59:20.923 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.916+0000 7f61d1b05780 2 all 8 watchers are set, enabling cache 2023-05-24T20:59:20.923 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.917+0000 7f61b6a81700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-05-24T20:59:20.923 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.917+0000 7f61d1b05780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-05-24T20:59:20.923 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.917+0000 7f61d1b05780 20 check_secure_mon_conn(): mode 1 is insecure 2023-05-24T20:59:20.926 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.921+0000 7f61d1b05780 5 note: GC not initialized 2023-05-24T20:59:20.927 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.922+0000 7f61b2278700 20 reqs_thread_entry: start 2023-05-24T20:59:20.928 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.922+0000 7f61d1b05780 20 init_complete bucket index max shards: 11 2023-05-24T20:59:20.928 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.922+0000 7f61d1b05780 10 rgw notify: Started notification manager with: 1 workers 2023-05-24T20:59:20.928 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.922+0000 7f61b0274700 20 rgw notify: INFO: next queues processing will happen at: Wed May 24 20:59:51 2023 2023-05-24T20:59:20.928 INFO:teuthology.orchestra.run.smithi071.stderr: 2023-05-24T20:59:20.928 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.922+0000 7f61afa52700 20 reqs_thread_entry: start 2023-05-24T20:59:20.935 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.929+0000 7f61b6a81700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-05-24T20:59:20.936 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.931+0000 7f61d1b05780 20 remove_watcher() i=0 2023-05-24T20:59:20.937 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.931+0000 7f61d1b05780 2 removed watcher, disabling cache 2023-05-24T20:59:20.937 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.932+0000 7f61d1b05780 20 remove_watcher() i=1 2023-05-24T20:59:20.940 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.934+0000 7f61d1b05780 20 remove_watcher() i=2 2023-05-24T20:59:20.941 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.935+0000 7f61d1b05780 20 remove_watcher() i=3 2023-05-24T20:59:20.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.936+0000 7f61d1b05780 20 remove_watcher() i=4 2023-05-24T20:59:20.943 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.937+0000 7f61d1b05780 20 remove_watcher() i=5 2023-05-24T20:59:20.944 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.938+0000 7f61d1b05780 20 remove_watcher() i=6 2023-05-24T20:59:20.945 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:20.939+0000 7f61d1b05780 20 remove_watcher() i=7 2023-05-24T20:59:20.947 INFO:teuthology.orchestra.run.smithi071.stdout:[] 2023-05-24T20:59:20.949 DEBUG:tasks.util.rgw: json result: [] 2023-05-24T20:59:20.949 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-05-24T20:59:20.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-05-24T20:59:20.950 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-05-24T20:59:21.000 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T20:59:21.001 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T20:59:21.009 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:21.003+0000 7f1f3ab17780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-05-24T20:59:21.009 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:21.003+0000 7f1f3ab17780 20 check_secure_mon_conn(): mode 1 is insecure 2023-05-24T20:59:21.012 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:21.006+0000 7f1f1c28c700 20 reqs_thread_entry: start 2023-05-24T20:59:21.018 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T20:59:21.019 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "d8e90b5c-9231-4987-9ad9-41ae93a71a62", 2023-05-24T20:59:21.019 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "default", 2023-05-24T20:59:21.019 INFO:teuthology.orchestra.run.smithi071.stdout: "domain_root": "default.rgw.meta:root", 2023-05-24T20:59:21.019 INFO:teuthology.orchestra.run.smithi071.stdout: "control_pool": "default.rgw.control", 2023-05-24T20:59:21.019 INFO:teuthology.orchestra.run.smithi071.stdout: "gc_pool": "default.rgw.log:gc", 2023-05-24T20:59:21.019 INFO:teuthology.orchestra.run.smithi071.stdout: "lc_pool": "default.rgw.log:lc", 2023-05-24T20:59:21.020 INFO:teuthology.orchestra.run.smithi071.stdout: "log_pool": "default.rgw.log", 2023-05-24T20:59:21.020 INFO:teuthology.orchestra.run.smithi071.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-05-24T20:59:21.020 INFO:teuthology.orchestra.run.smithi071.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-05-24T20:59:21.020 INFO:teuthology.orchestra.run.smithi071.stdout: "roles_pool": "default.rgw.meta:roles", 2023-05-24T20:59:21.020 INFO:teuthology.orchestra.run.smithi071.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-05-24T20:59:21.020 INFO:teuthology.orchestra.run.smithi071.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-05-24T20:59:21.020 INFO:teuthology.orchestra.run.smithi071.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-05-24T20:59:21.020 INFO:teuthology.orchestra.run.smithi071.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-05-24T20:59:21.021 INFO:teuthology.orchestra.run.smithi071.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-05-24T20:59:21.021 INFO:teuthology.orchestra.run.smithi071.stdout: "otp_pool": "default.rgw.otp", 2023-05-24T20:59:21.021 INFO:teuthology.orchestra.run.smithi071.stdout: "system_key": { 2023-05-24T20:59:21.021 INFO:teuthology.orchestra.run.smithi071.stdout: "access_key": "", 2023-05-24T20:59:21.021 INFO:teuthology.orchestra.run.smithi071.stdout: "secret_key": "" 2023-05-24T20:59:21.021 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T20:59:21.021 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_pools": [ 2023-05-24T20:59:21.022 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T20:59:21.022 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "default-placement", 2023-05-24T20:59:21.022 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T20:59:21.022 INFO:teuthology.orchestra.run.smithi071.stdout: "index_pool": "default.rgw.buckets.index", 2023-05-24T20:59:21.022 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": { 2023-05-24T20:59:21.022 INFO:teuthology.orchestra.run.smithi071.stdout: "STANDARD": { 2023-05-24T20:59:21.022 INFO:teuthology.orchestra.run.smithi071.stdout: "data_pool": "default.rgw.buckets.data", 2023-05-24T20:59:21.022 INFO:teuthology.orchestra.run.smithi071.stdout: "compression_type": "random" 2023-05-24T20:59:21.023 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T20:59:21.023 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T20:59:21.023 INFO:teuthology.orchestra.run.smithi071.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-05-24T20:59:21.023 INFO:teuthology.orchestra.run.smithi071.stdout: "index_type": 0, 2023-05-24T20:59:21.023 INFO:teuthology.orchestra.run.smithi071.stdout: "inline_data": true 2023-05-24T20:59:21.023 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T20:59:21.023 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T20:59:21.023 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T20:59:21.024 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "", 2023-05-24T20:59:21.024 INFO:teuthology.orchestra.run.smithi071.stdout: "notif_pool": "default.rgw.log:notif" 2023-05-24T20:59:21.024 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T20:59:21.025 DEBUG:tasks.util.rgw: json result: {'id': 'd8e90b5c-9231-4987-9ad9-41ae93a71a62', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-05-24T20:59:21.026 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2023-05-24T20:59:21.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2023-05-24T20:59:21.026 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2023-05-24T20:59:21.144 INFO:teuthology.orchestra.run.smithi157.stderr:ignoring --setuser ceph since I am not root 2023-05-24T20:59:21.145 INFO:teuthology.orchestra.run.smithi157.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T20:59:21.156 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:21.149+0000 7ff9e1bbb780 20 rados->read ofs=0 len=0 2023-05-24T20:59:22.568 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:22.561+0000 7ff9e1bbb780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-24T20:59:22.569 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:22.561+0000 7ff9e1bbb780 20 realm 2023-05-24T20:59:22.569 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:22.561+0000 7ff9e1bbb780 20 rados->read ofs=0 len=0 2023-05-24T20:59:22.570 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:22.562+0000 7ff9e1bbb780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-24T20:59:22.570 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:22.562+0000 7ff9e1bbb780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-05-24T20:59:22.570 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:22.563+0000 7ff9e1bbb780 20 rados->read ofs=0 len=0 2023-05-24T20:59:22.570 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:22.563+0000 7ff9e1bbb780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-24T20:59:22.570 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:22.563+0000 7ff9e1bbb780 20 rados->read ofs=0 len=0 2023-05-24T20:59:22.574 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:22.568+0000 7ff9e1bbb780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-24T20:59:22.575 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:22.568+0000 7ff9e1bbb780 20 rados->read ofs=0 len=0 2023-05-24T20:59:22.576 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:22.568+0000 7ff9e1bbb780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-24T20:59:22.576 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:22.569+0000 7ff9e1bbb780 20 rados->read ofs=0 len=0 2023-05-24T20:59:22.576 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:22.569+0000 7ff9e1bbb780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-24T20:59:22.577 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:22.569+0000 7ff9e1bbb780 10 Creating default zonegroup 2023-05-24T20:59:22.577 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:22.569+0000 7ff9e1bbb780 10 couldn't find old data placement pools config, setting up new ones for the zone 2023-05-24T20:59:22.585 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:22.579+0000 7ff9e1bbb780 10 WARNING: driver->list_zones() returned r=-2 2023-05-24T20:59:22.586 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:22.579+0000 7ff9e1bbb780 20 rados->read ofs=0 len=0 2023-05-24T20:59:22.586 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:22.579+0000 7ff9e1bbb780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-24T20:59:22.588 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:22.581+0000 7ff9e1bbb780 20 rados->read ofs=0 len=0 2023-05-24T20:59:22.589 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:22.581+0000 7ff9e1bbb780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-24T20:59:22.589 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:22.581+0000 7ff9e1bbb780 10 could not read realm id: (2) No such file or directory 2023-05-24T20:59:22.589 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:22.581+0000 7ff9e1bbb780 10 WARNING: failed to set zone as default, r=-22 2023-05-24T20:59:22.589 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:22.581+0000 7ff9e1bbb780 20 rados->read ofs=0 len=0 2023-05-24T20:59:22.590 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:22.582+0000 7ff9e1bbb780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-24T20:59:22.590 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:22.584+0000 7ff9e1bbb780 20 rados->read ofs=0 len=0 2023-05-24T20:59:22.592 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:22.584+0000 7ff9e1bbb780 20 rados_obj.operate() r=0 bl.length=874 2023-05-24T20:59:22.593 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:22.584+0000 7ff9e1bbb780 20 rados->read ofs=0 len=0 2023-05-24T20:59:22.593 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:22.585+0000 7ff9e1bbb780 20 rados_obj.operate() r=0 bl.length=398 2023-05-24T20:59:22.593 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:22.585+0000 7ff9e1bbb780 10 Using default name default 2023-05-24T20:59:22.593 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:22.585+0000 7ff9e1bbb780 20 rados->read ofs=0 len=0 2023-05-24T20:59:22.593 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:22.585+0000 7ff9e1bbb780 20 rados_obj.operate() r=0 bl.length=46 2023-05-24T20:59:22.593 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:22.585+0000 7ff9e1bbb780 20 rados->read ofs=0 len=0 2023-05-24T20:59:22.593 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:22.585+0000 7ff9e1bbb780 20 rados_obj.operate() r=0 bl.length=874 2023-05-24T20:59:22.594 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:22.585+0000 7ff9e1bbb780 20 zone default found 2023-05-24T20:59:22.594 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:22.585+0000 7ff9e1bbb780 4 Realm: () 2023-05-24T20:59:22.594 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:22.585+0000 7ff9e1bbb780 4 ZoneGroup: default (0d4bdc40-d4ec-4dfa-a11d-81b98a2ce9fd) 2023-05-24T20:59:22.594 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:22.585+0000 7ff9e1bbb780 4 Zone: default (b8ed9848-66be-469e-9423-6255523de54d) 2023-05-24T20:59:22.594 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:22.585+0000 7ff9e1bbb780 10 cannot find current period zonegroup using local zonegroup configuration 2023-05-24T20:59:22.594 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:22.585+0000 7ff9e1bbb780 20 zonegroup default 2023-05-24T20:59:22.594 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:22.585+0000 7ff9e1bbb780 20 rados->read ofs=0 len=0 2023-05-24T20:59:22.595 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:22.586+0000 7ff9e1bbb780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-24T20:59:22.595 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:22.586+0000 7ff9e1bbb780 20 rados->read ofs=0 len=0 2023-05-24T20:59:24.576 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:24.569+0000 7ff9e1bbb780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-24T20:59:24.576 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:24.569+0000 7ff9e1bbb780 20 rados->read ofs=0 len=0 2023-05-24T20:59:24.581 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:24.575+0000 7ff9e1bbb780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-24T20:59:24.582 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:24.575+0000 7ff9e1bbb780 20 started sync module instance, tier type = 2023-05-24T20:59:24.582 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:24.575+0000 7ff9e1bbb780 20 started zone id=b8ed9848-66be-469e-9423-6255523de54d (name=default) with tier type = 2023-05-24T20:59:26.597 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:26.590+0000 7ff9e1bbb780 20 add_watcher() i=0 2023-05-24T20:59:26.598 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:26.590+0000 7ff9e1bbb780 20 add_watcher() i=1 2023-05-24T20:59:26.598 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:26.590+0000 7ff9e1bbb780 20 add_watcher() i=2 2023-05-24T20:59:26.598 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:26.590+0000 7ff9e1bbb780 20 add_watcher() i=3 2023-05-24T20:59:26.598 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:26.590+0000 7ff9e1bbb780 20 add_watcher() i=4 2023-05-24T20:59:26.598 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:26.590+0000 7ff9e1bbb780 20 add_watcher() i=5 2023-05-24T20:59:26.598 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:26.590+0000 7ff9e1bbb780 20 add_watcher() i=6 2023-05-24T20:59:26.599 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:26.591+0000 7ff9e1bbb780 20 add_watcher() i=7 2023-05-24T20:59:26.599 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:26.591+0000 7ff9e1bbb780 2 all 8 watchers are set, enabling cache 2023-05-24T20:59:26.600 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:26.592+0000 7ff9e1bbb780 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2023-05-24T20:59:26.600 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:26.592+0000 7ff9e1bbb780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2023-05-24T20:59:26.600 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:26.592+0000 7ff9e1bbb780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-05-24T20:59:26.600 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:26.592+0000 7ff9e1bbb780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2023-05-24T20:59:26.600 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:26.592+0000 7ff9e1bbb780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2023-05-24T20:59:26.600 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:26.592+0000 7ff9e1bbb780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-05-24T20:59:26.600 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:26.592+0000 7ff9e1bbb780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2023-05-24T20:59:26.601 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:26.592+0000 7ff9e1bbb780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2023-05-24T20:59:26.601 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:26.592+0000 7ff9e1bbb780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-05-24T20:59:26.601 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:26.592+0000 7ff9e1bbb780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2023-05-24T20:59:26.601 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:26.592+0000 7ff9e1bbb780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2023-05-24T20:59:26.601 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:26.592+0000 7ff9e1bbb780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-05-24T20:59:26.601 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:26.593+0000 7ff9e1bbb780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2023-05-24T20:59:26.601 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:26.593+0000 7ff9e1bbb780 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1210 entering 2023-05-24T20:59:26.602 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:26.594+0000 7ff9e1bbb780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-05-24T20:59:26.603 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:26.596+0000 7ff9c6b37700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-05-24T20:59:26.603 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:26.596+0000 7ff9e1bbb780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-05-24T20:59:26.603 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:26.596+0000 7ff9e1bbb780 20 check_secure_mon_conn(): mode 1 is insecure 2023-05-24T20:59:29.578 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.570+0000 7ff9e1bbb780 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2023-05-24T20:59:29.579 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.570+0000 7ff9e1bbb780 5 note: GC not initialized 2023-05-24T20:59:29.580 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.571+0000 7ff9c232e700 20 reqs_thread_entry: start 2023-05-24T20:59:29.581 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.571+0000 7ff9e1bbb780 20 init_complete bucket index max shards: 11 2023-05-24T20:59:29.581 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.571+0000 7ff9e1bbb780 10 rgw notify: Started notification manager with: 1 workers 2023-05-24T20:59:29.581 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.572+0000 7ff9bfb29700 20 reqs_thread_entry: start 2023-05-24T20:59:29.581 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.572+0000 7ff9c032a700 20 rgw notify: INFO: next queues processing will happen at: Wed May 24 20:59:59 2023 2023-05-24T20:59:29.581 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-05-24T20:59:29.600 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.594+0000 7ff9c6b37700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-05-24T20:59:29.603 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.596+0000 7ff9e1bbb780 20 remove_watcher() i=0 2023-05-24T20:59:29.603 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.596+0000 7ff9e1bbb780 2 removed watcher, disabling cache 2023-05-24T20:59:29.603 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.597+0000 7ff9e1bbb780 20 remove_watcher() i=1 2023-05-24T20:59:29.604 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.598+0000 7ff9e1bbb780 20 remove_watcher() i=2 2023-05-24T20:59:29.606 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.599+0000 7ff9e1bbb780 20 remove_watcher() i=3 2023-05-24T20:59:29.607 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.600+0000 7ff9e1bbb780 20 remove_watcher() i=4 2023-05-24T20:59:29.608 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.601+0000 7ff9e1bbb780 20 remove_watcher() i=5 2023-05-24T20:59:29.609 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.602+0000 7ff9e1bbb780 20 remove_watcher() i=6 2023-05-24T20:59:29.610 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.603+0000 7ff9e1bbb780 20 remove_watcher() i=7 2023-05-24T20:59:29.612 INFO:teuthology.orchestra.run.smithi157.stdout:[] 2023-05-24T20:59:29.614 DEBUG:tasks.util.rgw: json result: [] 2023-05-24T20:59:29.615 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-05-24T20:59:29.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-05-24T20:59:29.616 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-05-24T20:59:29.670 INFO:teuthology.orchestra.run.smithi157.stderr:ignoring --setuser ceph since I am not root 2023-05-24T20:59:29.671 INFO:teuthology.orchestra.run.smithi157.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T20:59:29.679 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.673+0000 7fcb1ae96780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-05-24T20:59:29.680 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.673+0000 7fcb1ae96780 20 check_secure_mon_conn(): mode 1 is insecure 2023-05-24T20:59:29.683 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.676+0000 7fcafc60b700 20 reqs_thread_entry: start 2023-05-24T20:59:29.689 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2023-05-24T20:59:29.690 INFO:teuthology.orchestra.run.smithi157.stdout: "id": "b8ed9848-66be-469e-9423-6255523de54d", 2023-05-24T20:59:29.690 INFO:teuthology.orchestra.run.smithi157.stdout: "name": "default", 2023-05-24T20:59:29.690 INFO:teuthology.orchestra.run.smithi157.stdout: "domain_root": "default.rgw.meta:root", 2023-05-24T20:59:29.690 INFO:teuthology.orchestra.run.smithi157.stdout: "control_pool": "default.rgw.control", 2023-05-24T20:59:29.690 INFO:teuthology.orchestra.run.smithi157.stdout: "gc_pool": "default.rgw.log:gc", 2023-05-24T20:59:29.690 INFO:teuthology.orchestra.run.smithi157.stdout: "lc_pool": "default.rgw.log:lc", 2023-05-24T20:59:29.690 INFO:teuthology.orchestra.run.smithi157.stdout: "log_pool": "default.rgw.log", 2023-05-24T20:59:29.691 INFO:teuthology.orchestra.run.smithi157.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-05-24T20:59:29.691 INFO:teuthology.orchestra.run.smithi157.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-05-24T20:59:29.691 INFO:teuthology.orchestra.run.smithi157.stdout: "roles_pool": "default.rgw.meta:roles", 2023-05-24T20:59:29.691 INFO:teuthology.orchestra.run.smithi157.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-05-24T20:59:29.691 INFO:teuthology.orchestra.run.smithi157.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-05-24T20:59:29.691 INFO:teuthology.orchestra.run.smithi157.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-05-24T20:59:29.691 INFO:teuthology.orchestra.run.smithi157.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-05-24T20:59:29.692 INFO:teuthology.orchestra.run.smithi157.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-05-24T20:59:29.692 INFO:teuthology.orchestra.run.smithi157.stdout: "otp_pool": "default.rgw.otp", 2023-05-24T20:59:29.692 INFO:teuthology.orchestra.run.smithi157.stdout: "system_key": { 2023-05-24T20:59:29.692 INFO:teuthology.orchestra.run.smithi157.stdout: "access_key": "", 2023-05-24T20:59:29.692 INFO:teuthology.orchestra.run.smithi157.stdout: "secret_key": "" 2023-05-24T20:59:29.692 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T20:59:29.692 INFO:teuthology.orchestra.run.smithi157.stdout: "placement_pools": [ 2023-05-24T20:59:29.692 INFO:teuthology.orchestra.run.smithi157.stdout: { 2023-05-24T20:59:29.693 INFO:teuthology.orchestra.run.smithi157.stdout: "key": "default-placement", 2023-05-24T20:59:29.693 INFO:teuthology.orchestra.run.smithi157.stdout: "val": { 2023-05-24T20:59:29.693 INFO:teuthology.orchestra.run.smithi157.stdout: "index_pool": "default.rgw.buckets.index", 2023-05-24T20:59:29.693 INFO:teuthology.orchestra.run.smithi157.stdout: "storage_classes": { 2023-05-24T20:59:29.693 INFO:teuthology.orchestra.run.smithi157.stdout: "STANDARD": { 2023-05-24T20:59:29.693 INFO:teuthology.orchestra.run.smithi157.stdout: "data_pool": "default.rgw.buckets.data", 2023-05-24T20:59:29.693 INFO:teuthology.orchestra.run.smithi157.stdout: "compression_type": "random" 2023-05-24T20:59:29.694 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T20:59:29.694 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T20:59:29.694 INFO:teuthology.orchestra.run.smithi157.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-05-24T20:59:29.694 INFO:teuthology.orchestra.run.smithi157.stdout: "index_type": 0, 2023-05-24T20:59:29.694 INFO:teuthology.orchestra.run.smithi157.stdout: "inline_data": true 2023-05-24T20:59:29.694 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T20:59:29.694 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T20:59:29.695 INFO:teuthology.orchestra.run.smithi157.stdout: ], 2023-05-24T20:59:29.695 INFO:teuthology.orchestra.run.smithi157.stdout: "realm_id": "", 2023-05-24T20:59:29.695 INFO:teuthology.orchestra.run.smithi157.stdout: "notif_pool": "default.rgw.log:notif" 2023-05-24T20:59:29.695 INFO:teuthology.orchestra.run.smithi157.stdout:} 2023-05-24T20:59:29.696 DEBUG:tasks.util.rgw: json result: {'id': 'b8ed9848-66be-469e-9423-6255523de54d', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-05-24T20:59:29.697 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2023-05-24T20:59:29.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2023-05-24T20:59:29.697 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2023-05-24T20:59:29.741 INFO:teuthology.orchestra.run.smithi157.stderr:ignoring --setuser ceph since I am not root 2023-05-24T20:59:29.742 INFO:teuthology.orchestra.run.smithi157.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T20:59:29.750 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.743+0000 7f7a77c10780 20 rados->read ofs=0 len=0 2023-05-24T20:59:29.751 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.744+0000 7f7a77c10780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-24T20:59:29.751 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.744+0000 7f7a77c10780 20 realm 2023-05-24T20:59:29.751 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.744+0000 7f7a77c10780 20 rados->read ofs=0 len=0 2023-05-24T20:59:29.752 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.744+0000 7f7a77c10780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-24T20:59:29.752 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.744+0000 7f7a77c10780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-05-24T20:59:29.752 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.744+0000 7f7a77c10780 20 rados->read ofs=0 len=0 2023-05-24T20:59:29.752 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.744+0000 7f7a77c10780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-24T20:59:29.752 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.744+0000 7f7a77c10780 20 rados->read ofs=0 len=0 2023-05-24T20:59:29.753 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.745+0000 7f7a77c10780 20 rados_obj.operate() r=0 bl.length=46 2023-05-24T20:59:29.753 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.745+0000 7f7a77c10780 20 rados->read ofs=0 len=0 2023-05-24T20:59:29.753 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.745+0000 7f7a77c10780 20 rados_obj.operate() r=0 bl.length=890 2023-05-24T20:59:29.753 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.745+0000 7f7a77c10780 20 searching for the correct realm 2023-05-24T20:59:29.761 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.752+0000 7f7a77c10780 20 RGWRados::pool_iterate: got zone_info.b8ed9848-66be-469e-9423-6255523de54d 2023-05-24T20:59:29.761 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.752+0000 7f7a77c10780 20 RGWRados::pool_iterate: got zonegroup_info.0d4bdc40-d4ec-4dfa-a11d-81b98a2ce9fd 2023-05-24T20:59:29.761 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.752+0000 7f7a77c10780 20 RGWRados::pool_iterate: got zone_names.default 2023-05-24T20:59:29.762 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.752+0000 7f7a77c10780 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-05-24T20:59:29.762 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.752+0000 7f7a77c10780 20 rados->read ofs=0 len=0 2023-05-24T20:59:29.762 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.752+0000 7f7a77c10780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-24T20:59:29.762 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.752+0000 7f7a77c10780 20 rados->read ofs=0 len=0 2023-05-24T20:59:29.763 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.753+0000 7f7a77c10780 20 rados_obj.operate() r=0 bl.length=46 2023-05-24T20:59:29.763 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.753+0000 7f7a77c10780 20 rados->read ofs=0 len=0 2023-05-24T20:59:29.763 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.753+0000 7f7a77c10780 20 rados_obj.operate() r=0 bl.length=398 2023-05-24T20:59:29.764 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.753+0000 7f7a77c10780 20 zone default found 2023-05-24T20:59:29.764 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.753+0000 7f7a77c10780 4 Realm: () 2023-05-24T20:59:29.764 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.753+0000 7f7a77c10780 4 ZoneGroup: default (0d4bdc40-d4ec-4dfa-a11d-81b98a2ce9fd) 2023-05-24T20:59:29.764 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.753+0000 7f7a77c10780 4 Zone: default (b8ed9848-66be-469e-9423-6255523de54d) 2023-05-24T20:59:29.765 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.753+0000 7f7a77c10780 10 cannot find current period zonegroup using local zonegroup configuration 2023-05-24T20:59:29.765 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.753+0000 7f7a77c10780 20 zonegroup default 2023-05-24T20:59:29.765 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.753+0000 7f7a77c10780 20 rados->read ofs=0 len=0 2023-05-24T20:59:29.765 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.753+0000 7f7a77c10780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-24T20:59:29.765 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.753+0000 7f7a77c10780 20 rados->read ofs=0 len=0 2023-05-24T20:59:29.765 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.753+0000 7f7a77c10780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-24T20:59:29.765 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.753+0000 7f7a77c10780 20 rados->read ofs=0 len=0 2023-05-24T20:59:29.766 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.753+0000 7f7a77c10780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-24T20:59:29.766 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.753+0000 7f7a77c10780 20 started sync module instance, tier type = 2023-05-24T20:59:29.766 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.753+0000 7f7a77c10780 20 started zone id=b8ed9848-66be-469e-9423-6255523de54d (name=default) with tier type = 2023-05-24T20:59:29.768 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.761+0000 7f7a77c10780 20 add_watcher() i=0 2023-05-24T20:59:29.768 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.761+0000 7f7a77c10780 20 add_watcher() i=1 2023-05-24T20:59:29.768 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.761+0000 7f7a77c10780 20 add_watcher() i=2 2023-05-24T20:59:29.769 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.761+0000 7f7a77c10780 20 add_watcher() i=3 2023-05-24T20:59:29.769 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.761+0000 7f7a77c10780 20 add_watcher() i=4 2023-05-24T20:59:29.769 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.761+0000 7f7a77c10780 20 add_watcher() i=5 2023-05-24T20:59:29.769 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.761+0000 7f7a77c10780 20 add_watcher() i=6 2023-05-24T20:59:29.770 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.762+0000 7f7a77c10780 20 add_watcher() i=7 2023-05-24T20:59:29.770 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.762+0000 7f7a77c10780 2 all 8 watchers are set, enabling cache 2023-05-24T20:59:29.770 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.763+0000 7f7a5cb8c700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-05-24T20:59:29.770 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.763+0000 7f7a77c10780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-05-24T20:59:29.770 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.763+0000 7f7a77c10780 20 check_secure_mon_conn(): mode 1 is insecure 2023-05-24T20:59:29.773 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.766+0000 7f7a77c10780 5 note: GC not initialized 2023-05-24T20:59:29.774 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.767+0000 7f7a58383700 20 reqs_thread_entry: start 2023-05-24T20:59:29.774 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.767+0000 7f7a77c10780 20 init_complete bucket index max shards: 11 2023-05-24T20:59:29.774 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.768+0000 7f7a77c10780 10 rgw notify: Started notification manager with: 1 workers 2023-05-24T20:59:29.775 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.768+0000 7f7a5637f700 20 rgw notify: INFO: next queues processing will happen at: Wed May 24 20:59:59 2023 2023-05-24T20:59:29.775 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-05-24T20:59:29.775 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.768+0000 7f7a55b5d700 20 reqs_thread_entry: start 2023-05-24T20:59:29.781 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.775+0000 7f7a5cb8c700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-05-24T20:59:29.783 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.776+0000 7f7a77c10780 20 remove_watcher() i=0 2023-05-24T20:59:29.783 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.776+0000 7f7a77c10780 2 removed watcher, disabling cache 2023-05-24T20:59:29.784 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.777+0000 7f7a77c10780 20 remove_watcher() i=1 2023-05-24T20:59:29.784 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.778+0000 7f7a77c10780 20 remove_watcher() i=2 2023-05-24T20:59:29.785 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.779+0000 7f7a77c10780 20 remove_watcher() i=3 2023-05-24T20:59:29.786 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.779+0000 7f7a77c10780 20 remove_watcher() i=4 2023-05-24T20:59:29.787 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.780+0000 7f7a77c10780 20 remove_watcher() i=5 2023-05-24T20:59:29.787 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.781+0000 7f7a77c10780 20 remove_watcher() i=6 2023-05-24T20:59:29.788 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.782+0000 7f7a77c10780 20 remove_watcher() i=7 2023-05-24T20:59:29.790 INFO:teuthology.orchestra.run.smithi157.stdout:[] 2023-05-24T20:59:29.793 DEBUG:tasks.util.rgw: json result: [] 2023-05-24T20:59:29.793 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-05-24T20:59:29.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-05-24T20:59:29.794 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-05-24T20:59:29.844 INFO:teuthology.orchestra.run.smithi157.stderr:ignoring --setuser ceph since I am not root 2023-05-24T20:59:29.845 INFO:teuthology.orchestra.run.smithi157.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T20:59:29.853 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.846+0000 7fed9a148780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-05-24T20:59:29.854 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.846+0000 7fed9a148780 20 check_secure_mon_conn(): mode 1 is insecure 2023-05-24T20:59:29.856 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T20:59:29.850+0000 7fed7b8bd700 20 reqs_thread_entry: start 2023-05-24T20:59:29.863 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2023-05-24T20:59:29.863 INFO:teuthology.orchestra.run.smithi157.stdout: "id": "b8ed9848-66be-469e-9423-6255523de54d", 2023-05-24T20:59:29.864 INFO:teuthology.orchestra.run.smithi157.stdout: "name": "default", 2023-05-24T20:59:29.864 INFO:teuthology.orchestra.run.smithi157.stdout: "domain_root": "default.rgw.meta:root", 2023-05-24T20:59:29.864 INFO:teuthology.orchestra.run.smithi157.stdout: "control_pool": "default.rgw.control", 2023-05-24T20:59:29.865 INFO:teuthology.orchestra.run.smithi157.stdout: "gc_pool": "default.rgw.log:gc", 2023-05-24T20:59:29.865 INFO:teuthology.orchestra.run.smithi157.stdout: "lc_pool": "default.rgw.log:lc", 2023-05-24T20:59:29.865 INFO:teuthology.orchestra.run.smithi157.stdout: "log_pool": "default.rgw.log", 2023-05-24T20:59:29.865 INFO:teuthology.orchestra.run.smithi157.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-05-24T20:59:29.866 INFO:teuthology.orchestra.run.smithi157.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-05-24T20:59:29.866 INFO:teuthology.orchestra.run.smithi157.stdout: "roles_pool": "default.rgw.meta:roles", 2023-05-24T20:59:29.866 INFO:teuthology.orchestra.run.smithi157.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-05-24T20:59:29.866 INFO:teuthology.orchestra.run.smithi157.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-05-24T20:59:29.867 INFO:teuthology.orchestra.run.smithi157.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-05-24T20:59:29.867 INFO:teuthology.orchestra.run.smithi157.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-05-24T20:59:29.867 INFO:teuthology.orchestra.run.smithi157.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-05-24T20:59:29.868 INFO:teuthology.orchestra.run.smithi157.stdout: "otp_pool": "default.rgw.otp", 2023-05-24T20:59:29.868 INFO:teuthology.orchestra.run.smithi157.stdout: "system_key": { 2023-05-24T20:59:29.868 INFO:teuthology.orchestra.run.smithi157.stdout: "access_key": "", 2023-05-24T20:59:29.869 INFO:teuthology.orchestra.run.smithi157.stdout: "secret_key": "" 2023-05-24T20:59:29.870 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T20:59:29.871 INFO:teuthology.orchestra.run.smithi157.stdout: "placement_pools": [ 2023-05-24T20:59:29.871 INFO:teuthology.orchestra.run.smithi157.stdout: { 2023-05-24T20:59:29.871 INFO:teuthology.orchestra.run.smithi157.stdout: "key": "default-placement", 2023-05-24T20:59:29.872 INFO:teuthology.orchestra.run.smithi157.stdout: "val": { 2023-05-24T20:59:29.872 INFO:teuthology.orchestra.run.smithi157.stdout: "index_pool": "default.rgw.buckets.index", 2023-05-24T20:59:29.872 INFO:teuthology.orchestra.run.smithi157.stdout: "storage_classes": { 2023-05-24T20:59:29.873 INFO:teuthology.orchestra.run.smithi157.stdout: "STANDARD": { 2023-05-24T20:59:29.873 INFO:teuthology.orchestra.run.smithi157.stdout: "data_pool": "default.rgw.buckets.data", 2023-05-24T20:59:29.873 INFO:teuthology.orchestra.run.smithi157.stdout: "compression_type": "random" 2023-05-24T20:59:29.873 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T20:59:29.874 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T20:59:29.874 INFO:teuthology.orchestra.run.smithi157.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-05-24T20:59:29.874 INFO:teuthology.orchestra.run.smithi157.stdout: "index_type": 0, 2023-05-24T20:59:29.875 INFO:teuthology.orchestra.run.smithi157.stdout: "inline_data": true 2023-05-24T20:59:29.875 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T20:59:29.875 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T20:59:29.875 INFO:teuthology.orchestra.run.smithi157.stdout: ], 2023-05-24T20:59:29.876 INFO:teuthology.orchestra.run.smithi157.stdout: "realm_id": "", 2023-05-24T20:59:29.876 INFO:teuthology.orchestra.run.smithi157.stdout: "notif_pool": "default.rgw.log:notif" 2023-05-24T20:59:29.876 INFO:teuthology.orchestra.run.smithi157.stdout:} 2023-05-24T20:59:29.878 DEBUG:tasks.util.rgw: json result: {'id': 'b8ed9848-66be-469e-9423-6255523de54d', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-05-24T20:59:29.878 INFO:tasks.rgw:Starting rgw... 2023-05-24T20:59:29.879 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2023-05-24T20:59:29.879 INFO:tasks.rgw:Using beast as radosgw frontend 2023-05-24T20:59:29.879 INFO:tasks.rgw.c1.client.0:Restarting daemon 2023-05-24T20:59:29.880 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2023-05-24T20:59:29.884 INFO:tasks.rgw.c1.client.0:Started 2023-05-24T20:59:29.884 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2023-05-24T20:59:29.885 INFO:tasks.rgw:Using beast as radosgw frontend 2023-05-24T20:59:29.885 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-05-24T20:59:29.885 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-05-24T20:59:29.887 INFO:tasks.rgw.c1.client.1:Started 2023-05-24T20:59:29.888 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2023-05-24T20:59:29.889 INFO:tasks.rgw:Using beast as radosgw frontend 2023-05-24T20:59:29.890 INFO:tasks.rgw.c2.client.0:Restarting daemon 2023-05-24T20:59:29.890 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2023-05-24T20:59:29.892 INFO:tasks.rgw.c2.client.0:Started 2023-05-24T20:59:29.893 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2023-05-24T20:59:29.893 INFO:tasks.rgw:Using beast as radosgw frontend 2023-05-24T20:59:29.894 INFO:tasks.rgw.c2.client.1:Restarting daemon 2023-05-24T20:59:29.894 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2023-05-24T20:59:29.896 INFO:tasks.rgw.c2.client.1:Started 2023-05-24T20:59:29.897 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi071.front.sepia.ceph.com:8000/ 2023-05-24T20:59:29.897 DEBUG:teuthology.orchestra.run.smithi071:> curl http://smithi071.front.sepia.ceph.com:8000/ 2023-05-24T20:59:29.933 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-05-24T20:59:29.934 INFO:teuthology.orchestra.run.smithi071.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-05-24T20:59:29.935 INFO:teuthology.orchestra.run.smithi071.stderr: Dload Upload Total Spent Left Speed 2023-05-24T20:59:29.935 INFO:teuthology.orchestra.run.smithi071.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi071.front.sepia.ceph.com port 8000: Connection refused 2023-05-24T20:59:30.936 DEBUG:teuthology.orchestra.run.smithi071:> curl http://smithi071.front.sepia.ceph.com:8000/ 2023-05-24T20:59:30.958 INFO:teuthology.orchestra.run.smithi071.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-05-24T20:59:30.959 INFO:teuthology.orchestra.run.smithi071.stderr: Dload Upload Total Spent Left Speed 2023-05-24T20:59:30.959 INFO:teuthology.orchestra.run.smithi071.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-05-24T20:59:30.960 INFO:teuthology.orchestra.run.smithi071.stdout:anonymous 2023-05-24T20:59:30.960 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi071.front.sepia.ceph.com:8001/ 2023-05-24T20:59:30.961 DEBUG:teuthology.orchestra.run.smithi071:> curl http://smithi071.front.sepia.ceph.com:8001/ 2023-05-24T20:59:31.022 INFO:teuthology.orchestra.run.smithi071.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-05-24T20:59:31.022 INFO:teuthology.orchestra.run.smithi071.stderr: Dload Upload Total Spent Left Speed 2023-05-24T20:59:31.023 INFO:teuthology.orchestra.run.smithi071.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-05-24T20:59:31.024 INFO:teuthology.orchestra.run.smithi071.stdout:anonymous 2023-05-24T20:59:31.024 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi157.front.sepia.ceph.com:8000/ 2023-05-24T20:59:31.025 DEBUG:teuthology.orchestra.run.smithi157:> curl http://smithi157.front.sepia.ceph.com:8000/ 2023-05-24T20:59:31.061 INFO:teuthology.orchestra.run.smithi157.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-05-24T20:59:31.061 INFO:teuthology.orchestra.run.smithi157.stderr: Dload Upload Total Spent Left Speed 2023-05-24T20:59:31.063 INFO:teuthology.orchestra.run.smithi157.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-05-24T20:59:31.063 INFO:teuthology.orchestra.run.smithi157.stdout:anonymous 2023-05-24T20:59:31.064 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi157.front.sepia.ceph.com:8001/ 2023-05-24T20:59:31.064 DEBUG:teuthology.orchestra.run.smithi157:> curl http://smithi157.front.sepia.ceph.com:8001/ 2023-05-24T20:59:31.126 INFO:teuthology.orchestra.run.smithi157.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-05-24T20:59:31.127 INFO:teuthology.orchestra.run.smithi157.stderr: Dload Upload Total Spent Left Speed 2023-05-24T20:59:31.127 INFO:teuthology.orchestra.run.smithi157.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-05-24T20:59:31.128 INFO:teuthology.orchestra.run.smithi157.stdout:anonymous 2023-05-24T20:59:31.128 INFO:teuthology.run_tasks:Running task rgw-multisite... 2023-05-24T20:59:31.693 INFO:tasks.rgw_multisite:creating realm.. 2023-05-24T20:59:31.694 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T20:59:31.694 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T20:59:31.695 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-24T20:59:31.747 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T20:59:31.747 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T20:59:31.778 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T20:59:31.779 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T20:59:31.779 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "test-realm", 2023-05-24T20:59:31.779 INFO:teuthology.orchestra.run.smithi071.stdout: "current_period": "1cea3a6e-eb52-480e-95b7-3f6e43317e60", 2023-05-24T20:59:31.779 INFO:teuthology.orchestra.run.smithi071.stdout: "epoch": 1 2023-05-24T20:59:31.779 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T20:59:31.781 INFO:tasks.rgw_multisite:creating master zonegroup.. 2023-05-24T20:59:31.781 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T20:59:31.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T20:59:31.782 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup a --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-24T20:59:31.831 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T20:59:31.831 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T20:59:31.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:31.838+0000 7f8bff919780 -1 old format 2023-05-24T20:59:31.855 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T20:59:31.855 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "a", 2023-05-24T20:59:31.855 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a", 2023-05-24T20:59:31.855 INFO:teuthology.orchestra.run.smithi071.stdout: "api_name": "", 2023-05-24T20:59:31.855 INFO:teuthology.orchestra.run.smithi071.stdout: "is_master": true, 2023-05-24T20:59:31.856 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T20:59:31.856 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-05-24T20:59:31.856 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T20:59:31.856 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames": [], 2023-05-24T20:59:31.856 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames_s3website": [], 2023-05-24T20:59:31.856 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zone": "", 2023-05-24T20:59:31.856 INFO:teuthology.orchestra.run.smithi071.stdout: "zones": [], 2023-05-24T20:59:31.857 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_targets": [ 2023-05-24T20:59:31.857 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T20:59:31.857 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "default-placement", 2023-05-24T20:59:31.857 INFO:teuthology.orchestra.run.smithi071.stdout: "tags": [], 2023-05-24T20:59:31.857 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": [] 2023-05-24T20:59:31.857 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T20:59:31.857 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T20:59:31.858 INFO:teuthology.orchestra.run.smithi071.stdout: "default_placement": "default-placement", 2023-05-24T20:59:31.858 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T20:59:31.858 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_policy": { 2023-05-24T20:59:31.858 INFO:teuthology.orchestra.run.smithi071.stdout: "groups": [] 2023-05-24T20:59:31.858 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T20:59:31.858 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled_features": [ 2023-05-24T20:59:31.858 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T20:59:31.859 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T20:59:31.859 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T20:59:31.860 INFO:tasks.rgw_multisite:creating master zone.. 2023-05-24T20:59:31.860 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'Z6KKVUYPP0489K03', '--secret', 'EX5IDzHTe9AVqa8b1PSD8B5Y1CKI3jBg', '--endpoints', 'http://smithi071.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T20:59:31.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'Z6KKVUYPP0489K03', '--secret', 'EX5IDzHTe9AVqa8b1PSD8B5Y1CKI3jBg', '--endpoints', 'http://smithi071.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T20:59:31.861 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --default --master --access-key Z6KKVUYPP0489K03 --secret EX5IDzHTe9AVqa8b1PSD8B5Y1CKI3jBg --endpoints http://smithi071.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-24T20:59:31.955 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T20:59:31.956 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T20:59:31.969 INFO:teuthology.orchestra.run.smithi071.stderr:NOTICE: set zone's realm_id=4a7754f9-b4ac-4d00-9827-3910f2de91d2 2023-05-24T20:59:31.989 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T20:59:31.989 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T20:59:31.990 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a1", 2023-05-24T20:59:31.990 INFO:teuthology.orchestra.run.smithi071.stdout: "domain_root": "a1.rgw.meta:root", 2023-05-24T20:59:31.990 INFO:teuthology.orchestra.run.smithi071.stdout: "control_pool": "a1.rgw.control", 2023-05-24T20:59:31.990 INFO:teuthology.orchestra.run.smithi071.stdout: "gc_pool": "a1.rgw.log:gc", 2023-05-24T20:59:31.990 INFO:teuthology.orchestra.run.smithi071.stdout: "lc_pool": "a1.rgw.log:lc", 2023-05-24T20:59:31.990 INFO:teuthology.orchestra.run.smithi071.stdout: "log_pool": "a1.rgw.log", 2023-05-24T20:59:31.990 INFO:teuthology.orchestra.run.smithi071.stdout: "intent_log_pool": "a1.rgw.log:intent", 2023-05-24T20:59:31.990 INFO:teuthology.orchestra.run.smithi071.stdout: "usage_log_pool": "a1.rgw.log:usage", 2023-05-24T20:59:31.991 INFO:teuthology.orchestra.run.smithi071.stdout: "roles_pool": "a1.rgw.meta:roles", 2023-05-24T20:59:31.991 INFO:teuthology.orchestra.run.smithi071.stdout: "reshard_pool": "a1.rgw.log:reshard", 2023-05-24T20:59:31.991 INFO:teuthology.orchestra.run.smithi071.stdout: "user_keys_pool": "a1.rgw.meta:users.keys", 2023-05-24T20:59:31.992 INFO:teuthology.orchestra.run.smithi071.stdout: "user_email_pool": "a1.rgw.meta:users.email", 2023-05-24T20:59:31.992 INFO:teuthology.orchestra.run.smithi071.stdout: "user_swift_pool": "a1.rgw.meta:users.swift", 2023-05-24T20:59:31.992 INFO:teuthology.orchestra.run.smithi071.stdout: "user_uid_pool": "a1.rgw.meta:users.uid", 2023-05-24T20:59:31.992 INFO:teuthology.orchestra.run.smithi071.stdout: "otp_pool": "a1.rgw.otp", 2023-05-24T20:59:31.992 INFO:teuthology.orchestra.run.smithi071.stdout: "system_key": { 2023-05-24T20:59:31.992 INFO:teuthology.orchestra.run.smithi071.stdout: "access_key": "Z6KKVUYPP0489K03", 2023-05-24T20:59:31.993 INFO:teuthology.orchestra.run.smithi071.stdout: "secret_key": "EX5IDzHTe9AVqa8b1PSD8B5Y1CKI3jBg" 2023-05-24T20:59:31.993 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T20:59:31.993 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_pools": [ 2023-05-24T20:59:31.993 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T20:59:31.993 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "default-placement", 2023-05-24T20:59:31.993 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T20:59:31.994 INFO:teuthology.orchestra.run.smithi071.stdout: "index_pool": "a1.rgw.buckets.index", 2023-05-24T20:59:31.994 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": { 2023-05-24T20:59:31.994 INFO:teuthology.orchestra.run.smithi071.stdout: "STANDARD": { 2023-05-24T20:59:31.995 INFO:teuthology.orchestra.run.smithi071.stdout: "data_pool": "a1.rgw.buckets.data" 2023-05-24T20:59:31.995 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T20:59:31.995 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T20:59:31.995 INFO:teuthology.orchestra.run.smithi071.stdout: "data_extra_pool": "a1.rgw.buckets.non-ec", 2023-05-24T20:59:31.995 INFO:teuthology.orchestra.run.smithi071.stdout: "index_type": 0, 2023-05-24T20:59:31.995 INFO:teuthology.orchestra.run.smithi071.stdout: "inline_data": true 2023-05-24T20:59:31.996 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T20:59:31.996 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T20:59:31.996 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T20:59:31.996 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T20:59:31.996 INFO:teuthology.orchestra.run.smithi071.stdout: "notif_pool": "a1.rgw.log:notif" 2023-05-24T20:59:31.997 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T20:59:31.998 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph osd pool create a1.rgw.buckets.data 64 64 --cluster c1 2023-05-24T20:59:32.680 INFO:teuthology.orchestra.run.smithi071.stderr:pool 'a1.rgw.buckets.data' created 2023-05-24T20:59:32.697 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph osd pool application enable a1.rgw.buckets.data rgw --cluster c1 2023-05-24T20:59:33.704 INFO:teuthology.orchestra.run.smithi071.stderr:enabled application 'rgw' on pool 'a1.rgw.buckets.data' 2023-05-24T20:59:33.721 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T20:59:33.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T20:59:33.722 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-24T20:59:33.778 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T20:59:33.778 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T20:59:33.800 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T20:59:33.802 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T20:59:33.802 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a1", 2023-05-24T20:59:33.803 INFO:teuthology.orchestra.run.smithi071.stdout: "domain_root": "a1.rgw.meta:root", 2023-05-24T20:59:33.803 INFO:teuthology.orchestra.run.smithi071.stdout: "control_pool": "a1.rgw.control", 2023-05-24T20:59:33.803 INFO:teuthology.orchestra.run.smithi071.stdout: "gc_pool": "a1.rgw.log:gc", 2023-05-24T20:59:33.804 INFO:teuthology.orchestra.run.smithi071.stdout: "lc_pool": "a1.rgw.log:lc", 2023-05-24T20:59:33.804 INFO:teuthology.orchestra.run.smithi071.stdout: "log_pool": "a1.rgw.log", 2023-05-24T20:59:33.804 INFO:teuthology.orchestra.run.smithi071.stdout: "intent_log_pool": "a1.rgw.log:intent", 2023-05-24T20:59:33.804 INFO:teuthology.orchestra.run.smithi071.stdout: "usage_log_pool": "a1.rgw.log:usage", 2023-05-24T20:59:33.805 INFO:teuthology.orchestra.run.smithi071.stdout: "roles_pool": "a1.rgw.meta:roles", 2023-05-24T20:59:33.805 INFO:teuthology.orchestra.run.smithi071.stdout: "reshard_pool": "a1.rgw.log:reshard", 2023-05-24T20:59:33.805 INFO:teuthology.orchestra.run.smithi071.stdout: "user_keys_pool": "a1.rgw.meta:users.keys", 2023-05-24T20:59:33.806 INFO:teuthology.orchestra.run.smithi071.stdout: "user_email_pool": "a1.rgw.meta:users.email", 2023-05-24T20:59:33.806 INFO:teuthology.orchestra.run.smithi071.stdout: "user_swift_pool": "a1.rgw.meta:users.swift", 2023-05-24T20:59:33.806 INFO:teuthology.orchestra.run.smithi071.stdout: "user_uid_pool": "a1.rgw.meta:users.uid", 2023-05-24T20:59:33.806 INFO:teuthology.orchestra.run.smithi071.stdout: "otp_pool": "a1.rgw.otp", 2023-05-24T20:59:33.806 INFO:teuthology.orchestra.run.smithi071.stdout: "system_key": { 2023-05-24T20:59:33.807 INFO:teuthology.orchestra.run.smithi071.stdout: "access_key": "Z6KKVUYPP0489K03", 2023-05-24T20:59:33.807 INFO:teuthology.orchestra.run.smithi071.stdout: "secret_key": "EX5IDzHTe9AVqa8b1PSD8B5Y1CKI3jBg" 2023-05-24T20:59:33.807 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T20:59:33.807 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_pools": [ 2023-05-24T20:59:33.807 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T20:59:33.807 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "default-placement", 2023-05-24T20:59:33.808 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T20:59:33.808 INFO:teuthology.orchestra.run.smithi071.stdout: "index_pool": "a1.rgw.buckets.index", 2023-05-24T20:59:33.808 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": { 2023-05-24T20:59:33.809 INFO:teuthology.orchestra.run.smithi071.stdout: "STANDARD": { 2023-05-24T20:59:33.809 INFO:teuthology.orchestra.run.smithi071.stdout: "data_pool": "a1.rgw.buckets.data", 2023-05-24T20:59:33.810 INFO:teuthology.orchestra.run.smithi071.stdout: "compression_type": "random" 2023-05-24T20:59:33.810 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T20:59:33.810 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T20:59:33.810 INFO:teuthology.orchestra.run.smithi071.stdout: "data_extra_pool": "a1.rgw.buckets.non-ec", 2023-05-24T20:59:33.810 INFO:teuthology.orchestra.run.smithi071.stdout: "index_type": 0, 2023-05-24T20:59:33.811 INFO:teuthology.orchestra.run.smithi071.stdout: "inline_data": true 2023-05-24T20:59:33.811 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T20:59:33.811 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T20:59:33.811 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T20:59:33.811 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T20:59:33.811 INFO:teuthology.orchestra.run.smithi071.stdout: "notif_pool": "a1.rgw.log:notif" 2023-05-24T20:59:33.811 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T20:59:33.812 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T20:59:33.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T20:59:33.813 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-24T20:59:33.868 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T20:59:33.868 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T20:59:33.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:33.872+0000 7f1a74e9e780 0 period (1cea3a6e-eb52-480e-95b7-3f6e43317e60 does not have zone 299a3349-fae0-459e-8632-431fc36beb1b configured 2023-05-24T20:59:40.762 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T20:59:40.763 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T20:59:40.763 INFO:teuthology.orchestra.run.smithi071.stdout: "epoch": 1, 2023-05-24T20:59:40.763 INFO:teuthology.orchestra.run.smithi071.stdout: "predecessor_uuid": "1cea3a6e-eb52-480e-95b7-3f6e43317e60", 2023-05-24T20:59:40.763 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": [], 2023-05-24T20:59:40.763 INFO:teuthology.orchestra.run.smithi071.stdout: "period_map": { 2023-05-24T20:59:40.763 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T20:59:40.764 INFO:teuthology.orchestra.run.smithi071.stdout: "zonegroups": [ 2023-05-24T20:59:40.764 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T20:59:40.764 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "a", 2023-05-24T20:59:40.764 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a", 2023-05-24T20:59:40.764 INFO:teuthology.orchestra.run.smithi071.stdout: "api_name": "", 2023-05-24T20:59:40.765 INFO:teuthology.orchestra.run.smithi071.stdout: "is_master": true, 2023-05-24T20:59:40.765 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T20:59:40.765 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-05-24T20:59:40.765 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T20:59:40.766 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames": [], 2023-05-24T20:59:40.766 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames_s3website": [], 2023-05-24T20:59:40.766 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zone": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T20:59:40.766 INFO:teuthology.orchestra.run.smithi071.stdout: "zones": [ 2023-05-24T20:59:40.767 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T20:59:40.767 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T20:59:40.767 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a1", 2023-05-24T20:59:40.768 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T20:59:40.769 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-05-24T20:59:40.769 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T20:59:40.769 INFO:teuthology.orchestra.run.smithi071.stdout: "log_meta": false, 2023-05-24T20:59:40.769 INFO:teuthology.orchestra.run.smithi071.stdout: "log_data": false, 2023-05-24T20:59:40.769 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_index_max_shards": 11, 2023-05-24T20:59:40.769 INFO:teuthology.orchestra.run.smithi071.stdout: "read_only": false, 2023-05-24T20:59:40.770 INFO:teuthology.orchestra.run.smithi071.stdout: "tier_type": "", 2023-05-24T20:59:40.770 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from_all": true, 2023-05-24T20:59:40.770 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from": [], 2023-05-24T20:59:40.770 INFO:teuthology.orchestra.run.smithi071.stdout: "redirect_zone": "", 2023-05-24T20:59:40.770 INFO:teuthology.orchestra.run.smithi071.stdout: "supported_features": [ 2023-05-24T20:59:40.770 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T20:59:40.770 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T20:59:40.771 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T20:59:40.771 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T20:59:40.771 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_targets": [ 2023-05-24T20:59:40.771 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T20:59:40.771 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "default-placement", 2023-05-24T20:59:40.771 INFO:teuthology.orchestra.run.smithi071.stdout: "tags": [], 2023-05-24T20:59:40.771 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": [ 2023-05-24T20:59:40.772 INFO:teuthology.orchestra.run.smithi071.stdout: "STANDARD" 2023-05-24T20:59:40.772 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T20:59:40.772 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T20:59:40.772 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T20:59:40.772 INFO:teuthology.orchestra.run.smithi071.stdout: "default_placement": "default-placement", 2023-05-24T20:59:40.772 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T20:59:40.772 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_policy": { 2023-05-24T20:59:40.773 INFO:teuthology.orchestra.run.smithi071.stdout: "groups": [] 2023-05-24T20:59:40.773 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T20:59:40.773 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled_features": [ 2023-05-24T20:59:40.773 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T20:59:40.773 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T20:59:40.774 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T20:59:40.774 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T20:59:40.774 INFO:teuthology.orchestra.run.smithi071.stdout: "short_zone_ids": [ 2023-05-24T20:59:40.774 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T20:59:40.774 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T20:59:40.774 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 3764661814 2023-05-24T20:59:40.775 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T20:59:40.775 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T20:59:40.775 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T20:59:40.775 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zonegroup": "a", 2023-05-24T20:59:40.775 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zone": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T20:59:40.775 INFO:teuthology.orchestra.run.smithi071.stdout: "period_config": { 2023-05-24T20:59:40.775 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_quota": { 2023-05-24T20:59:40.776 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false, 2023-05-24T20:59:40.776 INFO:teuthology.orchestra.run.smithi071.stdout: "check_on_raw": false, 2023-05-24T20:59:40.776 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size": -1, 2023-05-24T20:59:40.776 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size_kb": 0, 2023-05-24T20:59:40.776 INFO:teuthology.orchestra.run.smithi071.stdout: "max_objects": -1 2023-05-24T20:59:40.776 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T20:59:40.776 INFO:teuthology.orchestra.run.smithi071.stdout: "user_quota": { 2023-05-24T20:59:40.777 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false, 2023-05-24T20:59:40.777 INFO:teuthology.orchestra.run.smithi071.stdout: "check_on_raw": false, 2023-05-24T20:59:40.777 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size": -1, 2023-05-24T20:59:40.777 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size_kb": 0, 2023-05-24T20:59:40.777 INFO:teuthology.orchestra.run.smithi071.stdout: "max_objects": -1 2023-05-24T20:59:40.777 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T20:59:40.777 INFO:teuthology.orchestra.run.smithi071.stdout: "user_ratelimit": { 2023-05-24T20:59:40.777 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_ops": 0, 2023-05-24T20:59:40.778 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_ops": 0, 2023-05-24T20:59:40.778 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_bytes": 0, 2023-05-24T20:59:40.778 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_bytes": 0, 2023-05-24T20:59:40.778 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false 2023-05-24T20:59:40.779 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T20:59:40.779 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_ratelimit": { 2023-05-24T20:59:40.779 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_ops": 0, 2023-05-24T20:59:40.779 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_ops": 0, 2023-05-24T20:59:40.779 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_bytes": 0, 2023-05-24T20:59:40.779 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_bytes": 0, 2023-05-24T20:59:40.779 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false 2023-05-24T20:59:40.779 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T20:59:40.780 INFO:teuthology.orchestra.run.smithi071.stdout: "anonymous_ratelimit": { 2023-05-24T20:59:40.780 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_ops": 0, 2023-05-24T20:59:40.780 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_ops": 0, 2023-05-24T20:59:40.780 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_bytes": 0, 2023-05-24T20:59:40.780 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_bytes": 0, 2023-05-24T20:59:40.780 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false 2023-05-24T20:59:40.780 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T20:59:40.781 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T20:59:40.781 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T20:59:40.781 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_name": "test-realm", 2023-05-24T20:59:40.781 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T20:59:40.781 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T20:59:40.783 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2023-05-24T20:59:40.783 INFO:teuthology.orchestra.run:waiting for 300 2023-05-24T20:59:40.824 INFO:tasks.rgw.c1.client.0.smithi071.stdout:2023-05-24T20:59:40.817+0000 7f625047e700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 110140) UID: 0 2023-05-24T20:59:40.824 INFO:tasks.rgw.c1.client.0.smithi071.stdout:2023-05-24T20:59:40.817+0000 7f62546e77c0 -1 shutting down 2023-05-24T20:59:44.171 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2023-05-24T20:59:46.885 INFO:tasks.rgw.c1.client.0:Stopped 2023-05-24T20:59:46.885 INFO:tasks.rgw.c1.client.0:Restarting daemon 2023-05-24T20:59:46.886 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2023-05-24T20:59:46.890 INFO:tasks.rgw.c1.client.0:Started 2023-05-24T20:59:46.891 DEBUG:teuthology.orchestra.run.smithi071:> curl http://smithi071.front.sepia.ceph.com:8000 2023-05-24T20:59:46.916 INFO:teuthology.orchestra.run.smithi071.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-05-24T20:59:46.917 INFO:teuthology.orchestra.run.smithi071.stderr: Dload Upload Total Spent Left Speed 2023-05-24T20:59:46.917 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-05-24T20:59:46.918 INFO:teuthology.orchestra.run.smithi071.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi071.front.sepia.ceph.com port 8000: Connection refused 2023-05-24T20:59:47.919 DEBUG:teuthology.orchestra.run.smithi071:> curl http://smithi071.front.sepia.ceph.com:8000 2023-05-24T20:59:47.941 INFO:teuthology.orchestra.run.smithi071.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-05-24T20:59:47.942 INFO:teuthology.orchestra.run.smithi071.stderr: Dload Upload Total Spent Left Speed 2023-05-24T20:59:47.943 INFO:teuthology.orchestra.run.smithi071.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-05-24T20:59:47.944 INFO:teuthology.orchestra.run.smithi071.stdout:anonymous 2023-05-24T20:59:47.944 INFO:tasks.rgw_multisite:creating admin user.. 2023-05-24T20:59:47.944 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'Z6KKVUYPP0489K03', '--secret', 'EX5IDzHTe9AVqa8b1PSD8B5Y1CKI3jBg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T20:59:47.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'Z6KKVUYPP0489K03', '--secret', 'EX5IDzHTe9AVqa8b1PSD8B5Y1CKI3jBg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T20:59:47.945 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key Z6KKVUYPP0489K03 --secret EX5IDzHTe9AVqa8b1PSD8B5Y1CKI3jBg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-24T20:59:48.039 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T20:59:48.039 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T20:59:48.096 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T20:59:48.096 INFO:teuthology.orchestra.run.smithi071.stdout: "user_id": "realm-admin", 2023-05-24T20:59:48.096 INFO:teuthology.orchestra.run.smithi071.stdout: "display_name": "Realm Admin", 2023-05-24T20:59:48.096 INFO:teuthology.orchestra.run.smithi071.stdout: "email": "", 2023-05-24T20:59:48.096 INFO:teuthology.orchestra.run.smithi071.stdout: "suspended": 0, 2023-05-24T20:59:48.097 INFO:teuthology.orchestra.run.smithi071.stdout: "max_buckets": 1000, 2023-05-24T20:59:48.097 INFO:teuthology.orchestra.run.smithi071.stdout: "subusers": [], 2023-05-24T20:59:48.097 INFO:teuthology.orchestra.run.smithi071.stdout: "keys": [ 2023-05-24T20:59:48.097 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T20:59:48.097 INFO:teuthology.orchestra.run.smithi071.stdout: "user": "realm-admin", 2023-05-24T20:59:48.097 INFO:teuthology.orchestra.run.smithi071.stdout: "access_key": "Z6KKVUYPP0489K03", 2023-05-24T20:59:48.097 INFO:teuthology.orchestra.run.smithi071.stdout: "secret_key": "EX5IDzHTe9AVqa8b1PSD8B5Y1CKI3jBg" 2023-05-24T20:59:48.097 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T20:59:48.098 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T20:59:48.098 INFO:teuthology.orchestra.run.smithi071.stdout: "swift_keys": [], 2023-05-24T20:59:48.098 INFO:teuthology.orchestra.run.smithi071.stdout: "caps": [], 2023-05-24T20:59:48.098 INFO:teuthology.orchestra.run.smithi071.stdout: "op_mask": "read, write, delete", 2023-05-24T20:59:48.099 INFO:teuthology.orchestra.run.smithi071.stdout: "system": true, 2023-05-24T20:59:48.099 INFO:teuthology.orchestra.run.smithi071.stdout: "default_placement": "", 2023-05-24T20:59:48.099 INFO:teuthology.orchestra.run.smithi071.stdout: "default_storage_class": "", 2023-05-24T20:59:48.099 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_tags": [], 2023-05-24T20:59:48.099 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_quota": { 2023-05-24T20:59:48.099 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false, 2023-05-24T20:59:48.099 INFO:teuthology.orchestra.run.smithi071.stdout: "check_on_raw": false, 2023-05-24T20:59:48.100 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size": -1, 2023-05-24T20:59:48.100 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size_kb": 0, 2023-05-24T20:59:48.100 INFO:teuthology.orchestra.run.smithi071.stdout: "max_objects": -1 2023-05-24T20:59:48.100 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T20:59:48.100 INFO:teuthology.orchestra.run.smithi071.stdout: "user_quota": { 2023-05-24T20:59:48.100 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false, 2023-05-24T20:59:48.100 INFO:teuthology.orchestra.run.smithi071.stdout: "check_on_raw": false, 2023-05-24T20:59:48.100 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size": -1, 2023-05-24T20:59:48.101 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size_kb": 0, 2023-05-24T20:59:48.101 INFO:teuthology.orchestra.run.smithi071.stdout: "max_objects": -1 2023-05-24T20:59:48.101 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T20:59:48.101 INFO:teuthology.orchestra.run.smithi071.stdout: "temp_url_keys": [], 2023-05-24T20:59:48.101 INFO:teuthology.orchestra.run.smithi071.stdout: "type": "rgw", 2023-05-24T20:59:48.101 INFO:teuthology.orchestra.run.smithi071.stdout: "mfa_ids": [] 2023-05-24T20:59:48.101 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T20:59:48.101 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T20:59:48.115 INFO:tasks.rgw_multisite:creating zone.. 2023-05-24T20:59:48.116 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', 'Z6KKVUYPP0489K03', '--secret', 'EX5IDzHTe9AVqa8b1PSD8B5Y1CKI3jBg', '--endpoints', 'http://smithi071.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T20:59:48.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', 'Z6KKVUYPP0489K03', '--secret', 'EX5IDzHTe9AVqa8b1PSD8B5Y1CKI3jBg', '--endpoints', 'http://smithi071.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T20:59:48.116 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --access-key Z6KKVUYPP0489K03 --secret EX5IDzHTe9AVqa8b1PSD8B5Y1CKI3jBg --endpoints http://smithi071.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-24T20:59:48.169 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T20:59:48.170 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T20:59:48.184 INFO:teuthology.orchestra.run.smithi071.stderr:NOTICE: set zone's realm_id=4a7754f9-b4ac-4d00-9827-3910f2de91d2 2023-05-24T20:59:48.199 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T20:59:48.199 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "5af1588c-f9b9-4687-a040-ce6e0cc808d9", 2023-05-24T20:59:48.199 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a2", 2023-05-24T20:59:48.200 INFO:teuthology.orchestra.run.smithi071.stdout: "domain_root": "a2.rgw.meta:root", 2023-05-24T20:59:48.200 INFO:teuthology.orchestra.run.smithi071.stdout: "control_pool": "a2.rgw.control", 2023-05-24T20:59:48.200 INFO:teuthology.orchestra.run.smithi071.stdout: "gc_pool": "a2.rgw.log:gc", 2023-05-24T20:59:48.200 INFO:teuthology.orchestra.run.smithi071.stdout: "lc_pool": "a2.rgw.log:lc", 2023-05-24T20:59:48.200 INFO:teuthology.orchestra.run.smithi071.stdout: "log_pool": "a2.rgw.log", 2023-05-24T20:59:48.200 INFO:teuthology.orchestra.run.smithi071.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-05-24T20:59:48.200 INFO:teuthology.orchestra.run.smithi071.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-05-24T20:59:48.200 INFO:teuthology.orchestra.run.smithi071.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-05-24T20:59:48.201 INFO:teuthology.orchestra.run.smithi071.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-05-24T20:59:48.201 INFO:teuthology.orchestra.run.smithi071.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-05-24T20:59:48.201 INFO:teuthology.orchestra.run.smithi071.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-05-24T20:59:48.201 INFO:teuthology.orchestra.run.smithi071.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-05-24T20:59:48.201 INFO:teuthology.orchestra.run.smithi071.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-05-24T20:59:48.201 INFO:teuthology.orchestra.run.smithi071.stdout: "otp_pool": "a2.rgw.otp", 2023-05-24T20:59:48.201 INFO:teuthology.orchestra.run.smithi071.stdout: "system_key": { 2023-05-24T20:59:48.202 INFO:teuthology.orchestra.run.smithi071.stdout: "access_key": "Z6KKVUYPP0489K03", 2023-05-24T20:59:48.202 INFO:teuthology.orchestra.run.smithi071.stdout: "secret_key": "EX5IDzHTe9AVqa8b1PSD8B5Y1CKI3jBg" 2023-05-24T20:59:48.202 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T20:59:48.202 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_pools": [ 2023-05-24T20:59:48.202 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T20:59:48.202 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "default-placement", 2023-05-24T20:59:48.202 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T20:59:48.202 INFO:teuthology.orchestra.run.smithi071.stdout: "index_pool": "a2.rgw.buckets.index", 2023-05-24T20:59:48.203 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": { 2023-05-24T20:59:48.203 INFO:teuthology.orchestra.run.smithi071.stdout: "STANDARD": { 2023-05-24T20:59:48.203 INFO:teuthology.orchestra.run.smithi071.stdout: "data_pool": "a2.rgw.buckets.data" 2023-05-24T20:59:48.203 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T20:59:48.203 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T20:59:48.203 INFO:teuthology.orchestra.run.smithi071.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-05-24T20:59:48.203 INFO:teuthology.orchestra.run.smithi071.stdout: "index_type": 0, 2023-05-24T20:59:48.203 INFO:teuthology.orchestra.run.smithi071.stdout: "inline_data": true 2023-05-24T20:59:48.204 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T20:59:48.204 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T20:59:48.204 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T20:59:48.204 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T20:59:48.204 INFO:teuthology.orchestra.run.smithi071.stdout: "notif_pool": "a2.rgw.log:notif" 2023-05-24T20:59:48.204 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T20:59:48.205 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph osd pool create a2.rgw.buckets.data 64 64 --cluster c1 2023-05-24T20:59:48.728 INFO:teuthology.orchestra.run.smithi071.stderr:pool 'a2.rgw.buckets.data' created 2023-05-24T20:59:48.743 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph osd pool application enable a2.rgw.buckets.data rgw --cluster c1 2023-05-24T20:59:49.748 INFO:teuthology.orchestra.run.smithi071.stderr:enabled application 'rgw' on pool 'a2.rgw.buckets.data' 2023-05-24T20:59:49.762 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T20:59:49.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T20:59:49.763 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-24T20:59:49.817 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T20:59:49.817 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T20:59:49.835 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T20:59:49.836 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "5af1588c-f9b9-4687-a040-ce6e0cc808d9", 2023-05-24T20:59:49.836 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a2", 2023-05-24T20:59:49.836 INFO:teuthology.orchestra.run.smithi071.stdout: "domain_root": "a2.rgw.meta:root", 2023-05-24T20:59:49.836 INFO:teuthology.orchestra.run.smithi071.stdout: "control_pool": "a2.rgw.control", 2023-05-24T20:59:49.836 INFO:teuthology.orchestra.run.smithi071.stdout: "gc_pool": "a2.rgw.log:gc", 2023-05-24T20:59:49.836 INFO:teuthology.orchestra.run.smithi071.stdout: "lc_pool": "a2.rgw.log:lc", 2023-05-24T20:59:49.836 INFO:teuthology.orchestra.run.smithi071.stdout: "log_pool": "a2.rgw.log", 2023-05-24T20:59:49.837 INFO:teuthology.orchestra.run.smithi071.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-05-24T20:59:49.837 INFO:teuthology.orchestra.run.smithi071.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-05-24T20:59:49.837 INFO:teuthology.orchestra.run.smithi071.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-05-24T20:59:49.837 INFO:teuthology.orchestra.run.smithi071.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-05-24T20:59:49.837 INFO:teuthology.orchestra.run.smithi071.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-05-24T20:59:49.837 INFO:teuthology.orchestra.run.smithi071.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-05-24T20:59:49.837 INFO:teuthology.orchestra.run.smithi071.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-05-24T20:59:49.838 INFO:teuthology.orchestra.run.smithi071.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-05-24T20:59:49.838 INFO:teuthology.orchestra.run.smithi071.stdout: "otp_pool": "a2.rgw.otp", 2023-05-24T20:59:49.838 INFO:teuthology.orchestra.run.smithi071.stdout: "system_key": { 2023-05-24T20:59:49.838 INFO:teuthology.orchestra.run.smithi071.stdout: "access_key": "Z6KKVUYPP0489K03", 2023-05-24T20:59:49.838 INFO:teuthology.orchestra.run.smithi071.stdout: "secret_key": "EX5IDzHTe9AVqa8b1PSD8B5Y1CKI3jBg" 2023-05-24T20:59:49.838 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T20:59:49.838 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_pools": [ 2023-05-24T20:59:49.838 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T20:59:49.839 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "default-placement", 2023-05-24T20:59:49.839 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T20:59:49.839 INFO:teuthology.orchestra.run.smithi071.stdout: "index_pool": "a2.rgw.buckets.index", 2023-05-24T20:59:49.839 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": { 2023-05-24T20:59:49.839 INFO:teuthology.orchestra.run.smithi071.stdout: "STANDARD": { 2023-05-24T20:59:49.839 INFO:teuthology.orchestra.run.smithi071.stdout: "data_pool": "a2.rgw.buckets.data", 2023-05-24T20:59:49.839 INFO:teuthology.orchestra.run.smithi071.stdout: "compression_type": "random" 2023-05-24T20:59:49.840 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T20:59:49.840 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T20:59:49.840 INFO:teuthology.orchestra.run.smithi071.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-05-24T20:59:49.840 INFO:teuthology.orchestra.run.smithi071.stdout: "index_type": 0, 2023-05-24T20:59:49.840 INFO:teuthology.orchestra.run.smithi071.stdout: "inline_data": true 2023-05-24T20:59:49.840 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T20:59:49.840 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T20:59:49.840 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T20:59:49.841 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T20:59:49.841 INFO:teuthology.orchestra.run.smithi071.stdout: "notif_pool": "a2.rgw.log:notif" 2023-05-24T20:59:49.841 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T20:59:49.842 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T20:59:49.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T20:59:49.843 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-24T20:59:49.877 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2023-05-24T20:59:49.890 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T20:59:49.891 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T20:59:49.902 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T20:59:49.895+0000 7efd40601780 0 period (5c04936b-70ea-4415-b4eb-3aad3dc612e5 does not have zone 5af1588c-f9b9-4687-a040-ce6e0cc808d9 configured 2023-05-24T20:59:56.776 INFO:teuthology.orchestra.run.smithi071.stderr:Sending period to new master zone 299a3349-fae0-459e-8632-431fc36beb1b 2023-05-24T20:59:56.808 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T20:59:56.809 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T20:59:56.809 INFO:teuthology.orchestra.run.smithi071.stdout: "epoch": 2, 2023-05-24T20:59:56.809 INFO:teuthology.orchestra.run.smithi071.stdout: "predecessor_uuid": "1cea3a6e-eb52-480e-95b7-3f6e43317e60", 2023-05-24T20:59:56.809 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": [], 2023-05-24T20:59:56.809 INFO:teuthology.orchestra.run.smithi071.stdout: "period_map": { 2023-05-24T20:59:56.810 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T20:59:56.810 INFO:teuthology.orchestra.run.smithi071.stdout: "zonegroups": [ 2023-05-24T20:59:56.810 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T20:59:56.810 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "a", 2023-05-24T20:59:56.810 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a", 2023-05-24T20:59:56.810 INFO:teuthology.orchestra.run.smithi071.stdout: "api_name": "", 2023-05-24T20:59:56.811 INFO:teuthology.orchestra.run.smithi071.stdout: "is_master": true, 2023-05-24T20:59:56.811 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T20:59:56.811 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-05-24T20:59:56.811 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T20:59:56.811 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames": [], 2023-05-24T20:59:56.811 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames_s3website": [], 2023-05-24T20:59:56.811 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zone": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T20:59:56.811 INFO:teuthology.orchestra.run.smithi071.stdout: "zones": [ 2023-05-24T20:59:56.812 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T20:59:56.812 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T20:59:56.812 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a1", 2023-05-24T20:59:56.812 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T20:59:56.812 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-05-24T20:59:56.812 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T20:59:56.813 INFO:teuthology.orchestra.run.smithi071.stdout: "log_meta": false, 2023-05-24T20:59:56.813 INFO:teuthology.orchestra.run.smithi071.stdout: "log_data": true, 2023-05-24T20:59:56.813 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_index_max_shards": 11, 2023-05-24T20:59:56.813 INFO:teuthology.orchestra.run.smithi071.stdout: "read_only": false, 2023-05-24T20:59:56.813 INFO:teuthology.orchestra.run.smithi071.stdout: "tier_type": "", 2023-05-24T20:59:56.813 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from_all": true, 2023-05-24T20:59:56.813 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from": [], 2023-05-24T20:59:56.814 INFO:teuthology.orchestra.run.smithi071.stdout: "redirect_zone": "", 2023-05-24T20:59:56.814 INFO:teuthology.orchestra.run.smithi071.stdout: "supported_features": [ 2023-05-24T20:59:56.814 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T20:59:56.814 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T20:59:56.814 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T20:59:56.814 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T20:59:56.814 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "5af1588c-f9b9-4687-a040-ce6e0cc808d9", 2023-05-24T20:59:56.814 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a2", 2023-05-24T20:59:56.815 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T20:59:56.815 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi071.front.sepia.ceph.com:8001" 2023-05-24T20:59:56.815 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T20:59:56.815 INFO:teuthology.orchestra.run.smithi071.stdout: "log_meta": false, 2023-05-24T20:59:56.815 INFO:teuthology.orchestra.run.smithi071.stdout: "log_data": true, 2023-05-24T20:59:56.815 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_index_max_shards": 11, 2023-05-24T20:59:56.815 INFO:teuthology.orchestra.run.smithi071.stdout: "read_only": false, 2023-05-24T20:59:56.816 INFO:teuthology.orchestra.run.smithi071.stdout: "tier_type": "", 2023-05-24T20:59:56.816 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from_all": true, 2023-05-24T20:59:56.816 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from": [], 2023-05-24T20:59:56.816 INFO:teuthology.orchestra.run.smithi071.stdout: "redirect_zone": "", 2023-05-24T20:59:56.816 INFO:teuthology.orchestra.run.smithi071.stdout: "supported_features": [ 2023-05-24T20:59:56.816 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T20:59:56.817 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T20:59:56.820 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T20:59:56.820 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T20:59:56.821 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_targets": [ 2023-05-24T20:59:56.821 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T20:59:56.821 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "default-placement", 2023-05-24T20:59:56.821 INFO:teuthology.orchestra.run.smithi071.stdout: "tags": [], 2023-05-24T20:59:56.821 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": [ 2023-05-24T20:59:56.821 INFO:teuthology.orchestra.run.smithi071.stdout: "STANDARD" 2023-05-24T20:59:56.821 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T20:59:56.822 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T20:59:56.822 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T20:59:56.822 INFO:teuthology.orchestra.run.smithi071.stdout: "default_placement": "default-placement", 2023-05-24T20:59:56.822 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T20:59:56.822 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_policy": { 2023-05-24T20:59:56.822 INFO:teuthology.orchestra.run.smithi071.stdout: "groups": [] 2023-05-24T20:59:56.822 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T20:59:56.823 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled_features": [ 2023-05-24T20:59:56.823 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T20:59:56.823 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T20:59:56.823 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T20:59:56.823 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T20:59:56.823 INFO:teuthology.orchestra.run.smithi071.stdout: "short_zone_ids": [ 2023-05-24T20:59:56.823 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T20:59:56.824 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T20:59:56.824 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 3764661814 2023-05-24T20:59:56.824 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T20:59:56.824 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T20:59:56.824 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "5af1588c-f9b9-4687-a040-ce6e0cc808d9", 2023-05-24T20:59:56.824 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 2266179370 2023-05-24T20:59:56.825 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T20:59:56.825 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T20:59:56.825 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T20:59:56.825 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zonegroup": "a", 2023-05-24T20:59:56.825 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zone": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T20:59:56.825 INFO:teuthology.orchestra.run.smithi071.stdout: "period_config": { 2023-05-24T20:59:56.825 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_quota": { 2023-05-24T20:59:56.826 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false, 2023-05-24T20:59:56.826 INFO:teuthology.orchestra.run.smithi071.stdout: "check_on_raw": false, 2023-05-24T20:59:56.826 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size": -1, 2023-05-24T20:59:56.826 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size_kb": 0, 2023-05-24T20:59:56.826 INFO:teuthology.orchestra.run.smithi071.stdout: "max_objects": -1 2023-05-24T20:59:56.826 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T20:59:56.826 INFO:teuthology.orchestra.run.smithi071.stdout: "user_quota": { 2023-05-24T20:59:56.826 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false, 2023-05-24T20:59:56.827 INFO:teuthology.orchestra.run.smithi071.stdout: "check_on_raw": false, 2023-05-24T20:59:56.827 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size": -1, 2023-05-24T20:59:56.827 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size_kb": 0, 2023-05-24T20:59:56.827 INFO:teuthology.orchestra.run.smithi071.stdout: "max_objects": -1 2023-05-24T20:59:56.827 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T20:59:56.827 INFO:teuthology.orchestra.run.smithi071.stdout: "user_ratelimit": { 2023-05-24T20:59:56.827 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_ops": 0, 2023-05-24T20:59:56.828 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_ops": 0, 2023-05-24T20:59:56.828 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_bytes": 0, 2023-05-24T20:59:56.828 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_bytes": 0, 2023-05-24T20:59:56.828 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false 2023-05-24T20:59:56.828 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T20:59:56.829 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_ratelimit": { 2023-05-24T20:59:56.829 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_ops": 0, 2023-05-24T20:59:56.829 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_ops": 0, 2023-05-24T20:59:56.829 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_bytes": 0, 2023-05-24T20:59:56.829 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_bytes": 0, 2023-05-24T20:59:56.829 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false 2023-05-24T20:59:56.829 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T20:59:56.830 INFO:teuthology.orchestra.run.smithi071.stdout: "anonymous_ratelimit": { 2023-05-24T20:59:56.830 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_ops": 0, 2023-05-24T20:59:56.831 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_ops": 0, 2023-05-24T20:59:56.831 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_bytes": 0, 2023-05-24T20:59:56.831 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_bytes": 0, 2023-05-24T20:59:56.831 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false 2023-05-24T20:59:56.831 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T20:59:56.831 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T20:59:56.832 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T20:59:56.832 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_name": "test-realm", 2023-05-24T20:59:56.832 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T20:59:56.832 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T20:59:56.833 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-05-24T20:59:56.833 INFO:teuthology.orchestra.run:waiting for 300 2023-05-24T20:59:56.834 INFO:tasks.rgw.c1.client.1.smithi071.stdout:2023-05-24T20:59:56.828+0000 7f41c8e33700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 110150) UID: 0 2023-05-24T20:59:56.834 INFO:tasks.rgw.c1.client.1.smithi071.stdout:2023-05-24T20:59:56.828+0000 7f41cd09c7c0 -1 shutting down 2023-05-24T21:00:01.187 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-05-24T21:00:02.935 INFO:tasks.rgw.c1.client.1:Stopped 2023-05-24T21:00:02.935 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-05-24T21:00:02.936 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-05-24T21:00:02.939 INFO:tasks.rgw.c1.client.1:Started 2023-05-24T21:00:02.939 DEBUG:teuthology.orchestra.run.smithi071:> curl http://smithi071.front.sepia.ceph.com:8001 2023-05-24T21:00:02.965 INFO:teuthology.orchestra.run.smithi071.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-05-24T21:00:02.965 INFO:teuthology.orchestra.run.smithi071.stderr: Dload Upload Total Spent Left Speed 2023-05-24T21:00:02.966 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-05-24T21:00:02.967 INFO:teuthology.orchestra.run.smithi071.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi071.front.sepia.ceph.com port 8001: Connection refused 2023-05-24T21:00:03.968 DEBUG:teuthology.orchestra.run.smithi071:> curl http://smithi071.front.sepia.ceph.com:8001 2023-05-24T21:00:03.991 INFO:teuthology.orchestra.run.smithi071.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-05-24T21:00:03.991 INFO:teuthology.orchestra.run.smithi071.stderr: Dload Upload Total Spent Left Speed 2023-05-24T21:00:03.992 INFO:teuthology.orchestra.run.smithi071.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-05-24T21:00:03.992 INFO:teuthology.orchestra.run.smithi071.stdout:anonymous 2023-05-24T21:00:03.993 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2023-05-24T21:00:03.993 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi071.front.sepia.ceph.com:8000', '--access-key', 'Z6KKVUYPP0489K03', '--secret', 'EX5IDzHTe9AVqa8b1PSD8B5Y1CKI3jBg', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:00:03.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi071.front.sepia.ceph.com:8000', '--access-key', 'Z6KKVUYPP0489K03', '--secret', 'EX5IDzHTe9AVqa8b1PSD8B5Y1CKI3jBg', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:00:03.994 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi071.front.sepia.ceph.com:8000 --access-key Z6KKVUYPP0489K03 --secret EX5IDzHTe9AVqa8b1PSD8B5Y1CKI3jBg --cluster c2 --debug-rgw 0 --debug-ms 0 2023-05-24T21:00:04.054 INFO:teuthology.orchestra.run.smithi157.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:00:04.054 INFO:teuthology.orchestra.run.smithi157.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:00:04.092 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2023-05-24T21:00:04.092 INFO:teuthology.orchestra.run.smithi157.stdout: "id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T21:00:04.093 INFO:teuthology.orchestra.run.smithi157.stdout: "name": "test-realm", 2023-05-24T21:00:04.093 INFO:teuthology.orchestra.run.smithi157.stdout: "current_period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:00:04.093 INFO:teuthology.orchestra.run.smithi157.stdout: "epoch": 2 2023-05-24T21:00:04.093 INFO:teuthology.orchestra.run.smithi157.stdout:} 2023-05-24T21:00:04.093 INFO:tasks.rgw_multisite:creating zonegroup.. 2023-05-24T21:00:04.094 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:00:04.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zonegroup', 'set', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:00:04.094 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zonegroup set --rgw-zonegroup b --rgw-realm test-realm --default --cluster c2 --debug-rgw 0 --debug-ms 0 2023-05-24T21:00:04.144 INFO:teuthology.orchestra.run.smithi157.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:00:04.145 INFO:teuthology.orchestra.run.smithi157.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:00:04.159 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T21:00:04.152+0000 7f2a2c3d5780 -1 old format 2023-05-24T21:00:04.165 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2023-05-24T21:00:04.165 INFO:teuthology.orchestra.run.smithi157.stdout: "id": "b", 2023-05-24T21:00:04.165 INFO:teuthology.orchestra.run.smithi157.stdout: "name": "b", 2023-05-24T21:00:04.165 INFO:teuthology.orchestra.run.smithi157.stdout: "api_name": "", 2023-05-24T21:00:04.165 INFO:teuthology.orchestra.run.smithi157.stdout: "is_master": false, 2023-05-24T21:00:04.165 INFO:teuthology.orchestra.run.smithi157.stdout: "endpoints": [ 2023-05-24T21:00:04.166 INFO:teuthology.orchestra.run.smithi157.stdout: "http://smithi157.front.sepia.ceph.com:8000" 2023-05-24T21:00:04.166 INFO:teuthology.orchestra.run.smithi157.stdout: ], 2023-05-24T21:00:04.166 INFO:teuthology.orchestra.run.smithi157.stdout: "hostnames": [], 2023-05-24T21:00:04.166 INFO:teuthology.orchestra.run.smithi157.stdout: "hostnames_s3website": [], 2023-05-24T21:00:04.166 INFO:teuthology.orchestra.run.smithi157.stdout: "master_zone": "", 2023-05-24T21:00:04.166 INFO:teuthology.orchestra.run.smithi157.stdout: "zones": [], 2023-05-24T21:00:04.166 INFO:teuthology.orchestra.run.smithi157.stdout: "placement_targets": [ 2023-05-24T21:00:04.166 INFO:teuthology.orchestra.run.smithi157.stdout: { 2023-05-24T21:00:04.167 INFO:teuthology.orchestra.run.smithi157.stdout: "name": "default-placement", 2023-05-24T21:00:04.167 INFO:teuthology.orchestra.run.smithi157.stdout: "tags": [], 2023-05-24T21:00:04.167 INFO:teuthology.orchestra.run.smithi157.stdout: "storage_classes": [] 2023-05-24T21:00:04.167 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:00:04.167 INFO:teuthology.orchestra.run.smithi157.stdout: ], 2023-05-24T21:00:04.167 INFO:teuthology.orchestra.run.smithi157.stdout: "default_placement": "default-placement", 2023-05-24T21:00:04.167 INFO:teuthology.orchestra.run.smithi157.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T21:00:04.168 INFO:teuthology.orchestra.run.smithi157.stdout: "sync_policy": { 2023-05-24T21:00:04.168 INFO:teuthology.orchestra.run.smithi157.stdout: "groups": [] 2023-05-24T21:00:04.168 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:00:04.168 INFO:teuthology.orchestra.run.smithi157.stdout: "enabled_features": [ 2023-05-24T21:00:04.168 INFO:teuthology.orchestra.run.smithi157.stdout: "resharding" 2023-05-24T21:00:04.168 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2023-05-24T21:00:04.168 INFO:teuthology.orchestra.run.smithi157.stdout:} 2023-05-24T21:00:04.169 INFO:tasks.rgw_multisite:creating zone.. 2023-05-24T21:00:04.170 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'Z6KKVUYPP0489K03', '--secret', 'EX5IDzHTe9AVqa8b1PSD8B5Y1CKI3jBg', '--endpoints', 'http://smithi157.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:00:04.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'Z6KKVUYPP0489K03', '--secret', 'EX5IDzHTe9AVqa8b1PSD8B5Y1CKI3jBg', '--endpoints', 'http://smithi157.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:00:04.170 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --default --master --access-key Z6KKVUYPP0489K03 --secret EX5IDzHTe9AVqa8b1PSD8B5Y1CKI3jBg --endpoints http://smithi157.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2023-05-24T21:00:04.266 INFO:teuthology.orchestra.run.smithi157.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:00:04.266 INFO:teuthology.orchestra.run.smithi157.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:00:04.281 INFO:teuthology.orchestra.run.smithi157.stderr:NOTICE: set zone's realm_id=4a7754f9-b4ac-4d00-9827-3910f2de91d2 2023-05-24T21:00:04.296 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2023-05-24T21:00:04.296 INFO:teuthology.orchestra.run.smithi157.stdout: "id": "ee18cd86-4684-4b3d-8de6-0075d3d925f1", 2023-05-24T21:00:04.297 INFO:teuthology.orchestra.run.smithi157.stdout: "name": "b1", 2023-05-24T21:00:04.297 INFO:teuthology.orchestra.run.smithi157.stdout: "domain_root": "b1.rgw.meta:root", 2023-05-24T21:00:04.297 INFO:teuthology.orchestra.run.smithi157.stdout: "control_pool": "b1.rgw.control", 2023-05-24T21:00:04.297 INFO:teuthology.orchestra.run.smithi157.stdout: "gc_pool": "b1.rgw.log:gc", 2023-05-24T21:00:04.297 INFO:teuthology.orchestra.run.smithi157.stdout: "lc_pool": "b1.rgw.log:lc", 2023-05-24T21:00:04.297 INFO:teuthology.orchestra.run.smithi157.stdout: "log_pool": "b1.rgw.log", 2023-05-24T21:00:04.298 INFO:teuthology.orchestra.run.smithi157.stdout: "intent_log_pool": "b1.rgw.log:intent", 2023-05-24T21:00:04.298 INFO:teuthology.orchestra.run.smithi157.stdout: "usage_log_pool": "b1.rgw.log:usage", 2023-05-24T21:00:04.298 INFO:teuthology.orchestra.run.smithi157.stdout: "roles_pool": "b1.rgw.meta:roles", 2023-05-24T21:00:04.298 INFO:teuthology.orchestra.run.smithi157.stdout: "reshard_pool": "b1.rgw.log:reshard", 2023-05-24T21:00:04.298 INFO:teuthology.orchestra.run.smithi157.stdout: "user_keys_pool": "b1.rgw.meta:users.keys", 2023-05-24T21:00:04.298 INFO:teuthology.orchestra.run.smithi157.stdout: "user_email_pool": "b1.rgw.meta:users.email", 2023-05-24T21:00:04.299 INFO:teuthology.orchestra.run.smithi157.stdout: "user_swift_pool": "b1.rgw.meta:users.swift", 2023-05-24T21:00:04.299 INFO:teuthology.orchestra.run.smithi157.stdout: "user_uid_pool": "b1.rgw.meta:users.uid", 2023-05-24T21:00:04.299 INFO:teuthology.orchestra.run.smithi157.stdout: "otp_pool": "b1.rgw.otp", 2023-05-24T21:00:04.299 INFO:teuthology.orchestra.run.smithi157.stdout: "system_key": { 2023-05-24T21:00:04.300 INFO:teuthology.orchestra.run.smithi157.stdout: "access_key": "Z6KKVUYPP0489K03", 2023-05-24T21:00:04.300 INFO:teuthology.orchestra.run.smithi157.stdout: "secret_key": "EX5IDzHTe9AVqa8b1PSD8B5Y1CKI3jBg" 2023-05-24T21:00:04.300 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:00:04.300 INFO:teuthology.orchestra.run.smithi157.stdout: "placement_pools": [ 2023-05-24T21:00:04.300 INFO:teuthology.orchestra.run.smithi157.stdout: { 2023-05-24T21:00:04.300 INFO:teuthology.orchestra.run.smithi157.stdout: "key": "default-placement", 2023-05-24T21:00:04.300 INFO:teuthology.orchestra.run.smithi157.stdout: "val": { 2023-05-24T21:00:04.301 INFO:teuthology.orchestra.run.smithi157.stdout: "index_pool": "b1.rgw.buckets.index", 2023-05-24T21:00:04.301 INFO:teuthology.orchestra.run.smithi157.stdout: "storage_classes": { 2023-05-24T21:00:04.301 INFO:teuthology.orchestra.run.smithi157.stdout: "STANDARD": { 2023-05-24T21:00:04.301 INFO:teuthology.orchestra.run.smithi157.stdout: "data_pool": "b1.rgw.buckets.data" 2023-05-24T21:00:04.301 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:00:04.301 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:00:04.301 INFO:teuthology.orchestra.run.smithi157.stdout: "data_extra_pool": "b1.rgw.buckets.non-ec", 2023-05-24T21:00:04.302 INFO:teuthology.orchestra.run.smithi157.stdout: "index_type": 0, 2023-05-24T21:00:04.302 INFO:teuthology.orchestra.run.smithi157.stdout: "inline_data": true 2023-05-24T21:00:04.302 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:00:04.302 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:00:04.302 INFO:teuthology.orchestra.run.smithi157.stdout: ], 2023-05-24T21:00:04.302 INFO:teuthology.orchestra.run.smithi157.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T21:00:04.302 INFO:teuthology.orchestra.run.smithi157.stdout: "notif_pool": "b1.rgw.log:notif" 2023-05-24T21:00:04.302 INFO:teuthology.orchestra.run.smithi157.stdout:} 2023-05-24T21:00:04.303 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph osd pool create b1.rgw.buckets.data 64 64 --cluster c2 2023-05-24T21:00:05.581 INFO:teuthology.orchestra.run.smithi157.stderr:pool 'b1.rgw.buckets.data' created 2023-05-24T21:00:05.598 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph osd pool application enable b1.rgw.buckets.data rgw --cluster c2 2023-05-24T21:00:06.589 INFO:teuthology.orchestra.run.smithi157.stderr:enabled application 'rgw' on pool 'b1.rgw.buckets.data' 2023-05-24T21:00:06.610 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:00:06.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:00:06.611 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2023-05-24T21:00:06.665 INFO:teuthology.orchestra.run.smithi157.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:00:06.666 INFO:teuthology.orchestra.run.smithi157.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:00:06.684 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2023-05-24T21:00:06.685 INFO:teuthology.orchestra.run.smithi157.stdout: "id": "ee18cd86-4684-4b3d-8de6-0075d3d925f1", 2023-05-24T21:00:06.685 INFO:teuthology.orchestra.run.smithi157.stdout: "name": "b1", 2023-05-24T21:00:06.685 INFO:teuthology.orchestra.run.smithi157.stdout: "domain_root": "b1.rgw.meta:root", 2023-05-24T21:00:06.686 INFO:teuthology.orchestra.run.smithi157.stdout: "control_pool": "b1.rgw.control", 2023-05-24T21:00:06.686 INFO:teuthology.orchestra.run.smithi157.stdout: "gc_pool": "b1.rgw.log:gc", 2023-05-24T21:00:06.686 INFO:teuthology.orchestra.run.smithi157.stdout: "lc_pool": "b1.rgw.log:lc", 2023-05-24T21:00:06.686 INFO:teuthology.orchestra.run.smithi157.stdout: "log_pool": "b1.rgw.log", 2023-05-24T21:00:06.686 INFO:teuthology.orchestra.run.smithi157.stdout: "intent_log_pool": "b1.rgw.log:intent", 2023-05-24T21:00:06.687 INFO:teuthology.orchestra.run.smithi157.stdout: "usage_log_pool": "b1.rgw.log:usage", 2023-05-24T21:00:06.687 INFO:teuthology.orchestra.run.smithi157.stdout: "roles_pool": "b1.rgw.meta:roles", 2023-05-24T21:00:06.687 INFO:teuthology.orchestra.run.smithi157.stdout: "reshard_pool": "b1.rgw.log:reshard", 2023-05-24T21:00:06.687 INFO:teuthology.orchestra.run.smithi157.stdout: "user_keys_pool": "b1.rgw.meta:users.keys", 2023-05-24T21:00:06.688 INFO:teuthology.orchestra.run.smithi157.stdout: "user_email_pool": "b1.rgw.meta:users.email", 2023-05-24T21:00:06.688 INFO:teuthology.orchestra.run.smithi157.stdout: "user_swift_pool": "b1.rgw.meta:users.swift", 2023-05-24T21:00:06.688 INFO:teuthology.orchestra.run.smithi157.stdout: "user_uid_pool": "b1.rgw.meta:users.uid", 2023-05-24T21:00:06.688 INFO:teuthology.orchestra.run.smithi157.stdout: "otp_pool": "b1.rgw.otp", 2023-05-24T21:00:06.689 INFO:teuthology.orchestra.run.smithi157.stdout: "system_key": { 2023-05-24T21:00:06.689 INFO:teuthology.orchestra.run.smithi157.stdout: "access_key": "Z6KKVUYPP0489K03", 2023-05-24T21:00:06.689 INFO:teuthology.orchestra.run.smithi157.stdout: "secret_key": "EX5IDzHTe9AVqa8b1PSD8B5Y1CKI3jBg" 2023-05-24T21:00:06.689 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:00:06.689 INFO:teuthology.orchestra.run.smithi157.stdout: "placement_pools": [ 2023-05-24T21:00:06.690 INFO:teuthology.orchestra.run.smithi157.stdout: { 2023-05-24T21:00:06.690 INFO:teuthology.orchestra.run.smithi157.stdout: "key": "default-placement", 2023-05-24T21:00:06.690 INFO:teuthology.orchestra.run.smithi157.stdout: "val": { 2023-05-24T21:00:06.690 INFO:teuthology.orchestra.run.smithi157.stdout: "index_pool": "b1.rgw.buckets.index", 2023-05-24T21:00:06.691 INFO:teuthology.orchestra.run.smithi157.stdout: "storage_classes": { 2023-05-24T21:00:06.691 INFO:teuthology.orchestra.run.smithi157.stdout: "STANDARD": { 2023-05-24T21:00:06.691 INFO:teuthology.orchestra.run.smithi157.stdout: "data_pool": "b1.rgw.buckets.data", 2023-05-24T21:00:06.691 INFO:teuthology.orchestra.run.smithi157.stdout: "compression_type": "random" 2023-05-24T21:00:06.691 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:00:06.692 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:00:06.692 INFO:teuthology.orchestra.run.smithi157.stdout: "data_extra_pool": "b1.rgw.buckets.non-ec", 2023-05-24T21:00:06.692 INFO:teuthology.orchestra.run.smithi157.stdout: "index_type": 0, 2023-05-24T21:00:06.692 INFO:teuthology.orchestra.run.smithi157.stdout: "inline_data": true 2023-05-24T21:00:06.693 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:00:06.693 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:00:06.693 INFO:teuthology.orchestra.run.smithi157.stdout: ], 2023-05-24T21:00:06.693 INFO:teuthology.orchestra.run.smithi157.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T21:00:06.694 INFO:teuthology.orchestra.run.smithi157.stdout: "notif_pool": "b1.rgw.log:notif" 2023-05-24T21:00:06.694 INFO:teuthology.orchestra.run.smithi157.stdout:} 2023-05-24T21:00:06.695 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:00:06.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:00:06.696 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2023-05-24T21:00:06.746 INFO:teuthology.orchestra.run.smithi157.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:00:06.747 INFO:teuthology.orchestra.run.smithi157.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:00:06.757 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T21:00:06.751+0000 7f4b4742c780 0 period (5c04936b-70ea-4415-b4eb-3aad3dc612e5 does not have zone ee18cd86-4684-4b3d-8de6-0075d3d925f1 configured 2023-05-24T21:00:06.896 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-05-24T21:00:13.636 INFO:teuthology.orchestra.run.smithi157.stderr:Sending period to new master zone 299a3349-fae0-459e-8632-431fc36beb1b 2023-05-24T21:00:13.666 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2023-05-24T21:00:13.667 INFO:teuthology.orchestra.run.smithi157.stdout: "id": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:00:13.667 INFO:teuthology.orchestra.run.smithi157.stdout: "epoch": 3, 2023-05-24T21:00:13.667 INFO:teuthology.orchestra.run.smithi157.stdout: "predecessor_uuid": "1cea3a6e-eb52-480e-95b7-3f6e43317e60", 2023-05-24T21:00:13.667 INFO:teuthology.orchestra.run.smithi157.stdout: "sync_status": [], 2023-05-24T21:00:13.667 INFO:teuthology.orchestra.run.smithi157.stdout: "period_map": { 2023-05-24T21:00:13.667 INFO:teuthology.orchestra.run.smithi157.stdout: "id": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:00:13.668 INFO:teuthology.orchestra.run.smithi157.stdout: "zonegroups": [ 2023-05-24T21:00:13.668 INFO:teuthology.orchestra.run.smithi157.stdout: { 2023-05-24T21:00:13.668 INFO:teuthology.orchestra.run.smithi157.stdout: "id": "a", 2023-05-24T21:00:13.668 INFO:teuthology.orchestra.run.smithi157.stdout: "name": "a", 2023-05-24T21:00:13.668 INFO:teuthology.orchestra.run.smithi157.stdout: "api_name": "", 2023-05-24T21:00:13.668 INFO:teuthology.orchestra.run.smithi157.stdout: "is_master": true, 2023-05-24T21:00:13.668 INFO:teuthology.orchestra.run.smithi157.stdout: "endpoints": [ 2023-05-24T21:00:13.669 INFO:teuthology.orchestra.run.smithi157.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-05-24T21:00:13.669 INFO:teuthology.orchestra.run.smithi157.stdout: ], 2023-05-24T21:00:13.669 INFO:teuthology.orchestra.run.smithi157.stdout: "hostnames": [], 2023-05-24T21:00:13.669 INFO:teuthology.orchestra.run.smithi157.stdout: "hostnames_s3website": [], 2023-05-24T21:00:13.669 INFO:teuthology.orchestra.run.smithi157.stdout: "master_zone": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T21:00:13.669 INFO:teuthology.orchestra.run.smithi157.stdout: "zones": [ 2023-05-24T21:00:13.670 INFO:teuthology.orchestra.run.smithi157.stdout: { 2023-05-24T21:00:13.670 INFO:teuthology.orchestra.run.smithi157.stdout: "id": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T21:00:13.670 INFO:teuthology.orchestra.run.smithi157.stdout: "name": "a1", 2023-05-24T21:00:13.670 INFO:teuthology.orchestra.run.smithi157.stdout: "endpoints": [ 2023-05-24T21:00:13.670 INFO:teuthology.orchestra.run.smithi157.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-05-24T21:00:13.670 INFO:teuthology.orchestra.run.smithi157.stdout: ], 2023-05-24T21:00:13.670 INFO:teuthology.orchestra.run.smithi157.stdout: "log_meta": false, 2023-05-24T21:00:13.670 INFO:teuthology.orchestra.run.smithi157.stdout: "log_data": true, 2023-05-24T21:00:13.671 INFO:teuthology.orchestra.run.smithi157.stdout: "bucket_index_max_shards": 11, 2023-05-24T21:00:13.671 INFO:teuthology.orchestra.run.smithi157.stdout: "read_only": false, 2023-05-24T21:00:13.671 INFO:teuthology.orchestra.run.smithi157.stdout: "tier_type": "", 2023-05-24T21:00:13.671 INFO:teuthology.orchestra.run.smithi157.stdout: "sync_from_all": true, 2023-05-24T21:00:13.671 INFO:teuthology.orchestra.run.smithi157.stdout: "sync_from": [], 2023-05-24T21:00:13.671 INFO:teuthology.orchestra.run.smithi157.stdout: "redirect_zone": "", 2023-05-24T21:00:13.671 INFO:teuthology.orchestra.run.smithi157.stdout: "supported_features": [ 2023-05-24T21:00:13.671 INFO:teuthology.orchestra.run.smithi157.stdout: "resharding" 2023-05-24T21:00:13.672 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2023-05-24T21:00:13.672 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:00:13.672 INFO:teuthology.orchestra.run.smithi157.stdout: { 2023-05-24T21:00:13.672 INFO:teuthology.orchestra.run.smithi157.stdout: "id": "5af1588c-f9b9-4687-a040-ce6e0cc808d9", 2023-05-24T21:00:13.672 INFO:teuthology.orchestra.run.smithi157.stdout: "name": "a2", 2023-05-24T21:00:13.672 INFO:teuthology.orchestra.run.smithi157.stdout: "endpoints": [ 2023-05-24T21:00:13.672 INFO:teuthology.orchestra.run.smithi157.stdout: "http://smithi071.front.sepia.ceph.com:8001" 2023-05-24T21:00:13.673 INFO:teuthology.orchestra.run.smithi157.stdout: ], 2023-05-24T21:00:13.673 INFO:teuthology.orchestra.run.smithi157.stdout: "log_meta": false, 2023-05-24T21:00:13.673 INFO:teuthology.orchestra.run.smithi157.stdout: "log_data": true, 2023-05-24T21:00:13.673 INFO:teuthology.orchestra.run.smithi157.stdout: "bucket_index_max_shards": 11, 2023-05-24T21:00:13.673 INFO:teuthology.orchestra.run.smithi157.stdout: "read_only": false, 2023-05-24T21:00:13.673 INFO:teuthology.orchestra.run.smithi157.stdout: "tier_type": "", 2023-05-24T21:00:13.673 INFO:teuthology.orchestra.run.smithi157.stdout: "sync_from_all": true, 2023-05-24T21:00:13.673 INFO:teuthology.orchestra.run.smithi157.stdout: "sync_from": [], 2023-05-24T21:00:13.674 INFO:teuthology.orchestra.run.smithi157.stdout: "redirect_zone": "", 2023-05-24T21:00:13.674 INFO:teuthology.orchestra.run.smithi157.stdout: "supported_features": [ 2023-05-24T21:00:13.674 INFO:teuthology.orchestra.run.smithi157.stdout: "resharding" 2023-05-24T21:00:13.674 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2023-05-24T21:00:13.674 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:00:13.674 INFO:teuthology.orchestra.run.smithi157.stdout: ], 2023-05-24T21:00:13.674 INFO:teuthology.orchestra.run.smithi157.stdout: "placement_targets": [ 2023-05-24T21:00:13.675 INFO:teuthology.orchestra.run.smithi157.stdout: { 2023-05-24T21:00:13.675 INFO:teuthology.orchestra.run.smithi157.stdout: "name": "default-placement", 2023-05-24T21:00:13.675 INFO:teuthology.orchestra.run.smithi157.stdout: "tags": [], 2023-05-24T21:00:13.675 INFO:teuthology.orchestra.run.smithi157.stdout: "storage_classes": [ 2023-05-24T21:00:13.676 INFO:teuthology.orchestra.run.smithi157.stdout: "STANDARD" 2023-05-24T21:00:13.676 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2023-05-24T21:00:13.676 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:00:13.676 INFO:teuthology.orchestra.run.smithi157.stdout: ], 2023-05-24T21:00:13.676 INFO:teuthology.orchestra.run.smithi157.stdout: "default_placement": "default-placement", 2023-05-24T21:00:13.676 INFO:teuthology.orchestra.run.smithi157.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T21:00:13.676 INFO:teuthology.orchestra.run.smithi157.stdout: "sync_policy": { 2023-05-24T21:00:13.677 INFO:teuthology.orchestra.run.smithi157.stdout: "groups": [] 2023-05-24T21:00:13.677 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:00:13.677 INFO:teuthology.orchestra.run.smithi157.stdout: "enabled_features": [ 2023-05-24T21:00:13.677 INFO:teuthology.orchestra.run.smithi157.stdout: "resharding" 2023-05-24T21:00:13.677 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2023-05-24T21:00:13.677 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:00:13.678 INFO:teuthology.orchestra.run.smithi157.stdout: { 2023-05-24T21:00:13.678 INFO:teuthology.orchestra.run.smithi157.stdout: "id": "b", 2023-05-24T21:00:13.678 INFO:teuthology.orchestra.run.smithi157.stdout: "name": "b", 2023-05-24T21:00:13.678 INFO:teuthology.orchestra.run.smithi157.stdout: "api_name": "", 2023-05-24T21:00:13.678 INFO:teuthology.orchestra.run.smithi157.stdout: "is_master": false, 2023-05-24T21:00:13.678 INFO:teuthology.orchestra.run.smithi157.stdout: "endpoints": [ 2023-05-24T21:00:13.679 INFO:teuthology.orchestra.run.smithi157.stdout: "http://smithi157.front.sepia.ceph.com:8000" 2023-05-24T21:00:13.679 INFO:teuthology.orchestra.run.smithi157.stdout: ], 2023-05-24T21:00:13.679 INFO:teuthology.orchestra.run.smithi157.stdout: "hostnames": [], 2023-05-24T21:00:13.679 INFO:teuthology.orchestra.run.smithi157.stdout: "hostnames_s3website": [], 2023-05-24T21:00:13.679 INFO:teuthology.orchestra.run.smithi157.stdout: "master_zone": "ee18cd86-4684-4b3d-8de6-0075d3d925f1", 2023-05-24T21:00:13.679 INFO:teuthology.orchestra.run.smithi157.stdout: "zones": [ 2023-05-24T21:00:13.679 INFO:teuthology.orchestra.run.smithi157.stdout: { 2023-05-24T21:00:13.680 INFO:teuthology.orchestra.run.smithi157.stdout: "id": "ee18cd86-4684-4b3d-8de6-0075d3d925f1", 2023-05-24T21:00:13.680 INFO:teuthology.orchestra.run.smithi157.stdout: "name": "b1", 2023-05-24T21:00:13.680 INFO:teuthology.orchestra.run.smithi157.stdout: "endpoints": [ 2023-05-24T21:00:13.680 INFO:teuthology.orchestra.run.smithi157.stdout: "http://smithi157.front.sepia.ceph.com:8000" 2023-05-24T21:00:13.680 INFO:teuthology.orchestra.run.smithi157.stdout: ], 2023-05-24T21:00:13.680 INFO:teuthology.orchestra.run.smithi157.stdout: "log_meta": false, 2023-05-24T21:00:13.680 INFO:teuthology.orchestra.run.smithi157.stdout: "log_data": false, 2023-05-24T21:00:13.681 INFO:teuthology.orchestra.run.smithi157.stdout: "bucket_index_max_shards": 11, 2023-05-24T21:00:13.681 INFO:teuthology.orchestra.run.smithi157.stdout: "read_only": false, 2023-05-24T21:00:13.681 INFO:teuthology.orchestra.run.smithi157.stdout: "tier_type": "", 2023-05-24T21:00:13.681 INFO:teuthology.orchestra.run.smithi157.stdout: "sync_from_all": true, 2023-05-24T21:00:13.681 INFO:teuthology.orchestra.run.smithi157.stdout: "sync_from": [], 2023-05-24T21:00:13.681 INFO:teuthology.orchestra.run.smithi157.stdout: "redirect_zone": "", 2023-05-24T21:00:13.681 INFO:teuthology.orchestra.run.smithi157.stdout: "supported_features": [ 2023-05-24T21:00:13.681 INFO:teuthology.orchestra.run.smithi157.stdout: "resharding" 2023-05-24T21:00:13.682 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2023-05-24T21:00:13.682 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:00:13.682 INFO:teuthology.orchestra.run.smithi157.stdout: ], 2023-05-24T21:00:13.682 INFO:teuthology.orchestra.run.smithi157.stdout: "placement_targets": [ 2023-05-24T21:00:13.683 INFO:teuthology.orchestra.run.smithi157.stdout: { 2023-05-24T21:00:13.683 INFO:teuthology.orchestra.run.smithi157.stdout: "name": "default-placement", 2023-05-24T21:00:13.683 INFO:teuthology.orchestra.run.smithi157.stdout: "tags": [], 2023-05-24T21:00:13.683 INFO:teuthology.orchestra.run.smithi157.stdout: "storage_classes": [ 2023-05-24T21:00:13.684 INFO:teuthology.orchestra.run.smithi157.stdout: "STANDARD" 2023-05-24T21:00:13.684 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2023-05-24T21:00:13.684 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:00:13.684 INFO:teuthology.orchestra.run.smithi157.stdout: ], 2023-05-24T21:00:13.684 INFO:teuthology.orchestra.run.smithi157.stdout: "default_placement": "default-placement", 2023-05-24T21:00:13.684 INFO:teuthology.orchestra.run.smithi157.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T21:00:13.684 INFO:teuthology.orchestra.run.smithi157.stdout: "sync_policy": { 2023-05-24T21:00:13.685 INFO:teuthology.orchestra.run.smithi157.stdout: "groups": [] 2023-05-24T21:00:13.685 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:00:13.685 INFO:teuthology.orchestra.run.smithi157.stdout: "enabled_features": [ 2023-05-24T21:00:13.685 INFO:teuthology.orchestra.run.smithi157.stdout: "resharding" 2023-05-24T21:00:13.685 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2023-05-24T21:00:13.685 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:00:13.685 INFO:teuthology.orchestra.run.smithi157.stdout: ], 2023-05-24T21:00:13.686 INFO:teuthology.orchestra.run.smithi157.stdout: "short_zone_ids": [ 2023-05-24T21:00:13.686 INFO:teuthology.orchestra.run.smithi157.stdout: { 2023-05-24T21:00:13.686 INFO:teuthology.orchestra.run.smithi157.stdout: "key": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T21:00:13.686 INFO:teuthology.orchestra.run.smithi157.stdout: "val": 3764661814 2023-05-24T21:00:13.686 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:00:13.686 INFO:teuthology.orchestra.run.smithi157.stdout: { 2023-05-24T21:00:13.686 INFO:teuthology.orchestra.run.smithi157.stdout: "key": "5af1588c-f9b9-4687-a040-ce6e0cc808d9", 2023-05-24T21:00:13.686 INFO:teuthology.orchestra.run.smithi157.stdout: "val": 2266179370 2023-05-24T21:00:13.687 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:00:13.687 INFO:teuthology.orchestra.run.smithi157.stdout: { 2023-05-24T21:00:13.687 INFO:teuthology.orchestra.run.smithi157.stdout: "key": "ee18cd86-4684-4b3d-8de6-0075d3d925f1", 2023-05-24T21:00:13.687 INFO:teuthology.orchestra.run.smithi157.stdout: "val": 489875403 2023-05-24T21:00:13.687 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:00:13.687 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2023-05-24T21:00:13.688 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:00:13.688 INFO:teuthology.orchestra.run.smithi157.stdout: "master_zonegroup": "a", 2023-05-24T21:00:13.688 INFO:teuthology.orchestra.run.smithi157.stdout: "master_zone": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T21:00:13.688 INFO:teuthology.orchestra.run.smithi157.stdout: "period_config": { 2023-05-24T21:00:13.688 INFO:teuthology.orchestra.run.smithi157.stdout: "bucket_quota": { 2023-05-24T21:00:13.688 INFO:teuthology.orchestra.run.smithi157.stdout: "enabled": false, 2023-05-24T21:00:13.688 INFO:teuthology.orchestra.run.smithi157.stdout: "check_on_raw": false, 2023-05-24T21:00:13.689 INFO:teuthology.orchestra.run.smithi157.stdout: "max_size": -1, 2023-05-24T21:00:13.689 INFO:teuthology.orchestra.run.smithi157.stdout: "max_size_kb": 0, 2023-05-24T21:00:13.689 INFO:teuthology.orchestra.run.smithi157.stdout: "max_objects": -1 2023-05-24T21:00:13.689 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:00:13.689 INFO:teuthology.orchestra.run.smithi157.stdout: "user_quota": { 2023-05-24T21:00:13.689 INFO:teuthology.orchestra.run.smithi157.stdout: "enabled": false, 2023-05-24T21:00:13.689 INFO:teuthology.orchestra.run.smithi157.stdout: "check_on_raw": false, 2023-05-24T21:00:13.689 INFO:teuthology.orchestra.run.smithi157.stdout: "max_size": -1, 2023-05-24T21:00:13.690 INFO:teuthology.orchestra.run.smithi157.stdout: "max_size_kb": 0, 2023-05-24T21:00:13.690 INFO:teuthology.orchestra.run.smithi157.stdout: "max_objects": -1 2023-05-24T21:00:13.690 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:00:13.690 INFO:teuthology.orchestra.run.smithi157.stdout: "user_ratelimit": { 2023-05-24T21:00:13.690 INFO:teuthology.orchestra.run.smithi157.stdout: "max_read_ops": 0, 2023-05-24T21:00:13.690 INFO:teuthology.orchestra.run.smithi157.stdout: "max_write_ops": 0, 2023-05-24T21:00:13.690 INFO:teuthology.orchestra.run.smithi157.stdout: "max_read_bytes": 0, 2023-05-24T21:00:13.690 INFO:teuthology.orchestra.run.smithi157.stdout: "max_write_bytes": 0, 2023-05-24T21:00:13.691 INFO:teuthology.orchestra.run.smithi157.stdout: "enabled": false 2023-05-24T21:00:13.691 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:00:13.691 INFO:teuthology.orchestra.run.smithi157.stdout: "bucket_ratelimit": { 2023-05-24T21:00:13.691 INFO:teuthology.orchestra.run.smithi157.stdout: "max_read_ops": 0, 2023-05-24T21:00:13.691 INFO:teuthology.orchestra.run.smithi157.stdout: "max_write_ops": 0, 2023-05-24T21:00:13.691 INFO:teuthology.orchestra.run.smithi157.stdout: "max_read_bytes": 0, 2023-05-24T21:00:13.691 INFO:teuthology.orchestra.run.smithi157.stdout: "max_write_bytes": 0, 2023-05-24T21:00:13.692 INFO:teuthology.orchestra.run.smithi157.stdout: "enabled": false 2023-05-24T21:00:13.692 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:00:13.692 INFO:teuthology.orchestra.run.smithi157.stdout: "anonymous_ratelimit": { 2023-05-24T21:00:13.692 INFO:teuthology.orchestra.run.smithi157.stdout: "max_read_ops": 0, 2023-05-24T21:00:13.692 INFO:teuthology.orchestra.run.smithi157.stdout: "max_write_ops": 0, 2023-05-24T21:00:13.692 INFO:teuthology.orchestra.run.smithi157.stdout: "max_read_bytes": 0, 2023-05-24T21:00:13.692 INFO:teuthology.orchestra.run.smithi157.stdout: "max_write_bytes": 0, 2023-05-24T21:00:13.692 INFO:teuthology.orchestra.run.smithi157.stdout: "enabled": false 2023-05-24T21:00:13.693 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:00:13.693 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:00:13.693 INFO:teuthology.orchestra.run.smithi157.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T21:00:13.693 INFO:teuthology.orchestra.run.smithi157.stdout: "realm_name": "test-realm", 2023-05-24T21:00:13.693 INFO:teuthology.orchestra.run.smithi157.stdout: "realm_epoch": 2 2023-05-24T21:00:13.693 INFO:teuthology.orchestra.run.smithi157.stdout:} 2023-05-24T21:00:13.694 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2023-05-24T21:00:13.694 INFO:teuthology.orchestra.run:waiting for 300 2023-05-24T21:00:13.695 INFO:tasks.rgw.c2.client.0.smithi157.stdout:2023-05-24T21:00:13.688+0000 7f5ce2f7e700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 109917) UID: 0 2023-05-24T21:00:13.695 INFO:tasks.rgw.c2.client.0.smithi157.stdout:2023-05-24T21:00:13.688+0000 7f5ce71e77c0 -1 shutting down 2023-05-24T21:00:15.624 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2023-05-24T21:00:19.796 INFO:tasks.rgw.c2.client.0:Stopped 2023-05-24T21:00:19.796 INFO:tasks.rgw.c2.client.0:Restarting daemon 2023-05-24T21:00:19.797 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2023-05-24T21:00:19.800 INFO:tasks.rgw.c2.client.0:Started 2023-05-24T21:00:19.800 DEBUG:teuthology.orchestra.run.smithi157:> curl http://smithi157.front.sepia.ceph.com:8000 2023-05-24T21:00:19.825 INFO:teuthology.orchestra.run.smithi157.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-05-24T21:00:19.825 INFO:teuthology.orchestra.run.smithi157.stderr: Dload Upload Total Spent Left Speed 2023-05-24T21:00:19.826 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-05-24T21:00:19.827 INFO:teuthology.orchestra.run.smithi157.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi157.front.sepia.ceph.com port 8000: Connection refused 2023-05-24T21:00:20.829 DEBUG:teuthology.orchestra.run.smithi157:> curl http://smithi157.front.sepia.ceph.com:8000 2023-05-24T21:00:20.851 INFO:teuthology.orchestra.run.smithi157.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-05-24T21:00:20.851 INFO:teuthology.orchestra.run.smithi157.stderr: Dload Upload Total Spent Left Speed 2023-05-24T21:00:20.852 INFO:teuthology.orchestra.run.smithi157.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-05-24T21:00:20.853 INFO:teuthology.orchestra.run.smithi157.stdout:anonymous 2023-05-24T21:00:20.853 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2023-05-24T21:00:20.853 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi071.front.sepia.ceph.com:8000', '--access-key', 'Z6KKVUYPP0489K03', '--secret', 'EX5IDzHTe9AVqa8b1PSD8B5Y1CKI3jBg', '--url', 'http://smithi071.front.sepia.ceph.com:8000', '--access-key', 'Z6KKVUYPP0489K03', '--secret', 'EX5IDzHTe9AVqa8b1PSD8B5Y1CKI3jBg', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:00:20.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi071.front.sepia.ceph.com:8000', '--access-key', 'Z6KKVUYPP0489K03', '--secret', 'EX5IDzHTe9AVqa8b1PSD8B5Y1CKI3jBg', '--url', 'http://smithi071.front.sepia.ceph.com:8000', '--access-key', 'Z6KKVUYPP0489K03', '--secret', 'EX5IDzHTe9AVqa8b1PSD8B5Y1CKI3jBg', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:00:20.855 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi071.front.sepia.ceph.com:8000 --access-key Z6KKVUYPP0489K03 --secret EX5IDzHTe9AVqa8b1PSD8B5Y1CKI3jBg --url http://smithi071.front.sepia.ceph.com:8000 --access-key Z6KKVUYPP0489K03 --secret EX5IDzHTe9AVqa8b1PSD8B5Y1CKI3jBg --cluster c2 --debug-rgw 0 --debug-ms 0 2023-05-24T21:00:20.951 INFO:teuthology.orchestra.run.smithi157.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:00:20.951 INFO:teuthology.orchestra.run.smithi157.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:00:20.983 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2023-05-24T21:00:20.984 INFO:teuthology.orchestra.run.smithi157.stdout: "id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T21:00:20.984 INFO:teuthology.orchestra.run.smithi157.stdout: "name": "test-realm", 2023-05-24T21:00:20.984 INFO:teuthology.orchestra.run.smithi157.stdout: "current_period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:00:20.984 INFO:teuthology.orchestra.run.smithi157.stdout: "epoch": 2 2023-05-24T21:00:20.984 INFO:teuthology.orchestra.run.smithi157.stdout:} 2023-05-24T21:00:20.985 INFO:tasks.rgw_multisite:creating zone.. 2023-05-24T21:00:20.985 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--access-key', 'Z6KKVUYPP0489K03', '--secret', 'EX5IDzHTe9AVqa8b1PSD8B5Y1CKI3jBg', '--endpoints', 'http://smithi157.front.sepia.ceph.com:8001', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:00:20.986 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--access-key', 'Z6KKVUYPP0489K03', '--secret', 'EX5IDzHTe9AVqa8b1PSD8B5Y1CKI3jBg', '--endpoints', 'http://smithi157.front.sepia.ceph.com:8001', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:00:20.986 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --access-key Z6KKVUYPP0489K03 --secret EX5IDzHTe9AVqa8b1PSD8B5Y1CKI3jBg --endpoints http://smithi157.front.sepia.ceph.com:8001 --cluster c2 --debug-rgw 0 --debug-ms 0 2023-05-24T21:00:21.039 INFO:teuthology.orchestra.run.smithi157.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:00:21.039 INFO:teuthology.orchestra.run.smithi157.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:00:21.053 INFO:teuthology.orchestra.run.smithi157.stderr:NOTICE: set zone's realm_id=4a7754f9-b4ac-4d00-9827-3910f2de91d2 2023-05-24T21:00:21.067 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2023-05-24T21:00:21.067 INFO:teuthology.orchestra.run.smithi157.stdout: "id": "7b8c3b69-c8ed-4621-b2ef-b89d3bbb7973", 2023-05-24T21:00:21.068 INFO:teuthology.orchestra.run.smithi157.stdout: "name": "b2", 2023-05-24T21:00:21.068 INFO:teuthology.orchestra.run.smithi157.stdout: "domain_root": "b2.rgw.meta:root", 2023-05-24T21:00:21.068 INFO:teuthology.orchestra.run.smithi157.stdout: "control_pool": "b2.rgw.control", 2023-05-24T21:00:21.068 INFO:teuthology.orchestra.run.smithi157.stdout: "gc_pool": "b2.rgw.log:gc", 2023-05-24T21:00:21.068 INFO:teuthology.orchestra.run.smithi157.stdout: "lc_pool": "b2.rgw.log:lc", 2023-05-24T21:00:21.068 INFO:teuthology.orchestra.run.smithi157.stdout: "log_pool": "b2.rgw.log", 2023-05-24T21:00:21.069 INFO:teuthology.orchestra.run.smithi157.stdout: "intent_log_pool": "b2.rgw.log:intent", 2023-05-24T21:00:21.069 INFO:teuthology.orchestra.run.smithi157.stdout: "usage_log_pool": "b2.rgw.log:usage", 2023-05-24T21:00:21.069 INFO:teuthology.orchestra.run.smithi157.stdout: "roles_pool": "b2.rgw.meta:roles", 2023-05-24T21:00:21.069 INFO:teuthology.orchestra.run.smithi157.stdout: "reshard_pool": "b2.rgw.log:reshard", 2023-05-24T21:00:21.069 INFO:teuthology.orchestra.run.smithi157.stdout: "user_keys_pool": "b2.rgw.meta:users.keys", 2023-05-24T21:00:21.069 INFO:teuthology.orchestra.run.smithi157.stdout: "user_email_pool": "b2.rgw.meta:users.email", 2023-05-24T21:00:21.069 INFO:teuthology.orchestra.run.smithi157.stdout: "user_swift_pool": "b2.rgw.meta:users.swift", 2023-05-24T21:00:21.070 INFO:teuthology.orchestra.run.smithi157.stdout: "user_uid_pool": "b2.rgw.meta:users.uid", 2023-05-24T21:00:21.070 INFO:teuthology.orchestra.run.smithi157.stdout: "otp_pool": "b2.rgw.otp", 2023-05-24T21:00:21.070 INFO:teuthology.orchestra.run.smithi157.stdout: "system_key": { 2023-05-24T21:00:21.070 INFO:teuthology.orchestra.run.smithi157.stdout: "access_key": "Z6KKVUYPP0489K03", 2023-05-24T21:00:21.070 INFO:teuthology.orchestra.run.smithi157.stdout: "secret_key": "EX5IDzHTe9AVqa8b1PSD8B5Y1CKI3jBg" 2023-05-24T21:00:21.070 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:00:21.070 INFO:teuthology.orchestra.run.smithi157.stdout: "placement_pools": [ 2023-05-24T21:00:21.071 INFO:teuthology.orchestra.run.smithi157.stdout: { 2023-05-24T21:00:21.071 INFO:teuthology.orchestra.run.smithi157.stdout: "key": "default-placement", 2023-05-24T21:00:21.071 INFO:teuthology.orchestra.run.smithi157.stdout: "val": { 2023-05-24T21:00:21.071 INFO:teuthology.orchestra.run.smithi157.stdout: "index_pool": "b2.rgw.buckets.index", 2023-05-24T21:00:21.071 INFO:teuthology.orchestra.run.smithi157.stdout: "storage_classes": { 2023-05-24T21:00:21.071 INFO:teuthology.orchestra.run.smithi157.stdout: "STANDARD": { 2023-05-24T21:00:21.071 INFO:teuthology.orchestra.run.smithi157.stdout: "data_pool": "b2.rgw.buckets.data" 2023-05-24T21:00:21.072 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:00:21.072 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:00:21.072 INFO:teuthology.orchestra.run.smithi157.stdout: "data_extra_pool": "b2.rgw.buckets.non-ec", 2023-05-24T21:00:21.072 INFO:teuthology.orchestra.run.smithi157.stdout: "index_type": 0, 2023-05-24T21:00:21.072 INFO:teuthology.orchestra.run.smithi157.stdout: "inline_data": true 2023-05-24T21:00:21.072 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:00:21.072 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:00:21.073 INFO:teuthology.orchestra.run.smithi157.stdout: ], 2023-05-24T21:00:21.073 INFO:teuthology.orchestra.run.smithi157.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T21:00:21.073 INFO:teuthology.orchestra.run.smithi157.stdout: "notif_pool": "b2.rgw.log:notif" 2023-05-24T21:00:21.073 INFO:teuthology.orchestra.run.smithi157.stdout:} 2023-05-24T21:00:21.074 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph osd pool create b2.rgw.buckets.data 64 64 --cluster c2 2023-05-24T21:00:21.330 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2023-05-24T21:00:21.581 INFO:teuthology.orchestra.run.smithi157.stderr:pool 'b2.rgw.buckets.data' created 2023-05-24T21:00:21.598 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph osd pool application enable b2.rgw.buckets.data rgw --cluster c2 2023-05-24T21:00:22.591 INFO:teuthology.orchestra.run.smithi157.stderr:enabled application 'rgw' on pool 'b2.rgw.buckets.data' 2023-05-24T21:00:22.612 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:00:22.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:00:22.613 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2023-05-24T21:00:22.670 INFO:teuthology.orchestra.run.smithi157.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:00:22.670 INFO:teuthology.orchestra.run.smithi157.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:00:22.689 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2023-05-24T21:00:22.689 INFO:teuthology.orchestra.run.smithi157.stdout: "id": "7b8c3b69-c8ed-4621-b2ef-b89d3bbb7973", 2023-05-24T21:00:22.689 INFO:teuthology.orchestra.run.smithi157.stdout: "name": "b2", 2023-05-24T21:00:22.690 INFO:teuthology.orchestra.run.smithi157.stdout: "domain_root": "b2.rgw.meta:root", 2023-05-24T21:00:22.690 INFO:teuthology.orchestra.run.smithi157.stdout: "control_pool": "b2.rgw.control", 2023-05-24T21:00:22.690 INFO:teuthology.orchestra.run.smithi157.stdout: "gc_pool": "b2.rgw.log:gc", 2023-05-24T21:00:22.690 INFO:teuthology.orchestra.run.smithi157.stdout: "lc_pool": "b2.rgw.log:lc", 2023-05-24T21:00:22.690 INFO:teuthology.orchestra.run.smithi157.stdout: "log_pool": "b2.rgw.log", 2023-05-24T21:00:22.690 INFO:teuthology.orchestra.run.smithi157.stdout: "intent_log_pool": "b2.rgw.log:intent", 2023-05-24T21:00:22.690 INFO:teuthology.orchestra.run.smithi157.stdout: "usage_log_pool": "b2.rgw.log:usage", 2023-05-24T21:00:22.691 INFO:teuthology.orchestra.run.smithi157.stdout: "roles_pool": "b2.rgw.meta:roles", 2023-05-24T21:00:22.691 INFO:teuthology.orchestra.run.smithi157.stdout: "reshard_pool": "b2.rgw.log:reshard", 2023-05-24T21:00:22.691 INFO:teuthology.orchestra.run.smithi157.stdout: "user_keys_pool": "b2.rgw.meta:users.keys", 2023-05-24T21:00:22.691 INFO:teuthology.orchestra.run.smithi157.stdout: "user_email_pool": "b2.rgw.meta:users.email", 2023-05-24T21:00:22.691 INFO:teuthology.orchestra.run.smithi157.stdout: "user_swift_pool": "b2.rgw.meta:users.swift", 2023-05-24T21:00:22.691 INFO:teuthology.orchestra.run.smithi157.stdout: "user_uid_pool": "b2.rgw.meta:users.uid", 2023-05-24T21:00:22.691 INFO:teuthology.orchestra.run.smithi157.stdout: "otp_pool": "b2.rgw.otp", 2023-05-24T21:00:22.692 INFO:teuthology.orchestra.run.smithi157.stdout: "system_key": { 2023-05-24T21:00:22.692 INFO:teuthology.orchestra.run.smithi157.stdout: "access_key": "Z6KKVUYPP0489K03", 2023-05-24T21:00:22.692 INFO:teuthology.orchestra.run.smithi157.stdout: "secret_key": "EX5IDzHTe9AVqa8b1PSD8B5Y1CKI3jBg" 2023-05-24T21:00:22.692 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:00:22.692 INFO:teuthology.orchestra.run.smithi157.stdout: "placement_pools": [ 2023-05-24T21:00:22.692 INFO:teuthology.orchestra.run.smithi157.stdout: { 2023-05-24T21:00:22.692 INFO:teuthology.orchestra.run.smithi157.stdout: "key": "default-placement", 2023-05-24T21:00:22.692 INFO:teuthology.orchestra.run.smithi157.stdout: "val": { 2023-05-24T21:00:22.693 INFO:teuthology.orchestra.run.smithi157.stdout: "index_pool": "b2.rgw.buckets.index", 2023-05-24T21:00:22.693 INFO:teuthology.orchestra.run.smithi157.stdout: "storage_classes": { 2023-05-24T21:00:22.693 INFO:teuthology.orchestra.run.smithi157.stdout: "STANDARD": { 2023-05-24T21:00:22.693 INFO:teuthology.orchestra.run.smithi157.stdout: "data_pool": "b2.rgw.buckets.data", 2023-05-24T21:00:22.693 INFO:teuthology.orchestra.run.smithi157.stdout: "compression_type": "random" 2023-05-24T21:00:22.693 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:00:22.693 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:00:22.694 INFO:teuthology.orchestra.run.smithi157.stdout: "data_extra_pool": "b2.rgw.buckets.non-ec", 2023-05-24T21:00:22.694 INFO:teuthology.orchestra.run.smithi157.stdout: "index_type": 0, 2023-05-24T21:00:22.694 INFO:teuthology.orchestra.run.smithi157.stdout: "inline_data": true 2023-05-24T21:00:22.694 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:00:22.694 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:00:22.694 INFO:teuthology.orchestra.run.smithi157.stdout: ], 2023-05-24T21:00:22.694 INFO:teuthology.orchestra.run.smithi157.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T21:00:22.694 INFO:teuthology.orchestra.run.smithi157.stdout: "notif_pool": "b2.rgw.log:notif" 2023-05-24T21:00:22.695 INFO:teuthology.orchestra.run.smithi157.stdout:} 2023-05-24T21:00:22.696 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:00:22.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:00:22.697 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2023-05-24T21:00:22.745 INFO:teuthology.orchestra.run.smithi157.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:00:22.746 INFO:teuthology.orchestra.run.smithi157.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:00:22.757 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T21:00:22.751+0000 7fdfa5a4e780 0 period (5c04936b-70ea-4415-b4eb-3aad3dc612e5 does not have zone 7b8c3b69-c8ed-4621-b2ef-b89d3bbb7973 configured 2023-05-24T21:00:29.643 INFO:teuthology.orchestra.run.smithi157.stderr:Sending period to new master zone 299a3349-fae0-459e-8632-431fc36beb1b 2023-05-24T21:00:29.674 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2023-05-24T21:00:29.674 INFO:teuthology.orchestra.run.smithi157.stdout: "id": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:00:29.675 INFO:teuthology.orchestra.run.smithi157.stdout: "epoch": 4, 2023-05-24T21:00:29.675 INFO:teuthology.orchestra.run.smithi157.stdout: "predecessor_uuid": "1cea3a6e-eb52-480e-95b7-3f6e43317e60", 2023-05-24T21:00:29.675 INFO:teuthology.orchestra.run.smithi157.stdout: "sync_status": [], 2023-05-24T21:00:29.675 INFO:teuthology.orchestra.run.smithi157.stdout: "period_map": { 2023-05-24T21:00:29.675 INFO:teuthology.orchestra.run.smithi157.stdout: "id": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:00:29.675 INFO:teuthology.orchestra.run.smithi157.stdout: "zonegroups": [ 2023-05-24T21:00:29.675 INFO:teuthology.orchestra.run.smithi157.stdout: { 2023-05-24T21:00:29.676 INFO:teuthology.orchestra.run.smithi157.stdout: "id": "a", 2023-05-24T21:00:29.676 INFO:teuthology.orchestra.run.smithi157.stdout: "name": "a", 2023-05-24T21:00:29.676 INFO:teuthology.orchestra.run.smithi157.stdout: "api_name": "", 2023-05-24T21:00:29.676 INFO:teuthology.orchestra.run.smithi157.stdout: "is_master": true, 2023-05-24T21:00:29.676 INFO:teuthology.orchestra.run.smithi157.stdout: "endpoints": [ 2023-05-24T21:00:29.676 INFO:teuthology.orchestra.run.smithi157.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-05-24T21:00:29.676 INFO:teuthology.orchestra.run.smithi157.stdout: ], 2023-05-24T21:00:29.677 INFO:teuthology.orchestra.run.smithi157.stdout: "hostnames": [], 2023-05-24T21:00:29.677 INFO:teuthology.orchestra.run.smithi157.stdout: "hostnames_s3website": [], 2023-05-24T21:00:29.677 INFO:teuthology.orchestra.run.smithi157.stdout: "master_zone": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T21:00:29.677 INFO:teuthology.orchestra.run.smithi157.stdout: "zones": [ 2023-05-24T21:00:29.677 INFO:teuthology.orchestra.run.smithi157.stdout: { 2023-05-24T21:00:29.677 INFO:teuthology.orchestra.run.smithi157.stdout: "id": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T21:00:29.677 INFO:teuthology.orchestra.run.smithi157.stdout: "name": "a1", 2023-05-24T21:00:29.678 INFO:teuthology.orchestra.run.smithi157.stdout: "endpoints": [ 2023-05-24T21:00:29.678 INFO:teuthology.orchestra.run.smithi157.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-05-24T21:00:29.678 INFO:teuthology.orchestra.run.smithi157.stdout: ], 2023-05-24T21:00:29.678 INFO:teuthology.orchestra.run.smithi157.stdout: "log_meta": false, 2023-05-24T21:00:29.678 INFO:teuthology.orchestra.run.smithi157.stdout: "log_data": true, 2023-05-24T21:00:29.678 INFO:teuthology.orchestra.run.smithi157.stdout: "bucket_index_max_shards": 11, 2023-05-24T21:00:29.678 INFO:teuthology.orchestra.run.smithi157.stdout: "read_only": false, 2023-05-24T21:00:29.678 INFO:teuthology.orchestra.run.smithi157.stdout: "tier_type": "", 2023-05-24T21:00:29.679 INFO:teuthology.orchestra.run.smithi157.stdout: "sync_from_all": true, 2023-05-24T21:00:29.679 INFO:teuthology.orchestra.run.smithi157.stdout: "sync_from": [], 2023-05-24T21:00:29.679 INFO:teuthology.orchestra.run.smithi157.stdout: "redirect_zone": "", 2023-05-24T21:00:29.679 INFO:teuthology.orchestra.run.smithi157.stdout: "supported_features": [ 2023-05-24T21:00:29.679 INFO:teuthology.orchestra.run.smithi157.stdout: "resharding" 2023-05-24T21:00:29.679 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2023-05-24T21:00:29.679 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:00:29.680 INFO:teuthology.orchestra.run.smithi157.stdout: { 2023-05-24T21:00:29.680 INFO:teuthology.orchestra.run.smithi157.stdout: "id": "5af1588c-f9b9-4687-a040-ce6e0cc808d9", 2023-05-24T21:00:29.680 INFO:teuthology.orchestra.run.smithi157.stdout: "name": "a2", 2023-05-24T21:00:29.680 INFO:teuthology.orchestra.run.smithi157.stdout: "endpoints": [ 2023-05-24T21:00:29.680 INFO:teuthology.orchestra.run.smithi157.stdout: "http://smithi071.front.sepia.ceph.com:8001" 2023-05-24T21:00:29.680 INFO:teuthology.orchestra.run.smithi157.stdout: ], 2023-05-24T21:00:29.680 INFO:teuthology.orchestra.run.smithi157.stdout: "log_meta": false, 2023-05-24T21:00:29.680 INFO:teuthology.orchestra.run.smithi157.stdout: "log_data": true, 2023-05-24T21:00:29.681 INFO:teuthology.orchestra.run.smithi157.stdout: "bucket_index_max_shards": 11, 2023-05-24T21:00:29.681 INFO:teuthology.orchestra.run.smithi157.stdout: "read_only": false, 2023-05-24T21:00:29.681 INFO:teuthology.orchestra.run.smithi157.stdout: "tier_type": "", 2023-05-24T21:00:29.681 INFO:teuthology.orchestra.run.smithi157.stdout: "sync_from_all": true, 2023-05-24T21:00:29.681 INFO:teuthology.orchestra.run.smithi157.stdout: "sync_from": [], 2023-05-24T21:00:29.681 INFO:teuthology.orchestra.run.smithi157.stdout: "redirect_zone": "", 2023-05-24T21:00:29.681 INFO:teuthology.orchestra.run.smithi157.stdout: "supported_features": [ 2023-05-24T21:00:29.682 INFO:teuthology.orchestra.run.smithi157.stdout: "resharding" 2023-05-24T21:00:29.682 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2023-05-24T21:00:29.682 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:00:29.682 INFO:teuthology.orchestra.run.smithi157.stdout: ], 2023-05-24T21:00:29.682 INFO:teuthology.orchestra.run.smithi157.stdout: "placement_targets": [ 2023-05-24T21:00:29.682 INFO:teuthology.orchestra.run.smithi157.stdout: { 2023-05-24T21:00:29.682 INFO:teuthology.orchestra.run.smithi157.stdout: "name": "default-placement", 2023-05-24T21:00:29.682 INFO:teuthology.orchestra.run.smithi157.stdout: "tags": [], 2023-05-24T21:00:29.683 INFO:teuthology.orchestra.run.smithi157.stdout: "storage_classes": [ 2023-05-24T21:00:29.683 INFO:teuthology.orchestra.run.smithi157.stdout: "STANDARD" 2023-05-24T21:00:29.683 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2023-05-24T21:00:29.683 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:00:29.683 INFO:teuthology.orchestra.run.smithi157.stdout: ], 2023-05-24T21:00:29.683 INFO:teuthology.orchestra.run.smithi157.stdout: "default_placement": "default-placement", 2023-05-24T21:00:29.683 INFO:teuthology.orchestra.run.smithi157.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T21:00:29.683 INFO:teuthology.orchestra.run.smithi157.stdout: "sync_policy": { 2023-05-24T21:00:29.684 INFO:teuthology.orchestra.run.smithi157.stdout: "groups": [] 2023-05-24T21:00:29.684 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:00:29.684 INFO:teuthology.orchestra.run.smithi157.stdout: "enabled_features": [ 2023-05-24T21:00:29.684 INFO:teuthology.orchestra.run.smithi157.stdout: "resharding" 2023-05-24T21:00:29.684 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2023-05-24T21:00:29.684 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:00:29.684 INFO:teuthology.orchestra.run.smithi157.stdout: { 2023-05-24T21:00:29.684 INFO:teuthology.orchestra.run.smithi157.stdout: "id": "b", 2023-05-24T21:00:29.685 INFO:teuthology.orchestra.run.smithi157.stdout: "name": "b", 2023-05-24T21:00:29.685 INFO:teuthology.orchestra.run.smithi157.stdout: "api_name": "", 2023-05-24T21:00:29.685 INFO:teuthology.orchestra.run.smithi157.stdout: "is_master": false, 2023-05-24T21:00:29.685 INFO:teuthology.orchestra.run.smithi157.stdout: "endpoints": [ 2023-05-24T21:00:29.685 INFO:teuthology.orchestra.run.smithi157.stdout: "http://smithi157.front.sepia.ceph.com:8000" 2023-05-24T21:00:29.685 INFO:teuthology.orchestra.run.smithi157.stdout: ], 2023-05-24T21:00:29.685 INFO:teuthology.orchestra.run.smithi157.stdout: "hostnames": [], 2023-05-24T21:00:29.686 INFO:teuthology.orchestra.run.smithi157.stdout: "hostnames_s3website": [], 2023-05-24T21:00:29.686 INFO:teuthology.orchestra.run.smithi157.stdout: "master_zone": "ee18cd86-4684-4b3d-8de6-0075d3d925f1", 2023-05-24T21:00:29.686 INFO:teuthology.orchestra.run.smithi157.stdout: "zones": [ 2023-05-24T21:00:29.686 INFO:teuthology.orchestra.run.smithi157.stdout: { 2023-05-24T21:00:29.686 INFO:teuthology.orchestra.run.smithi157.stdout: "id": "7b8c3b69-c8ed-4621-b2ef-b89d3bbb7973", 2023-05-24T21:00:29.686 INFO:teuthology.orchestra.run.smithi157.stdout: "name": "b2", 2023-05-24T21:00:29.686 INFO:teuthology.orchestra.run.smithi157.stdout: "endpoints": [ 2023-05-24T21:00:29.686 INFO:teuthology.orchestra.run.smithi157.stdout: "http://smithi157.front.sepia.ceph.com:8001" 2023-05-24T21:00:29.687 INFO:teuthology.orchestra.run.smithi157.stdout: ], 2023-05-24T21:00:29.687 INFO:teuthology.orchestra.run.smithi157.stdout: "log_meta": false, 2023-05-24T21:00:29.687 INFO:teuthology.orchestra.run.smithi157.stdout: "log_data": true, 2023-05-24T21:00:29.687 INFO:teuthology.orchestra.run.smithi157.stdout: "bucket_index_max_shards": 11, 2023-05-24T21:00:29.687 INFO:teuthology.orchestra.run.smithi157.stdout: "read_only": false, 2023-05-24T21:00:29.687 INFO:teuthology.orchestra.run.smithi157.stdout: "tier_type": "", 2023-05-24T21:00:29.687 INFO:teuthology.orchestra.run.smithi157.stdout: "sync_from_all": true, 2023-05-24T21:00:29.687 INFO:teuthology.orchestra.run.smithi157.stdout: "sync_from": [], 2023-05-24T21:00:29.688 INFO:teuthology.orchestra.run.smithi157.stdout: "redirect_zone": "", 2023-05-24T21:00:29.688 INFO:teuthology.orchestra.run.smithi157.stdout: "supported_features": [ 2023-05-24T21:00:29.688 INFO:teuthology.orchestra.run.smithi157.stdout: "resharding" 2023-05-24T21:00:29.688 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2023-05-24T21:00:29.688 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:00:29.688 INFO:teuthology.orchestra.run.smithi157.stdout: { 2023-05-24T21:00:29.689 INFO:teuthology.orchestra.run.smithi157.stdout: "id": "ee18cd86-4684-4b3d-8de6-0075d3d925f1", 2023-05-24T21:00:29.689 INFO:teuthology.orchestra.run.smithi157.stdout: "name": "b1", 2023-05-24T21:00:29.690 INFO:teuthology.orchestra.run.smithi157.stdout: "endpoints": [ 2023-05-24T21:00:29.690 INFO:teuthology.orchestra.run.smithi157.stdout: "http://smithi157.front.sepia.ceph.com:8000" 2023-05-24T21:00:29.690 INFO:teuthology.orchestra.run.smithi157.stdout: ], 2023-05-24T21:00:29.690 INFO:teuthology.orchestra.run.smithi157.stdout: "log_meta": false, 2023-05-24T21:00:29.690 INFO:teuthology.orchestra.run.smithi157.stdout: "log_data": true, 2023-05-24T21:00:29.690 INFO:teuthology.orchestra.run.smithi157.stdout: "bucket_index_max_shards": 11, 2023-05-24T21:00:29.690 INFO:teuthology.orchestra.run.smithi157.stdout: "read_only": false, 2023-05-24T21:00:29.691 INFO:teuthology.orchestra.run.smithi157.stdout: "tier_type": "", 2023-05-24T21:00:29.691 INFO:teuthology.orchestra.run.smithi157.stdout: "sync_from_all": true, 2023-05-24T21:00:29.691 INFO:teuthology.orchestra.run.smithi157.stdout: "sync_from": [], 2023-05-24T21:00:29.691 INFO:teuthology.orchestra.run.smithi157.stdout: "redirect_zone": "", 2023-05-24T21:00:29.691 INFO:teuthology.orchestra.run.smithi157.stdout: "supported_features": [ 2023-05-24T21:00:29.691 INFO:teuthology.orchestra.run.smithi157.stdout: "resharding" 2023-05-24T21:00:29.691 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2023-05-24T21:00:29.692 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:00:29.692 INFO:teuthology.orchestra.run.smithi157.stdout: ], 2023-05-24T21:00:29.692 INFO:teuthology.orchestra.run.smithi157.stdout: "placement_targets": [ 2023-05-24T21:00:29.692 INFO:teuthology.orchestra.run.smithi157.stdout: { 2023-05-24T21:00:29.692 INFO:teuthology.orchestra.run.smithi157.stdout: "name": "default-placement", 2023-05-24T21:00:29.692 INFO:teuthology.orchestra.run.smithi157.stdout: "tags": [], 2023-05-24T21:00:29.692 INFO:teuthology.orchestra.run.smithi157.stdout: "storage_classes": [ 2023-05-24T21:00:29.692 INFO:teuthology.orchestra.run.smithi157.stdout: "STANDARD" 2023-05-24T21:00:29.693 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2023-05-24T21:00:29.693 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:00:29.693 INFO:teuthology.orchestra.run.smithi157.stdout: ], 2023-05-24T21:00:29.693 INFO:teuthology.orchestra.run.smithi157.stdout: "default_placement": "default-placement", 2023-05-24T21:00:29.693 INFO:teuthology.orchestra.run.smithi157.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T21:00:29.693 INFO:teuthology.orchestra.run.smithi157.stdout: "sync_policy": { 2023-05-24T21:00:29.693 INFO:teuthology.orchestra.run.smithi157.stdout: "groups": [] 2023-05-24T21:00:29.694 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:00:29.694 INFO:teuthology.orchestra.run.smithi157.stdout: "enabled_features": [ 2023-05-24T21:00:29.694 INFO:teuthology.orchestra.run.smithi157.stdout: "resharding" 2023-05-24T21:00:29.694 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2023-05-24T21:00:29.694 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:00:29.694 INFO:teuthology.orchestra.run.smithi157.stdout: ], 2023-05-24T21:00:29.694 INFO:teuthology.orchestra.run.smithi157.stdout: "short_zone_ids": [ 2023-05-24T21:00:29.694 INFO:teuthology.orchestra.run.smithi157.stdout: { 2023-05-24T21:00:29.695 INFO:teuthology.orchestra.run.smithi157.stdout: "key": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T21:00:29.695 INFO:teuthology.orchestra.run.smithi157.stdout: "val": 3764661814 2023-05-24T21:00:29.695 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:00:29.695 INFO:teuthology.orchestra.run.smithi157.stdout: { 2023-05-24T21:00:29.695 INFO:teuthology.orchestra.run.smithi157.stdout: "key": "5af1588c-f9b9-4687-a040-ce6e0cc808d9", 2023-05-24T21:00:29.695 INFO:teuthology.orchestra.run.smithi157.stdout: "val": 2266179370 2023-05-24T21:00:29.695 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:00:29.695 INFO:teuthology.orchestra.run.smithi157.stdout: { 2023-05-24T21:00:29.696 INFO:teuthology.orchestra.run.smithi157.stdout: "key": "7b8c3b69-c8ed-4621-b2ef-b89d3bbb7973", 2023-05-24T21:00:29.696 INFO:teuthology.orchestra.run.smithi157.stdout: "val": 1540450555 2023-05-24T21:00:29.696 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:00:29.696 INFO:teuthology.orchestra.run.smithi157.stdout: { 2023-05-24T21:00:29.696 INFO:teuthology.orchestra.run.smithi157.stdout: "key": "ee18cd86-4684-4b3d-8de6-0075d3d925f1", 2023-05-24T21:00:29.696 INFO:teuthology.orchestra.run.smithi157.stdout: "val": 489875403 2023-05-24T21:00:29.696 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:00:29.697 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2023-05-24T21:00:29.697 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:00:29.697 INFO:teuthology.orchestra.run.smithi157.stdout: "master_zonegroup": "a", 2023-05-24T21:00:29.697 INFO:teuthology.orchestra.run.smithi157.stdout: "master_zone": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T21:00:29.697 INFO:teuthology.orchestra.run.smithi157.stdout: "period_config": { 2023-05-24T21:00:29.697 INFO:teuthology.orchestra.run.smithi157.stdout: "bucket_quota": { 2023-05-24T21:00:29.697 INFO:teuthology.orchestra.run.smithi157.stdout: "enabled": false, 2023-05-24T21:00:29.697 INFO:teuthology.orchestra.run.smithi157.stdout: "check_on_raw": false, 2023-05-24T21:00:29.698 INFO:teuthology.orchestra.run.smithi157.stdout: "max_size": -1, 2023-05-24T21:00:29.698 INFO:teuthology.orchestra.run.smithi157.stdout: "max_size_kb": 0, 2023-05-24T21:00:29.698 INFO:teuthology.orchestra.run.smithi157.stdout: "max_objects": -1 2023-05-24T21:00:29.698 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:00:29.698 INFO:teuthology.orchestra.run.smithi157.stdout: "user_quota": { 2023-05-24T21:00:29.698 INFO:teuthology.orchestra.run.smithi157.stdout: "enabled": false, 2023-05-24T21:00:29.698 INFO:teuthology.orchestra.run.smithi157.stdout: "check_on_raw": false, 2023-05-24T21:00:29.698 INFO:teuthology.orchestra.run.smithi157.stdout: "max_size": -1, 2023-05-24T21:00:29.699 INFO:teuthology.orchestra.run.smithi157.stdout: "max_size_kb": 0, 2023-05-24T21:00:29.699 INFO:teuthology.orchestra.run.smithi157.stdout: "max_objects": -1 2023-05-24T21:00:29.699 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:00:29.699 INFO:teuthology.orchestra.run.smithi157.stdout: "user_ratelimit": { 2023-05-24T21:00:29.699 INFO:teuthology.orchestra.run.smithi157.stdout: "max_read_ops": 0, 2023-05-24T21:00:29.699 INFO:teuthology.orchestra.run.smithi157.stdout: "max_write_ops": 0, 2023-05-24T21:00:29.699 INFO:teuthology.orchestra.run.smithi157.stdout: "max_read_bytes": 0, 2023-05-24T21:00:29.700 INFO:teuthology.orchestra.run.smithi157.stdout: "max_write_bytes": 0, 2023-05-24T21:00:29.700 INFO:teuthology.orchestra.run.smithi157.stdout: "enabled": false 2023-05-24T21:00:29.700 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:00:29.700 INFO:teuthology.orchestra.run.smithi157.stdout: "bucket_ratelimit": { 2023-05-24T21:00:29.700 INFO:teuthology.orchestra.run.smithi157.stdout: "max_read_ops": 0, 2023-05-24T21:00:29.700 INFO:teuthology.orchestra.run.smithi157.stdout: "max_write_ops": 0, 2023-05-24T21:00:29.700 INFO:teuthology.orchestra.run.smithi157.stdout: "max_read_bytes": 0, 2023-05-24T21:00:29.700 INFO:teuthology.orchestra.run.smithi157.stdout: "max_write_bytes": 0, 2023-05-24T21:00:29.701 INFO:teuthology.orchestra.run.smithi157.stdout: "enabled": false 2023-05-24T21:00:29.701 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:00:29.701 INFO:teuthology.orchestra.run.smithi157.stdout: "anonymous_ratelimit": { 2023-05-24T21:00:29.701 INFO:teuthology.orchestra.run.smithi157.stdout: "max_read_ops": 0, 2023-05-24T21:00:29.701 INFO:teuthology.orchestra.run.smithi157.stdout: "max_write_ops": 0, 2023-05-24T21:00:29.701 INFO:teuthology.orchestra.run.smithi157.stdout: "max_read_bytes": 0, 2023-05-24T21:00:29.701 INFO:teuthology.orchestra.run.smithi157.stdout: "max_write_bytes": 0, 2023-05-24T21:00:29.701 INFO:teuthology.orchestra.run.smithi157.stdout: "enabled": false 2023-05-24T21:00:29.702 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:00:29.702 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:00:29.702 INFO:teuthology.orchestra.run.smithi157.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T21:00:29.702 INFO:teuthology.orchestra.run.smithi157.stdout: "realm_name": "test-realm", 2023-05-24T21:00:29.702 INFO:teuthology.orchestra.run.smithi157.stdout: "realm_epoch": 2 2023-05-24T21:00:29.702 INFO:teuthology.orchestra.run.smithi157.stdout:} 2023-05-24T21:00:29.703 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2023-05-24T21:00:29.703 INFO:teuthology.orchestra.run:waiting for 300 2023-05-24T21:00:29.704 INFO:tasks.rgw.c2.client.1.smithi157.stdout:2023-05-24T21:00:29.698+0000 7f498e5f2700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 109920) UID: 0 2023-05-24T21:00:29.704 INFO:tasks.rgw.c2.client.1.smithi157.stdout:2023-05-24T21:00:29.698+0000 7f499285b7c0 -1 shutting down 2023-05-24T21:00:32.640 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2023-05-24T21:00:35.805 INFO:tasks.rgw.c2.client.1:Stopped 2023-05-24T21:00:35.805 INFO:tasks.rgw.c2.client.1:Restarting daemon 2023-05-24T21:00:35.806 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2023-05-24T21:00:35.808 INFO:tasks.rgw.c2.client.1:Started 2023-05-24T21:00:35.809 DEBUG:teuthology.orchestra.run.smithi157:> curl http://smithi157.front.sepia.ceph.com:8001 2023-05-24T21:00:35.834 INFO:teuthology.orchestra.run.smithi157.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-05-24T21:00:35.834 INFO:teuthology.orchestra.run.smithi157.stderr: Dload Upload Total Spent Left Speed 2023-05-24T21:00:35.835 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-05-24T21:00:35.836 INFO:teuthology.orchestra.run.smithi157.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi157.front.sepia.ceph.com port 8001: Connection refused 2023-05-24T21:00:36.836 DEBUG:teuthology.orchestra.run.smithi157:> curl http://smithi157.front.sepia.ceph.com:8001 2023-05-24T21:00:36.860 INFO:teuthology.orchestra.run.smithi157.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-05-24T21:00:36.861 INFO:teuthology.orchestra.run.smithi157.stderr: Dload Upload Total Spent Left Speed 2023-05-24T21:00:36.861 INFO:teuthology.orchestra.run.smithi157.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-05-24T21:00:36.862 INFO:teuthology.orchestra.run.smithi157.stdout:anonymous 2023-05-24T21:00:36.863 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2023-05-24T21:00:36.863 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2023-05-24T21:00:36.888 INFO:tasks.rgw_multisite_tests:importing tests from /home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/../src/test/rgw/rgw_multi 2023-05-24T21:00:36.994 INFO:tasks.rgw_multisite_tests:creating test user.. 2023-05-24T21:00:36.994 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:00:36.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:00:36.995 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --caps 'roles=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-24T21:00:37.059 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:00:37.060 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:00:37.108 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:00:37.108 INFO:teuthology.orchestra.run.smithi071.stdout: "user_id": "rgw-multisite-test-user", 2023-05-24T21:00:37.108 INFO:teuthology.orchestra.run.smithi071.stdout: "display_name": "Multisite Test User", 2023-05-24T21:00:37.109 INFO:teuthology.orchestra.run.smithi071.stdout: "email": "", 2023-05-24T21:00:37.109 INFO:teuthology.orchestra.run.smithi071.stdout: "suspended": 0, 2023-05-24T21:00:37.109 INFO:teuthology.orchestra.run.smithi071.stdout: "max_buckets": 1000, 2023-05-24T21:00:37.109 INFO:teuthology.orchestra.run.smithi071.stdout: "subusers": [], 2023-05-24T21:00:37.109 INFO:teuthology.orchestra.run.smithi071.stdout: "keys": [ 2023-05-24T21:00:37.110 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:00:37.110 INFO:teuthology.orchestra.run.smithi071.stdout: "user": "rgw-multisite-test-user", 2023-05-24T21:00:37.110 INFO:teuthology.orchestra.run.smithi071.stdout: "access_key": "YAS2LRYUNTM46AEFZJ98", 2023-05-24T21:00:37.110 INFO:teuthology.orchestra.run.smithi071.stdout: "secret_key": "CDVXIAbCwKsiYiEpdkrdsMxCBpkC3s8CqmVBmhuW" 2023-05-24T21:00:37.111 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:00:37.111 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:00:37.111 INFO:teuthology.orchestra.run.smithi071.stdout: "swift_keys": [], 2023-05-24T21:00:37.112 INFO:teuthology.orchestra.run.smithi071.stdout: "caps": [ 2023-05-24T21:00:37.112 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:00:37.112 INFO:teuthology.orchestra.run.smithi071.stdout: "type": "roles", 2023-05-24T21:00:37.112 INFO:teuthology.orchestra.run.smithi071.stdout: "perm": "*" 2023-05-24T21:00:37.113 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:00:37.113 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:00:37.113 INFO:teuthology.orchestra.run.smithi071.stdout: "op_mask": "read, write, delete", 2023-05-24T21:00:37.113 INFO:teuthology.orchestra.run.smithi071.stdout: "default_placement": "", 2023-05-24T21:00:37.113 INFO:teuthology.orchestra.run.smithi071.stdout: "default_storage_class": "", 2023-05-24T21:00:37.113 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_tags": [], 2023-05-24T21:00:37.113 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_quota": { 2023-05-24T21:00:37.114 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false, 2023-05-24T21:00:37.114 INFO:teuthology.orchestra.run.smithi071.stdout: "check_on_raw": false, 2023-05-24T21:00:37.114 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size": -1, 2023-05-24T21:00:37.114 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size_kb": 0, 2023-05-24T21:00:37.114 INFO:teuthology.orchestra.run.smithi071.stdout: "max_objects": -1 2023-05-24T21:00:37.114 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:00:37.114 INFO:teuthology.orchestra.run.smithi071.stdout: "user_quota": { 2023-05-24T21:00:37.114 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false, 2023-05-24T21:00:37.115 INFO:teuthology.orchestra.run.smithi071.stdout: "check_on_raw": false, 2023-05-24T21:00:37.115 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size": -1, 2023-05-24T21:00:37.115 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size_kb": 0, 2023-05-24T21:00:37.115 INFO:teuthology.orchestra.run.smithi071.stdout: "max_objects": -1 2023-05-24T21:00:37.115 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:00:37.115 INFO:teuthology.orchestra.run.smithi071.stdout: "temp_url_keys": [], 2023-05-24T21:00:37.115 INFO:teuthology.orchestra.run.smithi071.stdout: "type": "rgw", 2023-05-24T21:00:37.116 INFO:teuthology.orchestra.run.smithi071.stdout: "mfa_ids": [] 2023-05-24T21:00:37.116 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:00:37.116 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T21:00:37.120 INFO:rgw_multi.tests:meta checkpoint 2023-05-24T21:00:37.121 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:00:37.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:00:37.122 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:00:37.171 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:00:37.172 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:00:37.206 INFO:teuthology.orchestra.run.smithi071.stderr:No --period given, using current period=5c04936b-70ea-4415-b4eb-3aad3dc612e5 2023-05-24T21:00:37.230 INFO:teuthology.orchestra.run.smithi071.stdout:[ 2023-05-24T21:00:37.230 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:00:37.231 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962037.098901_69.1", 2023-05-24T21:00:37.231 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:00:37.098901Z" 2023-05-24T21:00:37.231 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:00:37.231 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:00:37.231 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:00:37.232 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:00:37.232 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:00:37.232 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:00:37.232 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:00:37.232 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:00:37.233 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:00:37.233 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:00:37.233 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:00:37.233 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:00:37.233 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:00:37.233 INFO:teuthology.orchestra.run.smithi071.stdout:] 2023-05-24T21:00:37.234 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-24T21:00:37.235 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:00:37.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:00:37.235 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:00:37.285 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:00:37.285 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:00:37.336 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:00:37.329+0000 7f530e4b7780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:00:37.337 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:00:37.330+0000 7f530e4b7780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:00:37.358 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:00:37.358 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:00:37.358 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:00:37.359 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:00:37.359 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:00:37.359 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:00:37.360 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:00:37.360 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:00:37.360 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:00:37.360 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:00:37.361 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:00:37.361 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:00:37.361 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:00:37.361 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:00:37.361 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:00:37.361 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:00:37.361 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:00:37.362 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:00:37.362 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:00:37.362 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:00:37.362 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:00:37.362 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:00:37.362 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:00:37.362 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:00:37.363 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:00:37.363 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:00:37.363 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:00:37.363 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:00:37.364 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:00:37.364 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:00:37.364 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:00:37.364 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:00:37.364 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:00:37.365 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:00:37.365 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:00:37.366 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:00:37.366 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:00:37.366 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:00:37.366 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:00:37.366 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:00:37.366 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:00:37.366 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:00:37.367 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:00:37.367 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:00:37.367 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:00:37.367 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:00:37.368 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:00:37.368 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:00:37.368 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:00:37.369 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:00:37.369 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:00:37.369 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:00:37.370 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:00:37.370 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:00:37.370 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:00:37.371 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:00:37.371 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:00:37.371 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:00:37.372 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:00:37.372 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:00:37.372 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:00:37.372 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:00:37.373 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:00:37.373 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:00:37Z" 2023-05-24T21:00:37.373 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:00:37.374 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684962037.098901_69.1 target= 2023-05-24T21:00:38.347 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2023-05-24T21:00:42.375 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:00:42.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:00:42.376 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:00:42.434 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:00:42.434 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:00:42.477 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:00:42.470+0000 7f3fd2e9f780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:00:42.478 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:00:42.471+0000 7f3fd2e9f780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:00:42.493 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:00:42.493 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:00:42.494 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:00:42.494 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:00:42.494 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:00:42.494 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:00:42.494 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:00:42.494 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:00:42.494 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:00:42.495 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:00:42.495 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:00:42.495 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:00:42.495 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:00:42.495 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:00:42.495 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:00:42.495 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:00:42.495 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:00:42.496 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:00:42.496 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:00:42.496 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:00:42.496 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:00:42.496 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:00:42.496 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:00:42.496 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:00:42.497 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:00:42.497 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:00:42.497 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:00:42.497 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:00:42.497 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:00:42.497 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:00:42.497 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:00:42.497 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:00:42.498 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:00:42.498 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:00:42.498 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:00:42.498 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:00:42.498 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:00:42.498 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:00:42.498 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:00:42.498 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:00:42.499 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:00:42.499 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:00:42.499 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:00:42.499 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:00:42.499 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:00:42.499 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:00:42.499 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:00:42.500 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:00:42.500 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:00:42.500 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:00:42.500 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:00:42.500 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:00:42.500 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:00:42.500 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:00:42.500 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:00:42.501 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:00:42.501 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:00:42.501 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:00:42.501 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:00:42.501 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:00:42.501 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:00:42.501 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:00:42.501 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:00:42.502 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:00:42Z" 2023-05-24T21:00:42.502 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:00:42.503 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684962037.098901_69.1 target= 2023-05-24T21:00:47.504 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:00:47.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:00:47.505 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:00:47.562 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:00:47.562 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:00:47.609 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:00:47.603+0000 7efec3c4e780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:00:47.610 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:00:47.604+0000 7efec3c4e780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:00:47.630 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:00:47.631 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:00:47.631 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:00:47.631 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:00:47.631 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:00:47.631 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:00:47.631 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:00:47.631 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:00:47.632 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:00:47.632 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:00:47.632 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:00:47.632 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:00:47.632 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:00:47.632 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:00:47.632 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:00:47.633 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:00:47.633 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:00:47.633 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:00:47.633 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:00:47.633 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:00:47.634 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:00:47.634 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:00:47.634 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:00:47.634 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:00:47.634 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:00:47.634 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:00:47.634 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:00:47.635 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:00:47.635 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:00:47.635 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:00:47.635 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:00:47.635 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:00:47.636 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:00:47.636 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:00:47.636 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:00:47.636 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:00:47.637 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:00:47.637 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:00:47.637 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:00:47.637 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:00:47.638 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:00:47.638 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:00:47.638 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:00:47.639 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:00:47.639 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:00:47.639 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:00:47.639 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:00:47.640 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:00:47.640 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:00:47.640 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:00:47.640 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:00:47.641 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:00:47.641 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:00:47.641 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:00:47.642 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:00:47.642 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:00:47.642 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:00:47.642 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:00:47.643 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:00:47.643 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:00:47.643 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:00:47.644 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:00:47.644 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:00:47.644 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:00:47Z" 2023-05-24T21:00:47.644 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:00:47.646 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684962037.098901_69.1 target= 2023-05-24T21:00:52.647 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:00:52.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:00:52.648 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:00:52.706 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:00:52.707 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:00:52.759 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:00:52.753+0000 7f06dc2b4780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:00:52.760 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:00:52.754+0000 7f06dc2b4780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:00:52.775 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:00:52.776 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:00:52.776 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:00:52.776 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:00:52.776 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:00:52.776 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:00:52.776 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:00:52.777 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:00:52.777 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:00:52.777 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:00:52.777 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:00:52.777 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:00:52.777 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:00:52.777 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:00:52.778 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:00:52.778 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:00:52.778 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:00:52.779 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:00:52.779 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:00:52.779 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:00:52.779 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:00:52.779 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:00:52.779 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:00:52.779 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:00:52.780 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:00:52.780 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:00:52.780 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:00:52.780 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:00:52.780 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:00:52.780 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:00:52.780 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:00:52.780 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:00:52.781 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:00:52.781 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:00:52.781 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:00:52.781 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:00:52.781 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:00:52.781 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:00:52.781 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:00:52.782 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:00:52.782 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:00:52.782 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:00:52.782 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:00:52.782 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:00:52.782 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:00:52.782 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:00:52.783 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:00:52.783 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:00:52.783 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:00:52.783 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:00:52.783 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:00:52.783 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:00:52.783 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:00:52.783 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:00:52.784 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:00:52.784 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:00:52.784 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:00:52.784 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:00:52.784 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:00:52.784 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:00:52.784 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:00:52.785 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:00:52.785 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:00:52.785 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:00:52Z" 2023-05-24T21:00:52.785 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:00:52.786 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684962037.098901_69.1 target= 2023-05-24T21:00:57.787 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:00:57.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:00:57.788 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:00:57.842 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:00:57.842 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:00:57.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:00:57.875+0000 7f70f4c34780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:00:57.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:00:57.876+0000 7f70f4c34780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:00:57.893 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:00:57.893 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:00:57.893 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:00:57.893 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:00:57.893 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:00:57.894 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:00:57.894 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:00:57.894 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:00:57.894 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:00:57.894 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:00:57.894 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:00:57.894 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:00:57.895 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:00:57.895 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:00:57.895 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:00:57.895 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:00:57.895 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:00:57.895 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:00:57.895 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:00:57.896 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:00:57.896 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:00:57.896 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:00:57.896 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:00:57.896 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:00:57.896 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:00:57.896 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:00:57.896 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:00:57.897 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:00:57.897 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:00:57.897 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:00:57.897 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:00:57.897 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:00:57.897 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:00:57.897 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:00:57.898 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:00:57.898 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:00:57.898 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:00:57.898 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:00:57.898 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:00:57.898 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:00:57.898 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:00:57.898 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:00:57.899 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:00:57.899 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:00:57.899 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:00:57.899 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:00:57.899 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:00:57.899 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:00:57.899 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:00:57.899 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:00:57.900 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:00:57.900 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:00:57.900 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:00:57.900 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:00:57.900 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:00:57.900 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:00:57.900 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:00:57.901 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:00:57.901 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:00:57.901 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:00:57.901 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:00:57.901 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:00:57.901 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:00:57.901 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:00:57Z" 2023-05-24T21:00:57.901 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:00:57.902 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684962037.098901_69.1 target= 2023-05-24T21:01:02.903 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:01:02.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:01:02.905 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:01:02.963 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:01:02.964 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:01:03.001 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:01:02.995+0000 7fc8518a5780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:01:03.002 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:01:02.996+0000 7fc8518a5780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:01:03.020 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:01:03.021 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:01:03.021 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:01:03.021 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:01:03.021 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:01:03.021 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:01:03.022 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:03.022 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:03.022 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:01:03.022 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:03.022 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:01:03.022 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:03.023 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:03.023 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:03.023 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:03.023 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:01:03.023 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:03.023 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:03.023 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:03.024 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:03.024 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:03.024 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:03.024 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:01:03.024 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:03.024 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:03.024 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:03.025 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:03.025 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:01:03.025 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:03.025 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:03.025 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:03.025 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:03.025 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:03.026 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:03.026 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:01:03.026 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:03.026 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:03.026 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:03.026 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:03.026 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:01:03.026 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:03.027 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:03.027 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:03.027 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:03.027 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:03.028 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:03.028 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:01:03.028 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:03.029 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:03.029 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:03.029 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:03.030 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:01:03.030 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:03.030 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:03.030 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:03.031 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:03.031 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:03.031 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:01:03.031 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:03.031 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:01:03.031 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:01:03.032 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:01:03.032 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:03.032 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:01:02Z" 2023-05-24T21:01:03.032 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:01:03.033 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684962037.098901_69.1 target= 2023-05-24T21:01:08.035 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:01:08.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:01:08.036 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:01:08.092 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:01:08.093 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:01:08.137 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:01:08.131+0000 7f3221f58780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:01:08.137 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:01:08.132+0000 7f3221f58780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:01:08.157 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:01:08.158 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:01:08.158 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:01:08.158 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:01:08.158 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:01:08.158 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:01:08.159 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:08.159 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:08.159 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:01:08.159 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:08.159 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:01:08.160 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:08.160 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:08.160 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:08.161 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:08.161 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:01:08.161 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:08.161 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:08.162 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:08.162 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:08.162 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:08.163 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:08.163 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:01:08.163 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:08.163 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:08.164 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:08.164 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:08.164 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:01:08.164 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:08.165 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:08.165 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:08.165 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:08.166 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:08.166 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:08.166 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:01:08.166 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:08.166 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:08.166 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:08.167 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:08.167 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:01:08.167 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:08.167 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:08.167 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:08.167 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:08.167 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:08.168 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:08.168 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:01:08.168 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:08.168 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:08.168 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:08.168 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:08.168 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:01:08.169 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:08.169 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:08.169 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:08.169 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:08.169 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:08.169 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:01:08.170 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:08.170 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:01:08.170 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:01:08.171 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:01:08.171 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:08.171 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:01:08Z" 2023-05-24T21:01:08.171 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:01:08.172 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684962037.098901_69.1 target= 2023-05-24T21:01:13.173 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:01:13.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:01:13.174 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:01:13.231 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:01:13.232 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:01:13.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:01:13.259+0000 7f2e399d2780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:01:13.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:01:13.260+0000 7f2e399d2780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:01:13.278 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:01:13.278 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:01:13.278 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:01:13.278 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:01:13.279 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:01:13.279 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:01:13.279 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:13.279 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:13.279 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:01:13.279 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:13.279 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:01:13.280 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:13.280 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:13.280 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:13.280 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:13.280 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:01:13.280 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:13.280 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:13.281 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:13.281 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:13.281 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:13.281 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:13.281 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:01:13.281 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:13.281 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:13.281 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:13.282 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:13.282 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:01:13.282 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:13.282 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:13.282 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:13.282 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:13.282 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:13.282 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:13.283 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:01:13.283 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:13.283 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:13.283 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:13.283 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:13.283 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:01:13.283 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:13.284 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:13.284 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:13.284 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:13.284 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:13.284 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:13.284 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:01:13.284 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:13.284 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:13.285 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:13.285 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:13.285 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:01:13.285 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:13.285 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:13.285 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:13.285 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:13.285 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:13.286 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:01:13.286 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:13.286 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:01:13.286 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:01:13.286 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:01:13.286 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:13.286 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:01:13Z" 2023-05-24T21:01:13.286 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:01:13.287 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684962037.098901_69.1 target= 2023-05-24T21:01:18.288 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:01:18.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:01:18.289 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:01:18.345 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:01:18.346 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:01:18.398 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:01:18.391+0000 7f43cb3dc780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:01:18.398 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:01:18.392+0000 7f43cb3dc780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:01:18.414 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:01:18.415 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:01:18.415 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:01:18.415 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:01:18.415 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:01:18.415 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:01:18.415 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:18.416 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:18.416 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:01:18.416 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:18.416 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:01:18.416 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:18.416 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:18.416 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:18.417 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:18.417 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:01:18.417 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:18.417 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:18.417 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:18.417 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:18.417 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:18.417 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:18.418 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:01:18.418 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:18.418 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:18.418 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:18.418 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:18.418 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:01:18.418 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:18.419 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:18.419 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:18.419 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:18.419 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:18.419 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:18.419 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:01:18.419 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:18.419 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:18.420 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:18.420 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:18.420 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:01:18.420 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:18.420 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:18.420 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:18.420 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:18.420 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:18.421 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:18.421 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:01:18.421 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:18.421 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:18.421 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:18.421 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:18.421 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:01:18.422 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:18.422 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:18.422 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:18.422 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:18.422 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:18.422 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:01:18.422 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:18.422 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:01:18.423 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:01:18.423 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:01:18.423 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:18.423 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:01:18Z" 2023-05-24T21:01:18.423 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:01:18.424 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684962037.098901_69.1 target= 2023-05-24T21:01:23.425 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:01:23.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:01:23.426 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:01:23.484 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:01:23.485 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:01:23.531 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:01:23.524+0000 7fa690fad780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:01:23.532 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:01:23.525+0000 7fa690fad780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:01:23.550 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:01:23.550 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:01:23.551 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:01:23.551 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:01:23.551 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:01:23.551 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:01:23.551 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:23.551 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:23.551 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:01:23.552 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:23.552 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:01:23.552 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:23.552 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:23.552 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:23.553 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:23.553 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:01:23.553 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:23.553 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:23.553 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:23.553 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:23.554 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:23.554 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:23.554 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:01:23.554 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:23.554 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:23.555 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:23.555 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:23.555 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:01:23.555 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:23.555 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:23.555 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:23.555 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:23.555 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:23.556 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:23.556 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:01:23.556 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:23.556 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:23.556 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:23.556 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:23.556 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:01:23.556 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:23.557 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:23.557 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:23.557 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:23.557 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:23.558 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:23.558 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:01:23.558 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:23.558 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:23.558 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:23.558 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:23.558 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:01:23.558 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:23.559 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:23.559 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:23.559 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:23.559 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:23.559 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:01:23.559 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:23.559 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:01:23.559 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:01:23.560 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:01:23.560 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:23.560 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:01:23Z" 2023-05-24T21:01:23.560 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:01:23.561 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684962037.098901_69.1 target= 2023-05-24T21:01:28.562 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:01:28.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:01:28.563 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:01:28.620 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:01:28.621 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:01:28.652 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:01:28.646+0000 7f58315fd780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:01:28.653 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:01:28.647+0000 7f58315fd780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:01:28.665 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:01:28.666 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:01:28.666 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:01:28.666 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:01:28.666 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:01:28.666 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:01:28.666 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:28.666 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:28.667 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:01:28.667 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:28.667 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:01:28.667 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:28.667 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:28.667 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:28.667 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:28.668 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:01:28.668 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:28.668 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:28.668 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:28.668 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:28.668 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:28.668 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:28.669 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:01:28.669 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:28.669 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:28.669 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:28.669 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:28.669 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:01:28.669 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:28.670 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:28.670 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:28.670 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:28.670 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:28.670 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:28.670 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:01:28.670 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:28.671 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:28.671 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:28.671 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:28.671 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:01:28.671 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:28.671 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:28.671 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:28.671 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:28.672 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:28.672 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:28.672 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:01:28.672 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:28.672 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:28.672 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:28.672 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:28.673 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:01:28.673 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:28.673 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:28.673 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:28.673 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:28.673 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:28.673 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:01:28.673 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:28.674 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:01:28.674 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:01:28.674 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:01:28.674 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:28.674 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:01:28Z" 2023-05-24T21:01:28.674 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:01:28.675 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684962037.098901_69.1 target= 2023-05-24T21:01:33.677 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:01:33.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:01:33.678 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:01:33.738 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:01:33.739 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:01:33.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:01:33.789+0000 7fa945ad4780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:01:33.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:01:33.790+0000 7fa945ad4780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:01:33.809 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:01:33.809 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:01:33.810 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:01:33.810 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:01:33.810 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:01:33.810 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:01:33.810 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:33.810 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:33.811 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:01:33.811 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:33.811 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:01:33.811 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:33.811 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:33.811 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:33.811 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:33.811 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:01:33.812 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:33.812 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:33.812 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:33.812 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:33.812 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:33.812 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:33.813 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:01:33.813 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:33.813 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:33.813 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:33.813 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:33.813 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:01:33.813 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:33.813 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:33.814 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:33.814 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:33.814 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:33.814 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:33.814 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:01:33.815 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:33.815 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:33.815 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:33.815 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:33.816 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:01:33.817 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:33.817 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:33.817 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:33.818 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:33.818 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:33.818 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:33.818 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:01:33.818 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:33.818 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:33.818 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:33.819 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:33.819 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:01:33.819 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:33.819 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:33.819 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:33.819 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:33.819 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:33.819 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:01:33.820 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:33.820 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:01:33.820 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:01:33.820 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:01:33.820 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:33.820 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:01:33Z" 2023-05-24T21:01:33.820 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:01:33.821 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684962037.098901_69.1 target= 2023-05-24T21:01:38.822 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:01:38.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:01:38.824 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:01:38.881 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:01:38.881 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:01:38.914 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:01:38.907+0000 7f2ec19ab780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:01:38.915 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:01:38.908+0000 7f2ec19ab780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:01:38.929 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:01:38.930 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:01:38.930 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:01:38.930 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:01:38.930 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:01:38.930 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:01:38.931 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:38.931 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:38.931 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:01:38.931 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:38.931 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:01:38.931 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:38.932 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:38.932 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:38.932 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:38.932 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:01:38.933 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:38.933 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:38.933 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:38.933 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:38.933 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:38.934 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:38.934 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:01:38.934 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:38.934 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:38.934 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:38.934 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:38.934 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:01:38.935 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:38.935 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:38.935 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:38.935 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:38.936 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:38.936 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:38.936 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:01:38.936 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:38.936 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:38.937 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:38.937 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:38.937 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:01:38.937 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:38.937 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:38.937 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:38.938 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:38.939 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:38.940 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:38.940 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:01:38.940 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:38.940 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:38.940 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:38.940 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:38.941 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:01:38.941 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:38.941 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:38.941 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:38.941 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:38.941 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:38.941 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:01:38.941 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:38.942 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:01:38.942 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:01:38.942 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:01:38.942 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:38.942 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:01:38Z" 2023-05-24T21:01:38.942 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:01:38.943 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684962037.098901_69.1 target= 2023-05-24T21:01:43.944 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:01:43.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:01:43.946 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:01:44.004 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:01:44.005 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:01:44.046 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:01:44.040+0000 7f5d818b4780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:01:44.047 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:01:44.041+0000 7f5d818b4780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:01:44.058 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:01:44.059 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:01:44.059 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:01:44.059 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:01:44.060 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:01:44.060 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:01:44.060 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:44.060 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:44.061 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:01:44.061 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:44.061 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:01:44.061 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:44.061 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:44.062 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:44.062 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:44.062 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:01:44.062 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:44.063 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:44.063 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:44.063 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:44.063 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:44.063 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:44.063 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:01:44.064 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:44.064 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:44.064 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:44.064 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:44.065 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:01:44.065 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:44.065 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:44.066 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:44.066 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:44.066 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:44.066 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:44.066 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:01:44.066 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:44.066 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:44.067 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:44.067 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:44.067 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:01:44.067 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:44.067 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:44.067 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:44.067 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:44.067 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:44.068 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:44.068 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:01:44.068 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:44.068 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:44.068 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:44.068 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:44.068 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:01:44.069 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:44.069 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:44.069 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:44.069 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:44.069 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:44.069 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:01:44.069 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:44.070 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:01:44.070 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:01:44.070 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:01:44.070 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:44.070 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:01:44Z" 2023-05-24T21:01:44.070 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:01:44.071 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684962037.098901_69.1 target= 2023-05-24T21:01:49.073 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:01:49.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:01:49.074 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:01:49.131 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:01:49.132 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:01:49.163 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:01:49.157+0000 7f8c4f367780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:01:49.164 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:01:49.158+0000 7f8c4f367780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:01:49.176 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:01:49.176 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:01:49.176 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:01:49.176 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:01:49.176 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:01:49.177 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:01:49.177 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:49.177 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:49.177 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:01:49.177 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:49.177 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:01:49.177 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:49.178 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:49.178 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:49.178 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:49.178 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:01:49.178 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:49.178 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:49.179 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:49.179 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:49.179 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:49.179 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:49.180 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:01:49.180 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:49.180 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:49.180 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:49.180 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:49.180 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:01:49.181 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:49.181 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:49.181 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:49.181 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:49.181 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:49.181 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:49.182 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:01:49.182 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:49.182 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:49.182 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:49.182 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:49.182 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:01:49.183 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:49.183 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:49.183 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:49.183 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:49.183 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:49.183 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:49.183 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:01:49.183 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:49.184 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:49.184 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:49.184 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:49.184 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:01:49.184 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:49.185 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:49.185 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:49.185 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:49.185 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:49.185 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:01:49.185 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:49.185 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:01:49.186 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:01:49.186 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:01:49.186 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:49.186 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:01:49Z" 2023-05-24T21:01:49.186 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:01:49.188 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684962037.098901_69.1 target= 2023-05-24T21:01:54.188 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:01:54.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:01:54.189 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:01:54.247 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:01:54.248 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:01:54.299 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:01:54.293+0000 7fa559876780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:01:54.300 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:01:54.294+0000 7fa559876780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:01:54.316 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:01:54.317 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:01:54.317 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:01:54.317 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:01:54.317 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:01:54.317 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:01:54.318 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:54.318 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:54.318 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:01:54.318 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:54.318 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:01:54.318 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:54.318 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:54.319 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:54.319 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:54.319 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:01:54.319 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:54.319 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:54.319 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:54.319 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:54.320 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:54.320 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:54.320 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:01:54.320 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:54.321 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:54.321 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:54.321 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:54.321 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:01:54.321 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:54.321 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:54.321 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:54.322 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:54.322 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:54.322 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:54.322 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:01:54.322 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:54.322 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:54.322 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:54.323 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:54.323 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:01:54.323 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:54.323 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:54.323 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:54.323 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:54.323 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:54.324 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:54.324 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:01:54.324 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:54.324 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:54.324 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:54.324 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:54.325 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:01:54.325 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:54.325 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:54.325 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:54.325 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:54.325 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:54.325 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:01:54.326 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:54.326 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:01:54.326 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:01:54.326 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:01:54.326 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:54.326 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:01:54Z" 2023-05-24T21:01:54.326 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:01:54.327 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684962037.098901_69.1 target= 2023-05-24T21:01:59.329 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:01:59.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:01:59.330 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:01:59.388 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:01:59.388 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:01:59.425 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:01:59.420+0000 7fcb533dc780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:01:59.426 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:01:59.421+0000 7fcb533dc780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:01:59.439 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:01:59.439 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:01:59.439 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:01:59.439 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:01:59.439 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:01:59.440 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:01:59.440 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:59.440 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:59.440 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:01:59.440 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:59.440 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:01:59.441 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:59.441 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:59.441 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:59.441 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:59.442 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:01:59.442 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:59.442 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:59.443 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:59.443 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:59.443 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:59.443 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:59.444 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:01:59.444 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:59.444 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:59.445 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:59.445 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:59.445 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:01:59.446 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:59.446 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:59.446 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:59.447 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:59.447 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:59.447 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:59.447 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:01:59.448 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:59.448 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:59.448 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:59.449 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:59.449 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:01:59.449 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:59.449 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:59.449 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:59.449 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:59.449 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:59.450 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:01:59.450 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:01:59.450 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:01:59.450 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:01:59.451 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:01:59.451 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:01:59.451 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:01:59.451 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:01:59.452 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:01:59.452 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:01:59.452 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:59.453 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:01:59.453 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:01:59.453 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:59.454 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:01:59.454 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:01:59.454 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:01:59.454 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:01:59.454 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:01:59Z" 2023-05-24T21:01:59.455 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:01:59.456 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684962037.098901_69.1 target= 2023-05-24T21:02:04.457 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:04.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:04.458 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:02:04.515 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:02:04.515 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:02:04.552 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:02:04.546+0000 7fa889239780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:02:04.553 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:02:04.547+0000 7fa889239780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:02:04.568 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:02:04.568 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:02:04.569 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:02:04.569 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:02:04.569 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:02:04.569 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:02:04.569 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:04.569 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:04.569 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:02:04.570 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:04.570 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:02:04.570 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:04.570 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:04.570 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:04.570 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:04.570 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:02:04.571 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:04.571 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:02:04.571 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:04.571 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:04.571 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:04.571 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:04.571 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:02:04.571 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:04.572 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:04.572 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:04.572 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:04.572 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:04.572 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:04.573 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:02:04.573 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:04.573 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:04.573 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:04.574 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:04.574 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:02:04.574 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:04.575 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:04.575 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:04.575 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:04.575 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:04.576 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:04.576 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:02:04.576 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:04.577 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:04.577 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:04.577 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:04.578 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:02:04.578 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:04.578 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:04.578 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:04.579 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:04.579 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:04.579 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:04.580 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:02:04.580 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:04.580 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:04.581 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:04.581 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:02:04.581 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:04.581 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:02:04.581 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:02:04.582 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:02:04.582 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:04.582 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:02:04Z" 2023-05-24T21:02:04.582 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:02:04.584 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684962037.098901_69.1 target= 2023-05-24T21:02:09.585 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:09.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:09.586 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:02:09.642 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:02:09.643 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:02:09.687 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:02:09.681+0000 7f20e9363780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:02:09.689 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:02:09.683+0000 7f20e9363780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:02:09.707 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:02:09.708 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:02:09.708 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:02:09.708 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:02:09.708 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:02:09.708 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:02:09.709 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:09.709 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:09.709 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:02:09.709 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:09.709 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:02:09.709 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:09.709 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:09.710 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:09.710 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:09.710 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:02:09.710 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:09.710 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:02:09.710 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:09.710 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:09.710 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:09.711 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:09.711 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:02:09.711 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:09.711 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:09.711 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:09.711 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:09.712 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:09.712 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:09.712 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:02:09.712 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:09.712 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:09.713 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:09.713 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:09.714 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:02:09.714 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:09.714 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:09.714 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:09.714 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:09.714 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:09.715 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:09.715 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:02:09.715 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:09.715 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:09.715 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:09.715 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:09.715 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:02:09.716 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:09.716 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:09.716 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:09.716 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:09.716 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:09.716 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:09.716 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:02:09.717 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:09.717 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:09.717 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:09.717 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:02:09.717 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:09.717 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:02:09.717 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:02:09.717 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:02:09.718 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:09.718 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:02:09Z" 2023-05-24T21:02:09.718 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:02:09.719 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684962037.098901_69.1 target= 2023-05-24T21:02:14.719 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:14.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:14.721 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:02:14.781 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:02:14.781 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:02:14.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:02:14.811+0000 7fe4a21cf780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:02:14.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:02:14.812+0000 7fe4a21cf780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:02:14.831 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:02:14.831 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:02:14.831 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:02:14.832 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:02:14.832 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:02:14.832 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:02:14.832 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:14.832 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:14.832 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:02:14.833 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:14.833 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:02:14.833 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:14.833 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:14.833 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:14.833 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:14.833 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:02:14.834 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:14.834 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:02:14.834 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:14.834 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:14.834 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:14.834 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:14.834 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:02:14.835 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:14.835 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:14.835 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:14.835 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:14.835 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:14.835 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:14.835 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:02:14.835 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:14.836 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:14.836 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:14.836 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:14.836 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:02:14.836 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:14.836 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:14.836 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:14.837 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:14.837 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:14.837 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:14.837 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:02:14.837 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:14.837 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:14.837 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:14.837 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:14.838 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:02:14.838 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:14.838 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:14.838 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:14.838 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:14.838 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:14.838 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:14.838 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:02:14.839 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:14.839 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:14.839 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:14.839 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:02:14.839 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:14.839 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:02:14.839 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:02:14.840 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:02:14.840 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:14.840 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:02:14Z" 2023-05-24T21:02:14.840 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:02:14.841 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684962037.098901_69.1 target= 2023-05-24T21:02:19.842 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:19.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:19.843 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:02:19.899 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:02:19.900 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:02:19.952 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:02:19.944+0000 7f9294b1a780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:02:19.952 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:02:19.945+0000 7f9294b1a780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:02:19.970 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:02:19.970 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:02:19.970 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:02:19.970 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:02:19.970 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:02:19.971 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:02:19.971 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:19.971 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:19.971 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:02:19.971 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:19.971 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:02:19.971 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:19.972 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:19.972 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:19.973 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:19.973 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:02:19.973 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:19.973 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:02:19.974 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:19.974 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:19.974 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:19.975 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:19.975 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:02:19.976 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:19.976 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:19.977 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:19.977 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:19.977 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:19.977 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:19.977 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:02:19.977 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:19.977 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:19.978 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:19.978 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:19.978 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:02:19.978 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:19.978 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:19.978 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:19.978 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:19.978 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:19.979 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:19.979 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:02:19.979 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:19.979 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:19.979 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:19.979 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:19.979 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:02:19.980 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:19.980 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:19.980 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:19.980 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:19.980 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:19.980 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:19.980 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:02:19.981 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:19.981 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:19.981 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:19.981 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:02:19.981 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:19.981 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:02:19.981 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:02:19.982 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:02:19.982 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:19.982 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:02:19Z" 2023-05-24T21:02:19.982 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:02:19.983 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684962037.098901_69.1 target= 2023-05-24T21:02:24.985 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:24.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:24.986 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:02:25.043 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:02:25.043 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:02:25.080 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:02:25.073+0000 7f724a30d780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:02:25.080 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:02:25.074+0000 7f724a30d780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:02:25.095 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:02:25.096 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:02:25.096 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:02:25.096 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:02:25.096 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:02:25.096 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:02:25.096 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:25.097 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:25.097 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:02:25.097 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:25.097 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:02:25.097 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:25.097 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:25.097 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:25.098 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:25.098 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:02:25.098 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:25.098 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:02:25.098 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:25.098 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:25.098 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:25.098 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:25.099 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:02:25.099 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:25.099 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:25.099 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:25.099 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:25.099 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:25.099 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:25.100 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:02:25.100 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:25.100 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:25.100 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:25.100 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:25.100 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:02:25.100 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:25.100 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:25.101 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:25.101 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:25.101 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:25.101 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:25.101 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:02:25.101 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:25.101 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:25.102 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:25.102 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:25.102 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:02:25.102 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:25.102 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:25.102 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:25.102 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:25.102 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:25.103 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:25.103 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:02:25.103 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:25.103 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:25.103 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:25.103 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:02:25.103 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:25.103 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:02:25.104 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:02:25.104 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:02:25.104 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:25.104 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:02:25Z" 2023-05-24T21:02:25.104 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:02:25.105 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684962037.098901_69.1 target= 2023-05-24T21:02:30.107 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:30.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:30.108 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:02:30.165 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:02:30.166 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:02:30.205 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:02:30.198+0000 7f0c1d210780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:02:30.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:02:30.199+0000 7f0c1d210780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:02:30.228 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:02:30.228 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:02:30.229 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:02:30.229 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:02:30.229 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:02:30.229 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:02:30.229 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:30.229 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:30.229 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:02:30.229 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:30.230 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:02:30.230 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:30.230 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:30.230 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:30.230 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:30.230 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:02:30.230 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:30.231 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:02:30.231 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:30.231 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:30.231 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:30.231 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:30.231 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:02:30.231 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:30.232 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:30.232 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:30.232 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:30.232 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:30.232 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:30.232 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:02:30.232 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:30.232 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:30.233 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:30.233 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:30.233 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:02:30.233 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:30.233 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:30.233 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:30.233 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:30.234 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:30.234 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:30.234 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:02:30.234 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:30.234 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:30.234 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:30.234 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:30.235 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:02:30.235 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:30.235 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:30.235 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:30.235 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:30.235 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:30.235 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:30.236 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:02:30.236 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:30.237 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:30.237 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:30.237 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:02:30.237 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:30.237 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:02:30.237 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:02:30.237 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:02:30.238 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:30.238 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:02:30Z" 2023-05-24T21:02:30.238 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:02:30.239 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684962037.098901_69.1 target= 2023-05-24T21:02:35.240 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:35.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:35.242 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:02:35.299 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:02:35.299 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:02:35.332 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:02:35.325+0000 7f74b83c5780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:02:35.332 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:02:35.326+0000 7f74b83c5780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:02:35.348 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:02:35.348 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:02:35.348 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:02:35.348 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:02:35.348 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:02:35.349 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:02:35.349 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:35.349 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:35.349 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:02:35.349 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:35.349 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:02:35.350 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:35.350 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:35.350 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962037.098901_69.1", 2023-05-24T21:02:35.350 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:35.350 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:02:35.350 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:35.351 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:00:37.098901Z", 2023-05-24T21:02:35.351 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:35.351 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:35.351 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:35.351 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:35.351 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:02:35.351 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:35.351 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:35.352 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:35.352 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:35.352 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:35.352 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:35.352 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:02:35.352 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:35.352 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:35.353 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:35.353 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:35.353 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:02:35.353 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:35.353 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:35.353 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:35.353 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:35.354 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:35.354 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:35.354 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:02:35.354 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:35.354 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:35.354 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:35.354 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:35.354 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:02:35.355 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:35.355 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:35.355 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:35.355 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:35.355 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:35.355 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:35.355 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:02:35.356 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:35.356 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:35.356 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:35.356 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:02:35.356 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:35.356 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:02:35.356 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:02:35.356 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:02:35.357 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:35.357 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:02:35Z" 2023-05-24T21:02:35.357 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:02:35.358 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-24T21:02:35.358 INFO:rgw_multi.tests:starting meta checkpoint for zone=b1 2023-05-24T21:02:35.358 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:35.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:35.359 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:02:35.418 INFO:teuthology.orchestra.run.smithi157.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:02:35.418 INFO:teuthology.orchestra.run.smithi157.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:02:35.449 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T21:02:35.443+0000 7fab4d138780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:02:35.450 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T21:02:35.444+0000 7fab4d138780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:02:35.462 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2023-05-24T21:02:35.462 INFO:teuthology.orchestra.run.smithi157.stdout: "sync_status": { 2023-05-24T21:02:35.462 INFO:teuthology.orchestra.run.smithi157.stdout: "info": { 2023-05-24T21:02:35.463 INFO:teuthology.orchestra.run.smithi157.stdout: "status": "sync", 2023-05-24T21:02:35.463 INFO:teuthology.orchestra.run.smithi157.stdout: "num_shards": 4, 2023-05-24T21:02:35.463 INFO:teuthology.orchestra.run.smithi157.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:02:35.463 INFO:teuthology.orchestra.run.smithi157.stdout: "realm_epoch": 2 2023-05-24T21:02:35.463 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:02:35.463 INFO:teuthology.orchestra.run.smithi157.stdout: "markers": [ 2023-05-24T21:02:35.463 INFO:teuthology.orchestra.run.smithi157.stdout: { 2023-05-24T21:02:35.464 INFO:teuthology.orchestra.run.smithi157.stdout: "key": 0, 2023-05-24T21:02:35.464 INFO:teuthology.orchestra.run.smithi157.stdout: "val": { 2023-05-24T21:02:35.464 INFO:teuthology.orchestra.run.smithi157.stdout: "state": 1, 2023-05-24T21:02:35.464 INFO:teuthology.orchestra.run.smithi157.stdout: "marker": "1_1684962037.098901_69.1", 2023-05-24T21:02:35.464 INFO:teuthology.orchestra.run.smithi157.stdout: "next_step_marker": "", 2023-05-24T21:02:35.464 INFO:teuthology.orchestra.run.smithi157.stdout: "total_entries": 1, 2023-05-24T21:02:35.465 INFO:teuthology.orchestra.run.smithi157.stdout: "pos": 0, 2023-05-24T21:02:35.465 INFO:teuthology.orchestra.run.smithi157.stdout: "timestamp": "2023-05-24T21:00:37.098901Z", 2023-05-24T21:02:35.465 INFO:teuthology.orchestra.run.smithi157.stdout: "realm_epoch": 2 2023-05-24T21:02:35.465 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:02:35.465 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:02:35.465 INFO:teuthology.orchestra.run.smithi157.stdout: { 2023-05-24T21:02:35.465 INFO:teuthology.orchestra.run.smithi157.stdout: "key": 1, 2023-05-24T21:02:35.465 INFO:teuthology.orchestra.run.smithi157.stdout: "val": { 2023-05-24T21:02:35.466 INFO:teuthology.orchestra.run.smithi157.stdout: "state": 1, 2023-05-24T21:02:35.466 INFO:teuthology.orchestra.run.smithi157.stdout: "marker": "", 2023-05-24T21:02:35.466 INFO:teuthology.orchestra.run.smithi157.stdout: "next_step_marker": "", 2023-05-24T21:02:35.466 INFO:teuthology.orchestra.run.smithi157.stdout: "total_entries": 0, 2023-05-24T21:02:35.466 INFO:teuthology.orchestra.run.smithi157.stdout: "pos": 0, 2023-05-24T21:02:35.466 INFO:teuthology.orchestra.run.smithi157.stdout: "timestamp": "0.000000", 2023-05-24T21:02:35.466 INFO:teuthology.orchestra.run.smithi157.stdout: "realm_epoch": 2 2023-05-24T21:02:35.467 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:02:35.467 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:02:35.467 INFO:teuthology.orchestra.run.smithi157.stdout: { 2023-05-24T21:02:35.467 INFO:teuthology.orchestra.run.smithi157.stdout: "key": 2, 2023-05-24T21:02:35.467 INFO:teuthology.orchestra.run.smithi157.stdout: "val": { 2023-05-24T21:02:35.467 INFO:teuthology.orchestra.run.smithi157.stdout: "state": 1, 2023-05-24T21:02:35.467 INFO:teuthology.orchestra.run.smithi157.stdout: "marker": "", 2023-05-24T21:02:35.467 INFO:teuthology.orchestra.run.smithi157.stdout: "next_step_marker": "", 2023-05-24T21:02:35.468 INFO:teuthology.orchestra.run.smithi157.stdout: "total_entries": 0, 2023-05-24T21:02:35.468 INFO:teuthology.orchestra.run.smithi157.stdout: "pos": 0, 2023-05-24T21:02:35.468 INFO:teuthology.orchestra.run.smithi157.stdout: "timestamp": "0.000000", 2023-05-24T21:02:35.468 INFO:teuthology.orchestra.run.smithi157.stdout: "realm_epoch": 2 2023-05-24T21:02:35.468 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:02:35.468 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:02:35.468 INFO:teuthology.orchestra.run.smithi157.stdout: { 2023-05-24T21:02:35.469 INFO:teuthology.orchestra.run.smithi157.stdout: "key": 3, 2023-05-24T21:02:35.469 INFO:teuthology.orchestra.run.smithi157.stdout: "val": { 2023-05-24T21:02:35.469 INFO:teuthology.orchestra.run.smithi157.stdout: "state": 1, 2023-05-24T21:02:35.469 INFO:teuthology.orchestra.run.smithi157.stdout: "marker": "", 2023-05-24T21:02:35.469 INFO:teuthology.orchestra.run.smithi157.stdout: "next_step_marker": "", 2023-05-24T21:02:35.469 INFO:teuthology.orchestra.run.smithi157.stdout: "total_entries": 0, 2023-05-24T21:02:35.469 INFO:teuthology.orchestra.run.smithi157.stdout: "pos": 0, 2023-05-24T21:02:35.469 INFO:teuthology.orchestra.run.smithi157.stdout: "timestamp": "0.000000", 2023-05-24T21:02:35.470 INFO:teuthology.orchestra.run.smithi157.stdout: "realm_epoch": 2 2023-05-24T21:02:35.470 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:02:35.470 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:02:35.470 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2023-05-24T21:02:35.470 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:02:35.470 INFO:teuthology.orchestra.run.smithi157.stdout: "full_sync": { 2023-05-24T21:02:35.470 INFO:teuthology.orchestra.run.smithi157.stdout: "total": 1, 2023-05-24T21:02:35.471 INFO:teuthology.orchestra.run.smithi157.stdout: "complete": 1 2023-05-24T21:02:35.471 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:02:35.471 INFO:teuthology.orchestra.run.smithi157.stdout: "current_time": "2023-05-24T21:02:35Z" 2023-05-24T21:02:35.471 INFO:teuthology.orchestra.run.smithi157.stdout:} 2023-05-24T21:02:35.472 INFO:rgw_multi.tests:finish meta checkpoint for zone=b1 2023-05-24T21:02:35.472 INFO:rgw_multi.tests:starting meta checkpoint for zone=b2 2023-05-24T21:02:35.473 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:35.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:35.474 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:02:35.525 INFO:teuthology.orchestra.run.smithi157.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:02:35.525 INFO:teuthology.orchestra.run.smithi157.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:02:35.551 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T21:02:35.546+0000 7f40e04a7780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:02:35.552 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T21:02:35.547+0000 7f40e04a7780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:02:35.563 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2023-05-24T21:02:35.563 INFO:teuthology.orchestra.run.smithi157.stdout: "sync_status": { 2023-05-24T21:02:35.564 INFO:teuthology.orchestra.run.smithi157.stdout: "info": { 2023-05-24T21:02:35.564 INFO:teuthology.orchestra.run.smithi157.stdout: "status": "sync", 2023-05-24T21:02:35.564 INFO:teuthology.orchestra.run.smithi157.stdout: "num_shards": 4, 2023-05-24T21:02:35.564 INFO:teuthology.orchestra.run.smithi157.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:02:35.564 INFO:teuthology.orchestra.run.smithi157.stdout: "realm_epoch": 2 2023-05-24T21:02:35.565 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:02:35.565 INFO:teuthology.orchestra.run.smithi157.stdout: "markers": [ 2023-05-24T21:02:35.565 INFO:teuthology.orchestra.run.smithi157.stdout: { 2023-05-24T21:02:35.565 INFO:teuthology.orchestra.run.smithi157.stdout: "key": 0, 2023-05-24T21:02:35.565 INFO:teuthology.orchestra.run.smithi157.stdout: "val": { 2023-05-24T21:02:35.565 INFO:teuthology.orchestra.run.smithi157.stdout: "state": 1, 2023-05-24T21:02:35.565 INFO:teuthology.orchestra.run.smithi157.stdout: "marker": "1_1684962037.098901_69.1", 2023-05-24T21:02:35.566 INFO:teuthology.orchestra.run.smithi157.stdout: "next_step_marker": "", 2023-05-24T21:02:35.566 INFO:teuthology.orchestra.run.smithi157.stdout: "total_entries": 1, 2023-05-24T21:02:35.566 INFO:teuthology.orchestra.run.smithi157.stdout: "pos": 0, 2023-05-24T21:02:35.566 INFO:teuthology.orchestra.run.smithi157.stdout: "timestamp": "2023-05-24T21:00:37.098901Z", 2023-05-24T21:02:35.566 INFO:teuthology.orchestra.run.smithi157.stdout: "realm_epoch": 2 2023-05-24T21:02:35.566 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:02:35.567 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:02:35.567 INFO:teuthology.orchestra.run.smithi157.stdout: { 2023-05-24T21:02:35.567 INFO:teuthology.orchestra.run.smithi157.stdout: "key": 1, 2023-05-24T21:02:35.567 INFO:teuthology.orchestra.run.smithi157.stdout: "val": { 2023-05-24T21:02:35.567 INFO:teuthology.orchestra.run.smithi157.stdout: "state": 1, 2023-05-24T21:02:35.567 INFO:teuthology.orchestra.run.smithi157.stdout: "marker": "", 2023-05-24T21:02:35.567 INFO:teuthology.orchestra.run.smithi157.stdout: "next_step_marker": "", 2023-05-24T21:02:35.568 INFO:teuthology.orchestra.run.smithi157.stdout: "total_entries": 0, 2023-05-24T21:02:35.568 INFO:teuthology.orchestra.run.smithi157.stdout: "pos": 0, 2023-05-24T21:02:35.568 INFO:teuthology.orchestra.run.smithi157.stdout: "timestamp": "0.000000", 2023-05-24T21:02:35.568 INFO:teuthology.orchestra.run.smithi157.stdout: "realm_epoch": 2 2023-05-24T21:02:35.568 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:02:35.568 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:02:35.569 INFO:teuthology.orchestra.run.smithi157.stdout: { 2023-05-24T21:02:35.569 INFO:teuthology.orchestra.run.smithi157.stdout: "key": 2, 2023-05-24T21:02:35.569 INFO:teuthology.orchestra.run.smithi157.stdout: "val": { 2023-05-24T21:02:35.569 INFO:teuthology.orchestra.run.smithi157.stdout: "state": 1, 2023-05-24T21:02:35.569 INFO:teuthology.orchestra.run.smithi157.stdout: "marker": "", 2023-05-24T21:02:35.569 INFO:teuthology.orchestra.run.smithi157.stdout: "next_step_marker": "", 2023-05-24T21:02:35.569 INFO:teuthology.orchestra.run.smithi157.stdout: "total_entries": 0, 2023-05-24T21:02:35.570 INFO:teuthology.orchestra.run.smithi157.stdout: "pos": 0, 2023-05-24T21:02:35.570 INFO:teuthology.orchestra.run.smithi157.stdout: "timestamp": "0.000000", 2023-05-24T21:02:35.570 INFO:teuthology.orchestra.run.smithi157.stdout: "realm_epoch": 2 2023-05-24T21:02:35.570 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:02:35.570 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:02:35.570 INFO:teuthology.orchestra.run.smithi157.stdout: { 2023-05-24T21:02:35.570 INFO:teuthology.orchestra.run.smithi157.stdout: "key": 3, 2023-05-24T21:02:35.571 INFO:teuthology.orchestra.run.smithi157.stdout: "val": { 2023-05-24T21:02:35.571 INFO:teuthology.orchestra.run.smithi157.stdout: "state": 1, 2023-05-24T21:02:35.571 INFO:teuthology.orchestra.run.smithi157.stdout: "marker": "", 2023-05-24T21:02:35.571 INFO:teuthology.orchestra.run.smithi157.stdout: "next_step_marker": "", 2023-05-24T21:02:35.571 INFO:teuthology.orchestra.run.smithi157.stdout: "total_entries": 0, 2023-05-24T21:02:35.572 INFO:teuthology.orchestra.run.smithi157.stdout: "pos": 0, 2023-05-24T21:02:35.572 INFO:teuthology.orchestra.run.smithi157.stdout: "timestamp": "0.000000", 2023-05-24T21:02:35.572 INFO:teuthology.orchestra.run.smithi157.stdout: "realm_epoch": 2 2023-05-24T21:02:35.573 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:02:35.573 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:02:35.573 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2023-05-24T21:02:35.573 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:02:35.573 INFO:teuthology.orchestra.run.smithi157.stdout: "full_sync": { 2023-05-24T21:02:35.574 INFO:teuthology.orchestra.run.smithi157.stdout: "total": 1, 2023-05-24T21:02:35.574 INFO:teuthology.orchestra.run.smithi157.stdout: "complete": 1 2023-05-24T21:02:35.574 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:02:35.574 INFO:teuthology.orchestra.run.smithi157.stdout: "current_time": "2023-05-24T21:02:35Z" 2023-05-24T21:02:35.574 INFO:teuthology.orchestra.run.smithi157.stdout:} 2023-05-24T21:02:35.575 INFO:rgw_multi.tests:finish meta checkpoint for zone=b2 2023-05-24T21:02:35.575 INFO:rgw_multi.tests:meta checkpoint 2023-05-24T21:02:35.576 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:35.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:35.576 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:02:35.627 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:02:35.627 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:02:35.671 INFO:teuthology.orchestra.run.smithi071.stderr:No --period given, using current period=5c04936b-70ea-4415-b4eb-3aad3dc612e5 2023-05-24T21:02:35.687 INFO:teuthology.orchestra.run.smithi071.stdout:[ 2023-05-24T21:02:35.687 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:35.687 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962037.098901_69.1", 2023-05-24T21:02:35.687 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:00:37.098901Z" 2023-05-24T21:02:35.687 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:35.688 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:35.688 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:35.688 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:02:35.688 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:35.688 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:35.688 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:35.688 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:02:35.689 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:35.689 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:35.689 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:35.689 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:02:35.689 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:35.689 INFO:teuthology.orchestra.run.smithi071.stdout:] 2023-05-24T21:02:35.690 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-24T21:02:35.690 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:35.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:35.691 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:02:35.742 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:02:35.742 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:02:35.783 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:02:35.776+0000 7f08d9bc9780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:02:35.783 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:02:35.777+0000 7f08d9bc9780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:02:35.797 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:02:35.798 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:02:35.798 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:02:35.798 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:02:35.798 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:02:35.799 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:02:35.799 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:35.799 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:35.799 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:02:35.799 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:35.799 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:02:35.799 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:35.799 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:35.800 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962037.098901_69.1", 2023-05-24T21:02:35.800 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:35.800 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:02:35.800 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:35.800 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:00:37.098901Z", 2023-05-24T21:02:35.800 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:35.801 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:35.801 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:35.801 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:35.801 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:02:35.801 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:35.801 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:35.801 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:35.802 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:35.802 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:35.802 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:35.802 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:02:35.802 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:35.802 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:35.802 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:35.802 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:35.803 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:02:35.803 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:35.803 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:35.803 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:35.803 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:35.803 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:35.803 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:35.804 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:02:35.804 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:35.804 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:35.804 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:35.804 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:35.804 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:02:35.804 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:35.805 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:35.805 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:35.805 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:35.805 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:35.805 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:35.805 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:02:35.805 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:35.805 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:35.806 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:35.806 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:02:35.806 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:35.806 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:02:35.806 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:02:35.806 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:02:35.806 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:35.806 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:02:35Z" 2023-05-24T21:02:35.807 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:02:35.808 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-24T21:02:35.808 INFO:rgw_multi.tests:starting meta checkpoint for zone=b1 2023-05-24T21:02:35.808 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:35.808 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:35.809 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:02:35.859 INFO:teuthology.orchestra.run.smithi157.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:02:35.859 INFO:teuthology.orchestra.run.smithi157.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:02:35.885 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T21:02:35.879+0000 7f709bf56780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:02:35.886 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T21:02:35.879+0000 7f709bf56780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:02:35.897 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2023-05-24T21:02:35.898 INFO:teuthology.orchestra.run.smithi157.stdout: "sync_status": { 2023-05-24T21:02:35.898 INFO:teuthology.orchestra.run.smithi157.stdout: "info": { 2023-05-24T21:02:35.898 INFO:teuthology.orchestra.run.smithi157.stdout: "status": "sync", 2023-05-24T21:02:35.898 INFO:teuthology.orchestra.run.smithi157.stdout: "num_shards": 4, 2023-05-24T21:02:35.899 INFO:teuthology.orchestra.run.smithi157.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:02:35.899 INFO:teuthology.orchestra.run.smithi157.stdout: "realm_epoch": 2 2023-05-24T21:02:35.899 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:02:35.899 INFO:teuthology.orchestra.run.smithi157.stdout: "markers": [ 2023-05-24T21:02:35.899 INFO:teuthology.orchestra.run.smithi157.stdout: { 2023-05-24T21:02:35.900 INFO:teuthology.orchestra.run.smithi157.stdout: "key": 0, 2023-05-24T21:02:35.900 INFO:teuthology.orchestra.run.smithi157.stdout: "val": { 2023-05-24T21:02:35.900 INFO:teuthology.orchestra.run.smithi157.stdout: "state": 1, 2023-05-24T21:02:35.900 INFO:teuthology.orchestra.run.smithi157.stdout: "marker": "1_1684962037.098901_69.1", 2023-05-24T21:02:35.900 INFO:teuthology.orchestra.run.smithi157.stdout: "next_step_marker": "", 2023-05-24T21:02:35.901 INFO:teuthology.orchestra.run.smithi157.stdout: "total_entries": 1, 2023-05-24T21:02:35.901 INFO:teuthology.orchestra.run.smithi157.stdout: "pos": 0, 2023-05-24T21:02:35.901 INFO:teuthology.orchestra.run.smithi157.stdout: "timestamp": "2023-05-24T21:00:37.098901Z", 2023-05-24T21:02:35.902 INFO:teuthology.orchestra.run.smithi157.stdout: "realm_epoch": 2 2023-05-24T21:02:35.902 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:02:35.902 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:02:35.902 INFO:teuthology.orchestra.run.smithi157.stdout: { 2023-05-24T21:02:35.902 INFO:teuthology.orchestra.run.smithi157.stdout: "key": 1, 2023-05-24T21:02:35.903 INFO:teuthology.orchestra.run.smithi157.stdout: "val": { 2023-05-24T21:02:35.903 INFO:teuthology.orchestra.run.smithi157.stdout: "state": 1, 2023-05-24T21:02:35.903 INFO:teuthology.orchestra.run.smithi157.stdout: "marker": "", 2023-05-24T21:02:35.903 INFO:teuthology.orchestra.run.smithi157.stdout: "next_step_marker": "", 2023-05-24T21:02:35.903 INFO:teuthology.orchestra.run.smithi157.stdout: "total_entries": 0, 2023-05-24T21:02:35.904 INFO:teuthology.orchestra.run.smithi157.stdout: "pos": 0, 2023-05-24T21:02:35.904 INFO:teuthology.orchestra.run.smithi157.stdout: "timestamp": "0.000000", 2023-05-24T21:02:35.904 INFO:teuthology.orchestra.run.smithi157.stdout: "realm_epoch": 2 2023-05-24T21:02:35.904 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:02:35.904 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:02:35.905 INFO:teuthology.orchestra.run.smithi157.stdout: { 2023-05-24T21:02:35.905 INFO:teuthology.orchestra.run.smithi157.stdout: "key": 2, 2023-05-24T21:02:35.905 INFO:teuthology.orchestra.run.smithi157.stdout: "val": { 2023-05-24T21:02:35.905 INFO:teuthology.orchestra.run.smithi157.stdout: "state": 1, 2023-05-24T21:02:35.905 INFO:teuthology.orchestra.run.smithi157.stdout: "marker": "", 2023-05-24T21:02:35.906 INFO:teuthology.orchestra.run.smithi157.stdout: "next_step_marker": "", 2023-05-24T21:02:35.906 INFO:teuthology.orchestra.run.smithi157.stdout: "total_entries": 0, 2023-05-24T21:02:35.906 INFO:teuthology.orchestra.run.smithi157.stdout: "pos": 0, 2023-05-24T21:02:35.906 INFO:teuthology.orchestra.run.smithi157.stdout: "timestamp": "0.000000", 2023-05-24T21:02:35.906 INFO:teuthology.orchestra.run.smithi157.stdout: "realm_epoch": 2 2023-05-24T21:02:35.907 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:02:35.907 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:02:35.907 INFO:teuthology.orchestra.run.smithi157.stdout: { 2023-05-24T21:02:35.907 INFO:teuthology.orchestra.run.smithi157.stdout: "key": 3, 2023-05-24T21:02:35.907 INFO:teuthology.orchestra.run.smithi157.stdout: "val": { 2023-05-24T21:02:35.908 INFO:teuthology.orchestra.run.smithi157.stdout: "state": 1, 2023-05-24T21:02:35.908 INFO:teuthology.orchestra.run.smithi157.stdout: "marker": "", 2023-05-24T21:02:35.908 INFO:teuthology.orchestra.run.smithi157.stdout: "next_step_marker": "", 2023-05-24T21:02:35.908 INFO:teuthology.orchestra.run.smithi157.stdout: "total_entries": 0, 2023-05-24T21:02:35.908 INFO:teuthology.orchestra.run.smithi157.stdout: "pos": 0, 2023-05-24T21:02:35.909 INFO:teuthology.orchestra.run.smithi157.stdout: "timestamp": "0.000000", 2023-05-24T21:02:35.909 INFO:teuthology.orchestra.run.smithi157.stdout: "realm_epoch": 2 2023-05-24T21:02:35.909 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:02:35.909 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:02:35.909 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2023-05-24T21:02:35.909 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:02:35.910 INFO:teuthology.orchestra.run.smithi157.stdout: "full_sync": { 2023-05-24T21:02:35.910 INFO:teuthology.orchestra.run.smithi157.stdout: "total": 1, 2023-05-24T21:02:35.910 INFO:teuthology.orchestra.run.smithi157.stdout: "complete": 1 2023-05-24T21:02:35.910 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:02:35.910 INFO:teuthology.orchestra.run.smithi157.stdout: "current_time": "2023-05-24T21:02:35Z" 2023-05-24T21:02:35.911 INFO:teuthology.orchestra.run.smithi157.stdout:} 2023-05-24T21:02:35.912 INFO:rgw_multi.tests:finish meta checkpoint for zone=b1 2023-05-24T21:02:35.912 INFO:rgw_multi.tests:starting meta checkpoint for zone=b2 2023-05-24T21:02:35.913 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:35.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:35.913 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:02:35.963 INFO:teuthology.orchestra.run.smithi157.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:02:35.964 INFO:teuthology.orchestra.run.smithi157.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:02:35.990 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T21:02:35.984+0000 7fbd2ad14780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:02:35.991 INFO:teuthology.orchestra.run.smithi157.stderr:2023-05-24T21:02:35.985+0000 7fbd2ad14780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:02:36.002 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2023-05-24T21:02:36.003 INFO:teuthology.orchestra.run.smithi157.stdout: "sync_status": { 2023-05-24T21:02:36.003 INFO:teuthology.orchestra.run.smithi157.stdout: "info": { 2023-05-24T21:02:36.003 INFO:teuthology.orchestra.run.smithi157.stdout: "status": "sync", 2023-05-24T21:02:36.003 INFO:teuthology.orchestra.run.smithi157.stdout: "num_shards": 4, 2023-05-24T21:02:36.003 INFO:teuthology.orchestra.run.smithi157.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:02:36.003 INFO:teuthology.orchestra.run.smithi157.stdout: "realm_epoch": 2 2023-05-24T21:02:36.003 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:02:36.004 INFO:teuthology.orchestra.run.smithi157.stdout: "markers": [ 2023-05-24T21:02:36.004 INFO:teuthology.orchestra.run.smithi157.stdout: { 2023-05-24T21:02:36.004 INFO:teuthology.orchestra.run.smithi157.stdout: "key": 0, 2023-05-24T21:02:36.004 INFO:teuthology.orchestra.run.smithi157.stdout: "val": { 2023-05-24T21:02:36.004 INFO:teuthology.orchestra.run.smithi157.stdout: "state": 1, 2023-05-24T21:02:36.004 INFO:teuthology.orchestra.run.smithi157.stdout: "marker": "1_1684962037.098901_69.1", 2023-05-24T21:02:36.004 INFO:teuthology.orchestra.run.smithi157.stdout: "next_step_marker": "", 2023-05-24T21:02:36.005 INFO:teuthology.orchestra.run.smithi157.stdout: "total_entries": 1, 2023-05-24T21:02:36.005 INFO:teuthology.orchestra.run.smithi157.stdout: "pos": 0, 2023-05-24T21:02:36.005 INFO:teuthology.orchestra.run.smithi157.stdout: "timestamp": "2023-05-24T21:00:37.098901Z", 2023-05-24T21:02:36.005 INFO:teuthology.orchestra.run.smithi157.stdout: "realm_epoch": 2 2023-05-24T21:02:36.005 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:02:36.005 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:02:36.005 INFO:teuthology.orchestra.run.smithi157.stdout: { 2023-05-24T21:02:36.005 INFO:teuthology.orchestra.run.smithi157.stdout: "key": 1, 2023-05-24T21:02:36.006 INFO:teuthology.orchestra.run.smithi157.stdout: "val": { 2023-05-24T21:02:36.006 INFO:teuthology.orchestra.run.smithi157.stdout: "state": 1, 2023-05-24T21:02:36.006 INFO:teuthology.orchestra.run.smithi157.stdout: "marker": "", 2023-05-24T21:02:36.006 INFO:teuthology.orchestra.run.smithi157.stdout: "next_step_marker": "", 2023-05-24T21:02:36.006 INFO:teuthology.orchestra.run.smithi157.stdout: "total_entries": 0, 2023-05-24T21:02:36.006 INFO:teuthology.orchestra.run.smithi157.stdout: "pos": 0, 2023-05-24T21:02:36.006 INFO:teuthology.orchestra.run.smithi157.stdout: "timestamp": "0.000000", 2023-05-24T21:02:36.007 INFO:teuthology.orchestra.run.smithi157.stdout: "realm_epoch": 2 2023-05-24T21:02:36.007 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:02:36.007 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:02:36.007 INFO:teuthology.orchestra.run.smithi157.stdout: { 2023-05-24T21:02:36.007 INFO:teuthology.orchestra.run.smithi157.stdout: "key": 2, 2023-05-24T21:02:36.007 INFO:teuthology.orchestra.run.smithi157.stdout: "val": { 2023-05-24T21:02:36.007 INFO:teuthology.orchestra.run.smithi157.stdout: "state": 1, 2023-05-24T21:02:36.007 INFO:teuthology.orchestra.run.smithi157.stdout: "marker": "", 2023-05-24T21:02:36.008 INFO:teuthology.orchestra.run.smithi157.stdout: "next_step_marker": "", 2023-05-24T21:02:36.008 INFO:teuthology.orchestra.run.smithi157.stdout: "total_entries": 0, 2023-05-24T21:02:36.008 INFO:teuthology.orchestra.run.smithi157.stdout: "pos": 0, 2023-05-24T21:02:36.008 INFO:teuthology.orchestra.run.smithi157.stdout: "timestamp": "0.000000", 2023-05-24T21:02:36.008 INFO:teuthology.orchestra.run.smithi157.stdout: "realm_epoch": 2 2023-05-24T21:02:36.008 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:02:36.008 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:02:36.009 INFO:teuthology.orchestra.run.smithi157.stdout: { 2023-05-24T21:02:36.009 INFO:teuthology.orchestra.run.smithi157.stdout: "key": 3, 2023-05-24T21:02:36.009 INFO:teuthology.orchestra.run.smithi157.stdout: "val": { 2023-05-24T21:02:36.009 INFO:teuthology.orchestra.run.smithi157.stdout: "state": 1, 2023-05-24T21:02:36.009 INFO:teuthology.orchestra.run.smithi157.stdout: "marker": "", 2023-05-24T21:02:36.009 INFO:teuthology.orchestra.run.smithi157.stdout: "next_step_marker": "", 2023-05-24T21:02:36.009 INFO:teuthology.orchestra.run.smithi157.stdout: "total_entries": 0, 2023-05-24T21:02:36.009 INFO:teuthology.orchestra.run.smithi157.stdout: "pos": 0, 2023-05-24T21:02:36.010 INFO:teuthology.orchestra.run.smithi157.stdout: "timestamp": "0.000000", 2023-05-24T21:02:36.010 INFO:teuthology.orchestra.run.smithi157.stdout: "realm_epoch": 2 2023-05-24T21:02:36.010 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:02:36.010 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-05-24T21:02:36.010 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2023-05-24T21:02:36.010 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:02:36.010 INFO:teuthology.orchestra.run.smithi157.stdout: "full_sync": { 2023-05-24T21:02:36.010 INFO:teuthology.orchestra.run.smithi157.stdout: "total": 1, 2023-05-24T21:02:36.011 INFO:teuthology.orchestra.run.smithi157.stdout: "complete": 1 2023-05-24T21:02:36.011 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-05-24T21:02:36.011 INFO:teuthology.orchestra.run.smithi157.stdout: "current_time": "2023-05-24T21:02:35Z" 2023-05-24T21:02:36.011 INFO:teuthology.orchestra.run.smithi157.stdout:} 2023-05-24T21:02:36.012 INFO:rgw_multi.tests:finish meta checkpoint for zone=b2 2023-05-24T21:02:36.012 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on '/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/../src/test/rgw/rgw_multi' with args=['tests.py'] 2023-05-24T21:02:36.461 DEBUG:boto:Using access key provided by client. 2023-05-24T21:02:36.699 DEBUG:boto:Using secret key provided by client. 2023-05-24T21:02:36.700 DEBUG:boto:Using access key provided by client. 2023-05-24T21:02:36.719 DEBUG:boto:Using secret key provided by client. 2023-05-24T21:02:36.720 DEBUG:boto:Using access key provided by client. 2023-05-24T21:02:36.721 DEBUG:boto:Using secret key provided by client. 2023-05-24T21:02:36.721 DEBUG:boto:Using access key provided by client. 2023-05-24T21:02:36.722 DEBUG:boto:Using secret key provided by client. 2023-05-24T21:02:36.723 INFO:rgw_multi.tests:create bucket zone=a1 name=qskrps-1 2023-05-24T21:02:36.723 DEBUG:boto:path=/qskrps-1/ 2023-05-24T21:02:36.723 DEBUG:boto:auth_path=/qskrps-1/ 2023-05-24T21:02:36.724 DEBUG:boto:Method: PUT 2023-05-24T21:02:36.724 DEBUG:boto:Path: /qskrps-1/ 2023-05-24T21:02:36.724 DEBUG:boto:Data: 2023-05-24T21:02:36.725 DEBUG:boto:Headers: {} 2023-05-24T21:02:36.725 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:02:36.725 DEBUG:boto:Port: 8000 2023-05-24T21:02:36.726 DEBUG:boto:Params: {} 2023-05-24T21:02:36.726 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T21:02:36.727 DEBUG:boto:Token: None 2023-05-24T21:02:36.727 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:02:36 GMT /qskrps-1/ 2023-05-24T21:02:36.731 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:HZ2ezwwdbMQ23x08mPqf8qs2B4E= 2023-05-24T21:02:36.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:02:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:HZ2ezwwdbMQ23x08mPqf8qs2B4E=', 'Content-Length': '0'} 2023-05-24T21:02:39.751 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e19c22069c8ee2ee-00646e7b6c-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:02:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:39.751 INFO:rgw_multi.tests:create bucket zone=a2 name=qskrps-2 2023-05-24T21:02:39.752 DEBUG:boto:path=/qskrps-2/ 2023-05-24T21:02:39.752 DEBUG:boto:auth_path=/qskrps-2/ 2023-05-24T21:02:39.753 DEBUG:boto:Method: PUT 2023-05-24T21:02:39.753 DEBUG:boto:Path: /qskrps-2/ 2023-05-24T21:02:39.753 DEBUG:boto:Data: 2023-05-24T21:02:39.754 DEBUG:boto:Headers: {} 2023-05-24T21:02:39.754 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:02:39.754 DEBUG:boto:Port: 8001 2023-05-24T21:02:39.755 DEBUG:boto:Params: {} 2023-05-24T21:02:39.755 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-24T21:02:39.755 DEBUG:boto:Token: None 2023-05-24T21:02:39.756 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:02:39 GMT /qskrps-2/ 2023-05-24T21:02:39.756 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5FPJlMfzWtWUkMHJGmZBdiaA6uQ= 2023-05-24T21:02:39.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:02:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5FPJlMfzWtWUkMHJGmZBdiaA6uQ=', 'Content-Length': '0'} 2023-05-24T21:02:42.757 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009e5d9d72aea3ca3a-00646e7b6f-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:02:42 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:42.757 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:42.758 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:42.758 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:02:42.809 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:02:42.809 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:02:42.845 INFO:teuthology.orchestra.run.smithi071.stderr:No --period given, using current period=5c04936b-70ea-4415-b4eb-3aad3dc612e5 2023-05-24T21:02:42.863 INFO:teuthology.orchestra.run.smithi071.stdout:[ 2023-05-24T21:02:42.863 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:42.864 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962037.098901_69.1", 2023-05-24T21:02:42.864 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:00:37.098901Z" 2023-05-24T21:02:42.864 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:42.864 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:42.864 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962159.779551_45.1", 2023-05-24T21:02:42.864 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:02:39.779551Z" 2023-05-24T21:02:42.864 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:42.865 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:42.865 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:42.865 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:02:42.865 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:42.865 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:42.865 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:42.866 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:02:42.866 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:42.866 INFO:teuthology.orchestra.run.smithi071.stdout:] 2023-05-24T21:02:42.867 DEBUG:rgw_multi.tests:master meta markers={0: '1_1684962037.098901_69.1', 1: '1_1684962159.779551_45.1', 2: '', 3: ''} 2023-05-24T21:02:42.867 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-24T21:02:42.868 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:42.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:42.868 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:02:42.917 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:02:42.918 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:02:42.956 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:02:42.950+0000 7fd6f8ab6780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:02:42.957 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:02:42.951+0000 7fd6f8ab6780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:02:42.968 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:02:42.968 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:02:42.968 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:02:42.968 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:02:42.969 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:02:42.969 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:02:42.969 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:42.969 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:42.969 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:02:42.969 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:42.970 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:02:42.970 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:42.970 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:42.970 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962037.098901_69.1", 2023-05-24T21:02:42.970 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:42.970 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:02:42.970 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:42.971 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:00:37.098901Z", 2023-05-24T21:02:42.971 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:42.971 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:42.971 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:42.971 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:42.971 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:02:42.971 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:42.972 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:42.972 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962159.728412_38.1", 2023-05-24T21:02:42.972 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:42.972 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:42.973 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:42.973 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:02:39.728412Z", 2023-05-24T21:02:42.973 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:42.973 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:42.973 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:42.973 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:42.973 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:02:42.974 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:42.974 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:42.974 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:42.974 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:42.974 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:42.974 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:42.974 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:02:42.975 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:42.975 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:42.975 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:42.975 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:42.975 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:02:42.975 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:42.975 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:42.976 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:42.976 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:42.976 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:42.976 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:42.976 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:02:42.976 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:42.976 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:42.977 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:42.977 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:02:42.977 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:42.977 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:02:42.978 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:02:42.978 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:02:42.978 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:42.978 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:02:42Z" 2023-05-24T21:02:42.978 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:02:42.979 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962037.098901_69.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:00:37.098901Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962159.728412_38.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:02:39.728412Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:02:42Z" } 2023-05-24T21:02:42.980 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962037.098901_69.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:00:37.098901Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962159.728412_38.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:02:39.728412Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:02:42.980 DEBUG:rgw_multi.tests:log_status={0: '1_1684962037.098901_69.1', 1: '1_1684962159.779551_45.1', 2: '', 3: ''} 2023-05-24T21:02:42.980 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962037.098901_69.1', 1: '1_1684962159.728412_38.1', 2: '', 3: ''} 2023-05-24T21:02:42.981 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1684962159.779551_45.1 target=1_1684962159.728412_38.1 2023-05-24T21:02:47.981 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:47.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:47.983 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:02:48.040 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:02:48.040 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:02:48.084 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:02:48.078+0000 7fe6b15f7780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:02:48.085 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:02:48.079+0000 7fe6b15f7780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:02:48.106 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:02:48.107 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:02:48.107 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:02:48.107 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:02:48.108 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:02:48.108 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:02:48.108 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:48.108 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:48.108 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:02:48.108 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:48.109 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:02:48.109 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:48.109 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:48.109 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962037.098901_69.1", 2023-05-24T21:02:48.109 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:48.109 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:02:48.109 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:48.110 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:00:37.098901Z", 2023-05-24T21:02:48.110 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:48.110 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:48.110 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:48.110 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:48.110 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:02:48.110 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:48.111 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:48.111 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962159.779551_45.1", 2023-05-24T21:02:48.111 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:48.111 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:48.111 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:48.111 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:02:39.779551Z", 2023-05-24T21:02:48.111 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:48.112 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:48.112 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:48.112 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:48.112 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:02:48.112 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:48.112 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:48.113 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:48.113 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:48.113 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:48.113 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:48.113 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:02:48.113 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:48.113 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:48.114 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:48.114 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:48.114 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:02:48.114 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:48.114 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:48.114 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:48.114 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:48.115 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:48.115 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:48.115 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:02:48.115 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:48.115 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:48.115 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:48.116 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:02:48.116 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:48.116 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:02:48.116 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:02:48.116 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:02:48.116 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:48.116 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:02:48Z" 2023-05-24T21:02:48.117 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:02:48.118 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962037.098901_69.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:00:37.098901Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962159.779551_45.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:02:39.779551Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:02:48Z" } 2023-05-24T21:02:48.118 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962037.098901_69.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:00:37.098901Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962159.779551_45.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:02:39.779551Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:02:48.118 DEBUG:rgw_multi.tests:log_status={0: '1_1684962037.098901_69.1', 1: '1_1684962159.779551_45.1', 2: '', 3: ''} 2023-05-24T21:02:48.119 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962037.098901_69.1', 1: '1_1684962159.779551_45.1', 2: '', 3: ''} 2023-05-24T21:02:48.119 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-24T21:02:48.120 DEBUG:boto:path=/qskrps-1/ 2023-05-24T21:02:48.120 DEBUG:boto:auth_path=/qskrps-1/ 2023-05-24T21:02:48.120 DEBUG:boto:Method: HEAD 2023-05-24T21:02:48.121 DEBUG:boto:Path: /qskrps-1/ 2023-05-24T21:02:48.121 DEBUG:boto:Data: 2023-05-24T21:02:48.121 DEBUG:boto:Headers: {} 2023-05-24T21:02:48.122 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:02:48.122 DEBUG:boto:Port: 8000 2023-05-24T21:02:48.123 DEBUG:boto:Params: {} 2023-05-24T21:02:48.123 DEBUG:boto:Token: None 2023-05-24T21:02:48.124 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:02:48 GMT /qskrps-1/ 2023-05-24T21:02:48.124 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:B5wd87fkLFKE+jrUtWA3dxI5kZY= 2023-05-24T21:02:48.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:02:48 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:B5wd87fkLFKE+jrUtWA3dxI5kZY=', 'Content-Length': '0'} 2023-05-24T21:02:48.127 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008a7e05ab05567e07-00646e7b78-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:02:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:48.127 DEBUG:boto:path=/qskrps-2/ 2023-05-24T21:02:48.128 DEBUG:boto:auth_path=/qskrps-2/ 2023-05-24T21:02:48.128 DEBUG:boto:Method: HEAD 2023-05-24T21:02:48.129 DEBUG:boto:Path: /qskrps-2/ 2023-05-24T21:02:48.129 DEBUG:boto:Data: 2023-05-24T21:02:48.129 DEBUG:boto:Headers: {} 2023-05-24T21:02:48.130 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:02:48.130 DEBUG:boto:Port: 8000 2023-05-24T21:02:48.130 DEBUG:boto:Params: {} 2023-05-24T21:02:48.131 DEBUG:boto:Token: None 2023-05-24T21:02:48.131 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:02:48 GMT /qskrps-2/ 2023-05-24T21:02:48.132 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:wdojSUuWNU5rG4P2dRcB5HM9gJA= 2023-05-24T21:02:48.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:02:48 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wdojSUuWNU5rG4P2dRcB5HM9gJA=', 'Content-Length': '0'} 2023-05-24T21:02:48.134 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cb3d2b768cef94b6-00646e7b78-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:02:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:48.135 DEBUG:boto:path=/qskrps-1/ 2023-05-24T21:02:48.136 DEBUG:boto:auth_path=/qskrps-1/ 2023-05-24T21:02:48.137 DEBUG:boto:Method: HEAD 2023-05-24T21:02:48.138 DEBUG:boto:Path: /qskrps-1/ 2023-05-24T21:02:48.139 DEBUG:boto:Data: 2023-05-24T21:02:48.139 DEBUG:boto:Headers: {} 2023-05-24T21:02:48.140 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:02:48.140 DEBUG:boto:Port: 8001 2023-05-24T21:02:48.141 DEBUG:boto:Params: {} 2023-05-24T21:02:48.142 DEBUG:boto:Token: None 2023-05-24T21:02:48.142 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:02:48 GMT /qskrps-1/ 2023-05-24T21:02:48.143 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:B5wd87fkLFKE+jrUtWA3dxI5kZY= 2023-05-24T21:02:48.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:02:48 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:B5wd87fkLFKE+jrUtWA3dxI5kZY=', 'Content-Length': '0'} 2023-05-24T21:02:48.145 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000854dfbecf0776dc9-00646e7b78-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:02:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:48.146 DEBUG:boto:path=/qskrps-2/ 2023-05-24T21:02:48.146 DEBUG:boto:auth_path=/qskrps-2/ 2023-05-24T21:02:48.147 DEBUG:boto:Method: HEAD 2023-05-24T21:02:48.147 DEBUG:boto:Path: /qskrps-2/ 2023-05-24T21:02:48.147 DEBUG:boto:Data: 2023-05-24T21:02:48.148 DEBUG:boto:Headers: {} 2023-05-24T21:02:48.148 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:02:48.148 DEBUG:boto:Port: 8001 2023-05-24T21:02:48.149 DEBUG:boto:Params: {} 2023-05-24T21:02:48.149 DEBUG:boto:Token: None 2023-05-24T21:02:48.149 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:02:48 GMT /qskrps-2/ 2023-05-24T21:02:48.150 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:wdojSUuWNU5rG4P2dRcB5HM9gJA= 2023-05-24T21:02:48.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:02:48 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wdojSUuWNU5rG4P2dRcB5HM9gJA=', 'Content-Length': '0'} 2023-05-24T21:02:48.152 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a3f18aa17dc56c89-00646e7b78-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:02:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:48.153 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_create ... ok 2023-05-24T21:02:48.163 INFO:rgw_multi.tests:create bucket zone=a1 name=qskrps-3 2023-05-24T21:02:48.163 DEBUG:boto:path=/qskrps-3/ 2023-05-24T21:02:48.164 DEBUG:boto:auth_path=/qskrps-3/ 2023-05-24T21:02:48.164 DEBUG:boto:Method: PUT 2023-05-24T21:02:48.164 DEBUG:boto:Path: /qskrps-3/ 2023-05-24T21:02:48.165 DEBUG:boto:Data: 2023-05-24T21:02:48.165 DEBUG:boto:Headers: {} 2023-05-24T21:02:48.165 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:02:48.166 DEBUG:boto:Port: 8000 2023-05-24T21:02:48.166 DEBUG:boto:Params: {} 2023-05-24T21:02:48.166 DEBUG:boto:Token: None 2023-05-24T21:02:48.167 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:02:48 GMT /qskrps-3/ 2023-05-24T21:02:48.167 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:+7pPPb/HrUJ7KLJI0Z0spEOfd0I= 2023-05-24T21:02:48.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:02:48 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:+7pPPb/HrUJ7KLJI0Z0spEOfd0I=', 'Content-Length': '0'} 2023-05-24T21:02:48.193 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000075e89d071d99ba35-00646e7b78-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:02:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:48.195 INFO:rgw_multi.tests:create bucket zone=a2 name=qskrps-4 2023-05-24T21:02:48.195 DEBUG:boto:path=/qskrps-4/ 2023-05-24T21:02:48.196 DEBUG:boto:auth_path=/qskrps-4/ 2023-05-24T21:02:48.196 DEBUG:boto:Method: PUT 2023-05-24T21:02:48.197 DEBUG:boto:Path: /qskrps-4/ 2023-05-24T21:02:48.198 DEBUG:boto:Data: 2023-05-24T21:02:48.198 DEBUG:boto:Headers: {} 2023-05-24T21:02:48.199 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:02:48.199 DEBUG:boto:Port: 8001 2023-05-24T21:02:48.200 DEBUG:boto:Params: {} 2023-05-24T21:02:48.200 DEBUG:boto:Token: None 2023-05-24T21:02:48.200 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:02:48 GMT /qskrps-4/ 2023-05-24T21:02:48.201 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:SQQSB9kDMLO9tgYhUUSdIcEgZDU= 2023-05-24T21:02:48.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:02:48 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:SQQSB9kDMLO9tgYhUUSdIcEgZDU=', 'Content-Length': '0'} 2023-05-24T21:02:48.242 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004b4c5d7d57eb929c-00646e7b78-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:02:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:48.243 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:48.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:48.244 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:02:48.295 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:02:48.296 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:02:48.326 INFO:teuthology.orchestra.run.smithi071.stderr:No --period given, using current period=5c04936b-70ea-4415-b4eb-3aad3dc612e5 2023-05-24T21:02:48.345 INFO:teuthology.orchestra.run.smithi071.stdout:[ 2023-05-24T21:02:48.346 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:48.346 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962037.098901_69.1", 2023-05-24T21:02:48.346 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:00:37.098901Z" 2023-05-24T21:02:48.346 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:48.346 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:48.347 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962168.223563_53.1", 2023-05-24T21:02:48.347 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:02:48.223563Z" 2023-05-24T21:02:48.347 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:48.347 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:48.347 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:48.347 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:02:48.348 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:48.348 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:48.348 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:48.348 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:02:48.348 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:48.348 INFO:teuthology.orchestra.run.smithi071.stdout:] 2023-05-24T21:02:48.349 DEBUG:rgw_multi.tests:master meta markers={0: '1_1684962037.098901_69.1', 1: '1_1684962168.223563_53.1', 2: '', 3: ''} 2023-05-24T21:02:48.350 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-24T21:02:48.350 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:48.350 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:48.351 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:02:48.400 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:02:48.400 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:02:48.448 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:02:48.442+0000 7f52369f6780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:02:48.448 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:02:48.443+0000 7f52369f6780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:02:48.488 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:02:48.488 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:02:48.489 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:02:48.489 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:02:48.489 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:02:48.489 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:02:48.489 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:48.489 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:48.490 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:02:48.490 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:48.490 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:02:48.490 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:48.490 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:48.490 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962037.098901_69.1", 2023-05-24T21:02:48.490 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:48.491 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:02:48.491 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:48.491 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:00:37.098901Z", 2023-05-24T21:02:48.491 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:48.491 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:48.491 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:48.491 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:48.492 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:02:48.492 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:48.492 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:48.492 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962168.206814_50.1", 2023-05-24T21:02:48.492 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:48.492 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:48.493 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:48.493 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:02:48.206814Z", 2023-05-24T21:02:48.493 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:48.493 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:48.493 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:48.493 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:48.493 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:02:48.494 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:48.494 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:48.494 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:48.494 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:48.494 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:48.494 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:48.494 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:02:48.495 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:48.495 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:48.495 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:48.495 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:48.495 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:02:48.495 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:48.495 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:48.496 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:48.496 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:48.496 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:48.496 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:48.496 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:02:48.496 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:48.497 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:48.497 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:48.497 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:02:48.497 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:48.498 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:02:48.498 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:02:48.498 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:02:48.498 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:48.498 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:02:48Z" 2023-05-24T21:02:48.498 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:02:48.499 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962037.098901_69.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:00:37.098901Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962168.206814_50.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:02:48.206814Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:02:48Z" } 2023-05-24T21:02:48.500 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962037.098901_69.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:00:37.098901Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962168.206814_50.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:02:48.206814Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:02:48.500 DEBUG:rgw_multi.tests:log_status={0: '1_1684962037.098901_69.1', 1: '1_1684962168.223563_53.1', 2: '', 3: ''} 2023-05-24T21:02:48.501 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962037.098901_69.1', 1: '1_1684962168.206814_50.1', 2: '', 3: ''} 2023-05-24T21:02:48.501 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1684962168.223563_53.1 target=1_1684962168.206814_50.1 2023-05-24T21:02:53.502 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:53.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:53.503 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:02:53.560 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:02:53.561 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:02:53.601 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:02:53.595+0000 7ff50bb2e780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:02:53.602 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:02:53.596+0000 7ff50bb2e780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:02:53.614 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:02:53.614 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:02:53.614 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:02:53.614 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:02:53.615 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:02:53.615 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:02:53.615 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:53.615 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:53.615 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:02:53.615 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:53.615 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:02:53.616 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:53.616 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:53.616 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962037.098901_69.1", 2023-05-24T21:02:53.616 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:53.616 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:02:53.616 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:53.617 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:00:37.098901Z", 2023-05-24T21:02:53.617 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:53.617 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:53.617 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:53.617 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:53.617 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:02:53.617 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:53.618 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:53.618 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962168.223563_53.1", 2023-05-24T21:02:53.618 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:53.618 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:53.618 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:53.618 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:02:48.223563Z", 2023-05-24T21:02:53.618 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:53.619 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:53.619 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:53.619 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:53.619 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:02:53.619 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:53.619 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:53.619 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:53.620 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:53.620 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:53.620 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:53.620 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:02:53.620 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:53.620 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:53.620 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:53.621 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:53.621 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:02:53.621 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:53.621 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:53.621 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:53.621 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:53.621 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:53.622 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:53.622 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:02:53.622 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:53.622 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:53.622 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:53.622 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:02:53.622 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:53.623 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:02:53.623 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:02:53.623 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:02:53.623 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:53.623 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:02:53Z" 2023-05-24T21:02:53.623 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:02:53.624 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962037.098901_69.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:00:37.098901Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962168.223563_53.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:02:48.223563Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:02:53Z" } 2023-05-24T21:02:53.625 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962037.098901_69.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:00:37.098901Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962168.223563_53.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:02:48.223563Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:02:53.625 DEBUG:rgw_multi.tests:log_status={0: '1_1684962037.098901_69.1', 1: '1_1684962168.223563_53.1', 2: '', 3: ''} 2023-05-24T21:02:53.626 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962037.098901_69.1', 1: '1_1684962168.223563_53.1', 2: '', 3: ''} 2023-05-24T21:02:53.626 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-24T21:02:53.626 DEBUG:boto:path=/qskrps-3/ 2023-05-24T21:02:53.627 DEBUG:boto:auth_path=/qskrps-3/ 2023-05-24T21:02:53.627 DEBUG:boto:Method: HEAD 2023-05-24T21:02:53.627 DEBUG:boto:Path: /qskrps-3/ 2023-05-24T21:02:53.628 DEBUG:boto:Data: 2023-05-24T21:02:53.628 DEBUG:boto:Headers: {} 2023-05-24T21:02:53.628 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:02:53.629 DEBUG:boto:Port: 8000 2023-05-24T21:02:53.629 DEBUG:boto:Params: {} 2023-05-24T21:02:53.629 DEBUG:boto:Token: None 2023-05-24T21:02:53.630 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:02:53 GMT /qskrps-3/ 2023-05-24T21:02:53.630 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:yjDz5izfWz/Nh92BT3ZMSWzlLA8= 2023-05-24T21:02:53.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:02:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:yjDz5izfWz/Nh92BT3ZMSWzlLA8=', 'Content-Length': '0'} 2023-05-24T21:02:53.633 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004f6909b2b4547d25-00646e7b7d-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:02:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:53.634 DEBUG:boto:path=/qskrps-4/ 2023-05-24T21:02:53.634 DEBUG:boto:auth_path=/qskrps-4/ 2023-05-24T21:02:53.634 DEBUG:boto:Method: HEAD 2023-05-24T21:02:53.635 DEBUG:boto:Path: /qskrps-4/ 2023-05-24T21:02:53.635 DEBUG:boto:Data: 2023-05-24T21:02:53.635 DEBUG:boto:Headers: {} 2023-05-24T21:02:53.636 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:02:53.636 DEBUG:boto:Port: 8000 2023-05-24T21:02:53.636 DEBUG:boto:Params: {} 2023-05-24T21:02:53.637 DEBUG:boto:Token: None 2023-05-24T21:02:53.637 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:02:53 GMT /qskrps-4/ 2023-05-24T21:02:53.638 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:qKNu1rA/J//sB9xbQ2byDJi4sWA= 2023-05-24T21:02:53.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:02:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qKNu1rA/J//sB9xbQ2byDJi4sWA=', 'Content-Length': '0'} 2023-05-24T21:02:53.640 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d6df41c3db33bb8c-00646e7b7d-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:02:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:53.641 DEBUG:boto:path=/qskrps-3/ 2023-05-24T21:02:53.641 DEBUG:boto:auth_path=/qskrps-3/ 2023-05-24T21:02:53.641 DEBUG:boto:Method: HEAD 2023-05-24T21:02:53.642 DEBUG:boto:Path: /qskrps-3/ 2023-05-24T21:02:53.642 DEBUG:boto:Data: 2023-05-24T21:02:53.643 DEBUG:boto:Headers: {} 2023-05-24T21:02:53.643 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:02:53.643 DEBUG:boto:Port: 8001 2023-05-24T21:02:53.643 DEBUG:boto:Params: {} 2023-05-24T21:02:53.644 DEBUG:boto:Token: None 2023-05-24T21:02:53.644 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:02:53 GMT /qskrps-3/ 2023-05-24T21:02:53.645 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:yjDz5izfWz/Nh92BT3ZMSWzlLA8= 2023-05-24T21:02:53.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:02:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:yjDz5izfWz/Nh92BT3ZMSWzlLA8=', 'Content-Length': '0'} 2023-05-24T21:02:53.647 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000abea8782b525b0c0-00646e7b7d-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:02:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:53.648 DEBUG:boto:path=/qskrps-4/ 2023-05-24T21:02:53.648 DEBUG:boto:auth_path=/qskrps-4/ 2023-05-24T21:02:53.649 DEBUG:boto:Method: HEAD 2023-05-24T21:02:53.649 DEBUG:boto:Path: /qskrps-4/ 2023-05-24T21:02:53.649 DEBUG:boto:Data: 2023-05-24T21:02:53.650 DEBUG:boto:Headers: {} 2023-05-24T21:02:53.650 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:02:53.650 DEBUG:boto:Port: 8001 2023-05-24T21:02:53.651 DEBUG:boto:Params: {} 2023-05-24T21:02:53.651 DEBUG:boto:Token: None 2023-05-24T21:02:53.651 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:02:53 GMT /qskrps-4/ 2023-05-24T21:02:53.652 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:qKNu1rA/J//sB9xbQ2byDJi4sWA= 2023-05-24T21:02:53.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:02:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qKNu1rA/J//sB9xbQ2byDJi4sWA=', 'Content-Length': '0'} 2023-05-24T21:02:53.654 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b474f73684531f8b-00646e7b7d-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:02:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:53.655 DEBUG:boto:path=/qskrps-3/ 2023-05-24T21:02:53.655 DEBUG:boto:auth_path=/qskrps-3/ 2023-05-24T21:02:53.655 DEBUG:boto:Method: PUT 2023-05-24T21:02:53.656 DEBUG:boto:Path: /qskrps-3/ 2023-05-24T21:02:53.656 DEBUG:boto:Data: 2023-05-24T21:02:53.656 DEBUG:boto:Headers: {} 2023-05-24T21:02:53.657 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:02:53.657 DEBUG:boto:Port: 8000 2023-05-24T21:02:53.657 DEBUG:boto:Params: {} 2023-05-24T21:02:53.658 DEBUG:boto:Token: None 2023-05-24T21:02:53.658 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:02:53 GMT /qskrps-3/ 2023-05-24T21:02:53.658 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:lgnAxHcy+Ins+ZIKUK9i1DaR/Mw= 2023-05-24T21:02:53.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:02:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:lgnAxHcy+Ins+ZIKUK9i1DaR/Mw=', 'Content-Length': '0'} 2023-05-24T21:02:53.661 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d42f96852fcb3d3a-00646e7b7d-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:02:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:53.662 DEBUG:boto:path=/qskrps-4/ 2023-05-24T21:02:53.662 DEBUG:boto:auth_path=/qskrps-4/ 2023-05-24T21:02:53.662 DEBUG:boto:Method: PUT 2023-05-24T21:02:53.663 DEBUG:boto:Path: /qskrps-4/ 2023-05-24T21:02:53.663 DEBUG:boto:Data: 2023-05-24T21:02:53.663 DEBUG:boto:Headers: {} 2023-05-24T21:02:53.664 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:02:53.664 DEBUG:boto:Port: 8000 2023-05-24T21:02:53.664 DEBUG:boto:Params: {} 2023-05-24T21:02:53.665 DEBUG:boto:Token: None 2023-05-24T21:02:53.665 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:02:53 GMT /qskrps-4/ 2023-05-24T21:02:53.665 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:U+p/togehtyy3dfCbS9gp3lKwOs= 2023-05-24T21:02:53.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:02:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:U+p/togehtyy3dfCbS9gp3lKwOs=', 'Content-Length': '0'} 2023-05-24T21:02:53.668 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005aa4e4ec94d0956f-00646e7b7d-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:02:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:53.669 DEBUG:boto:path=/qskrps-3/ 2023-05-24T21:02:53.669 DEBUG:boto:auth_path=/qskrps-3/ 2023-05-24T21:02:53.669 DEBUG:boto:Method: PUT 2023-05-24T21:02:53.670 DEBUG:boto:Path: /qskrps-3/ 2023-05-24T21:02:53.670 DEBUG:boto:Data: 2023-05-24T21:02:53.670 DEBUG:boto:Headers: {} 2023-05-24T21:02:53.671 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:02:53.671 DEBUG:boto:Port: 8001 2023-05-24T21:02:53.671 DEBUG:boto:Params: {} 2023-05-24T21:02:53.672 DEBUG:boto:Token: None 2023-05-24T21:02:53.672 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:02:53 GMT /qskrps-3/ 2023-05-24T21:02:53.672 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:lgnAxHcy+Ins+ZIKUK9i1DaR/Mw= 2023-05-24T21:02:53.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:02:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:lgnAxHcy+Ins+ZIKUK9i1DaR/Mw=', 'Content-Length': '0'} 2023-05-24T21:02:53.678 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006cb502ff8cdd541a-00646e7b7d-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:02:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:53.678 DEBUG:boto:path=/qskrps-4/ 2023-05-24T21:02:53.678 DEBUG:boto:auth_path=/qskrps-4/ 2023-05-24T21:02:53.679 DEBUG:boto:Method: PUT 2023-05-24T21:02:53.679 DEBUG:boto:Path: /qskrps-4/ 2023-05-24T21:02:53.679 DEBUG:boto:Data: 2023-05-24T21:02:53.680 DEBUG:boto:Headers: {} 2023-05-24T21:02:53.680 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:02:53.680 DEBUG:boto:Port: 8001 2023-05-24T21:02:53.681 DEBUG:boto:Params: {} 2023-05-24T21:02:53.681 DEBUG:boto:Token: None 2023-05-24T21:02:53.681 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:02:53 GMT /qskrps-4/ 2023-05-24T21:02:53.682 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:U+p/togehtyy3dfCbS9gp3lKwOs= 2023-05-24T21:02:53.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:02:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:U+p/togehtyy3dfCbS9gp3lKwOs=', 'Content-Length': '0'} 2023-05-24T21:02:53.728 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e52f285061465457-00646e7b7d-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:02:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:53.729 DEBUG:boto:path=/qskrps-3/ 2023-05-24T21:02:53.729 DEBUG:boto:auth_path=/qskrps-3/ 2023-05-24T21:02:53.729 DEBUG:boto:Method: HEAD 2023-05-24T21:02:53.730 DEBUG:boto:Path: /qskrps-3/ 2023-05-24T21:02:53.730 DEBUG:boto:Data: 2023-05-24T21:02:53.730 DEBUG:boto:Headers: {} 2023-05-24T21:02:53.731 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:02:53.731 DEBUG:boto:Port: 8000 2023-05-24T21:02:53.731 DEBUG:boto:Params: {} 2023-05-24T21:02:53.732 DEBUG:boto:Token: None 2023-05-24T21:02:53.732 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:02:53 GMT /qskrps-3/ 2023-05-24T21:02:53.733 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:yjDz5izfWz/Nh92BT3ZMSWzlLA8= 2023-05-24T21:02:53.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:02:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:yjDz5izfWz/Nh92BT3ZMSWzlLA8=', 'Content-Length': '0'} 2023-05-24T21:02:53.735 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ea5f3f3afce73a98-00646e7b7d-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:02:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:53.735 DEBUG:boto:path=/qskrps-4/ 2023-05-24T21:02:53.736 DEBUG:boto:auth_path=/qskrps-4/ 2023-05-24T21:02:53.736 DEBUG:boto:Method: HEAD 2023-05-24T21:02:53.737 DEBUG:boto:Path: /qskrps-4/ 2023-05-24T21:02:53.737 DEBUG:boto:Data: 2023-05-24T21:02:53.737 DEBUG:boto:Headers: {} 2023-05-24T21:02:53.738 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:02:53.738 DEBUG:boto:Port: 8000 2023-05-24T21:02:53.738 DEBUG:boto:Params: {} 2023-05-24T21:02:53.739 DEBUG:boto:Token: None 2023-05-24T21:02:53.739 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:02:53 GMT /qskrps-4/ 2023-05-24T21:02:53.739 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:qKNu1rA/J//sB9xbQ2byDJi4sWA= 2023-05-24T21:02:53.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:02:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qKNu1rA/J//sB9xbQ2byDJi4sWA=', 'Content-Length': '0'} 2023-05-24T21:02:53.742 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000088d6953f5c68b91c-00646e7b7d-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:02:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:53.742 DEBUG:boto:path=/qskrps-3/ 2023-05-24T21:02:53.743 DEBUG:boto:auth_path=/qskrps-3/ 2023-05-24T21:02:53.743 DEBUG:boto:Method: HEAD 2023-05-24T21:02:53.743 DEBUG:boto:Path: /qskrps-3/ 2023-05-24T21:02:53.744 DEBUG:boto:Data: 2023-05-24T21:02:53.744 DEBUG:boto:Headers: {} 2023-05-24T21:02:53.744 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:02:53.745 DEBUG:boto:Port: 8001 2023-05-24T21:02:53.745 DEBUG:boto:Params: {} 2023-05-24T21:02:53.745 DEBUG:boto:Token: None 2023-05-24T21:02:53.746 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:02:53 GMT /qskrps-3/ 2023-05-24T21:02:53.746 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:yjDz5izfWz/Nh92BT3ZMSWzlLA8= 2023-05-24T21:02:53.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:02:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:yjDz5izfWz/Nh92BT3ZMSWzlLA8=', 'Content-Length': '0'} 2023-05-24T21:02:53.749 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b4a249d142b25188-00646e7b7d-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:02:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:53.749 DEBUG:boto:path=/qskrps-4/ 2023-05-24T21:02:53.749 DEBUG:boto:auth_path=/qskrps-4/ 2023-05-24T21:02:53.750 DEBUG:boto:Method: HEAD 2023-05-24T21:02:53.750 DEBUG:boto:Path: /qskrps-4/ 2023-05-24T21:02:53.751 DEBUG:boto:Data: 2023-05-24T21:02:53.751 DEBUG:boto:Headers: {} 2023-05-24T21:02:53.751 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:02:53.752 DEBUG:boto:Port: 8001 2023-05-24T21:02:53.752 DEBUG:boto:Params: {} 2023-05-24T21:02:53.752 DEBUG:boto:Token: None 2023-05-24T21:02:53.753 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:02:53 GMT /qskrps-4/ 2023-05-24T21:02:53.753 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:qKNu1rA/J//sB9xbQ2byDJi4sWA= 2023-05-24T21:02:53.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:02:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qKNu1rA/J//sB9xbQ2byDJi4sWA=', 'Content-Length': '0'} 2023-05-24T21:02:53.755 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b800cc9618eeaf49-00646e7b7d-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:02:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:53.756 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:53.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:53.757 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:02:53.805 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:02:53.806 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:02:53.837 INFO:teuthology.orchestra.run.smithi071.stderr:No --period given, using current period=5c04936b-70ea-4415-b4eb-3aad3dc612e5 2023-05-24T21:02:53.861 INFO:teuthology.orchestra.run.smithi071.stdout:[ 2023-05-24T21:02:53.862 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:53.862 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962037.098901_69.1", 2023-05-24T21:02:53.862 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:00:37.098901Z" 2023-05-24T21:02:53.862 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:53.862 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:53.863 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962168.223563_53.1", 2023-05-24T21:02:53.863 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:02:48.223563Z" 2023-05-24T21:02:53.863 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:53.863 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:53.863 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:53.863 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:02:53.863 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:53.864 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:53.864 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:53.864 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:02:53.864 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:53.864 INFO:teuthology.orchestra.run.smithi071.stdout:] 2023-05-24T21:02:53.865 DEBUG:rgw_multi.tests:master meta markers={0: '1_1684962037.098901_69.1', 1: '1_1684962168.223563_53.1', 2: '', 3: ''} 2023-05-24T21:02:53.866 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-24T21:02:53.866 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:53.866 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:53.867 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:02:53.916 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:02:53.917 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:02:53.968 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:02:53.962+0000 7f3771c7c780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:02:53.968 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:02:53.963+0000 7f3771c7c780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:02:53.983 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:02:53.983 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:02:53.983 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:02:53.984 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:02:53.984 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:02:53.984 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:02:53.984 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:53.984 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:53.984 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:02:53.985 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:53.985 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:02:53.985 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:53.985 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:53.985 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962037.098901_69.1", 2023-05-24T21:02:53.985 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:53.986 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:02:53.986 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:53.986 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:00:37.098901Z", 2023-05-24T21:02:53.986 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:53.986 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:53.986 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:53.987 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:53.987 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:02:53.987 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:53.987 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:53.987 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962168.223563_53.1", 2023-05-24T21:02:53.987 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:53.987 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:53.988 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:53.988 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:02:48.223563Z", 2023-05-24T21:02:53.988 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:53.988 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:53.988 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:53.988 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:53.989 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:02:53.989 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:53.989 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:53.989 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:53.989 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:53.989 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:53.989 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:53.990 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:02:53.990 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:53.990 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:53.990 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:53.990 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:53.990 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:02:53.990 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:53.991 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:53.991 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:53.991 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:53.991 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:53.991 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:53.991 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:02:53.991 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:53.992 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:53.992 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:53.992 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:02:53.992 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:53.992 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:02:53.992 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:02:53.992 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:02:53.993 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:53.993 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:02:53Z" 2023-05-24T21:02:53.993 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:02:53.994 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962037.098901_69.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:00:37.098901Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962168.223563_53.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:02:48.223563Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:02:53Z" } 2023-05-24T21:02:53.994 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962037.098901_69.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:00:37.098901Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962168.223563_53.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:02:48.223563Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:02:53.995 DEBUG:rgw_multi.tests:log_status={0: '1_1684962037.098901_69.1', 1: '1_1684962168.223563_53.1', 2: '', 3: ''} 2023-05-24T21:02:53.995 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962037.098901_69.1', 1: '1_1684962168.223563_53.1', 2: '', 3: ''} 2023-05-24T21:02:53.995 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-24T21:02:53.996 DEBUG:boto:path=/qskrps-3/ 2023-05-24T21:02:53.996 DEBUG:boto:auth_path=/qskrps-3/ 2023-05-24T21:02:53.996 DEBUG:boto:Method: HEAD 2023-05-24T21:02:53.997 DEBUG:boto:Path: /qskrps-3/ 2023-05-24T21:02:53.997 DEBUG:boto:Data: 2023-05-24T21:02:53.998 DEBUG:boto:Headers: {} 2023-05-24T21:02:53.998 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:02:53.998 DEBUG:boto:Port: 8000 2023-05-24T21:02:53.999 DEBUG:boto:Params: {} 2023-05-24T21:02:53.999 DEBUG:boto:Token: None 2023-05-24T21:02:53.999 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:02:53 GMT /qskrps-3/ 2023-05-24T21:02:54.000 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:yjDz5izfWz/Nh92BT3ZMSWzlLA8= 2023-05-24T21:02:54.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:02:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:yjDz5izfWz/Nh92BT3ZMSWzlLA8=', 'Content-Length': '0'} 2023-05-24T21:02:54.002 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000097d50d0b69450f1b-00646e7b7d-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:02:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:54.003 DEBUG:boto:path=/qskrps-4/ 2023-05-24T21:02:54.003 DEBUG:boto:auth_path=/qskrps-4/ 2023-05-24T21:02:54.004 DEBUG:boto:Method: HEAD 2023-05-24T21:02:54.004 DEBUG:boto:Path: /qskrps-4/ 2023-05-24T21:02:54.004 DEBUG:boto:Data: 2023-05-24T21:02:54.005 DEBUG:boto:Headers: {} 2023-05-24T21:02:54.005 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:02:54.005 DEBUG:boto:Port: 8000 2023-05-24T21:02:54.006 DEBUG:boto:Params: {} 2023-05-24T21:02:54.006 DEBUG:boto:Token: None 2023-05-24T21:02:54.006 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:02:54 GMT /qskrps-4/ 2023-05-24T21:02:54.007 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:GucNc39lwZ64sJBHNnOxDShHKgg= 2023-05-24T21:02:54.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:02:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:GucNc39lwZ64sJBHNnOxDShHKgg=', 'Content-Length': '0'} 2023-05-24T21:02:54.009 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e3f2475bde902ee5-00646e7b7e-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:02:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:54.009 DEBUG:boto:path=/qskrps-3/ 2023-05-24T21:02:54.010 DEBUG:boto:auth_path=/qskrps-3/ 2023-05-24T21:02:54.010 DEBUG:boto:Method: HEAD 2023-05-24T21:02:54.010 DEBUG:boto:Path: /qskrps-3/ 2023-05-24T21:02:54.011 DEBUG:boto:Data: 2023-05-24T21:02:54.011 DEBUG:boto:Headers: {} 2023-05-24T21:02:54.011 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:02:54.012 DEBUG:boto:Port: 8001 2023-05-24T21:02:54.012 DEBUG:boto:Params: {} 2023-05-24T21:02:54.012 DEBUG:boto:Token: None 2023-05-24T21:02:54.013 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:02:54 GMT /qskrps-3/ 2023-05-24T21:02:54.013 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:I1YfzllOPSd6++ONU6Q37uEtWGc= 2023-05-24T21:02:54.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:02:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:I1YfzllOPSd6++ONU6Q37uEtWGc=', 'Content-Length': '0'} 2023-05-24T21:02:54.016 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cfafb32f717e7fc0-00646e7b7e-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:02:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:54.016 DEBUG:boto:path=/qskrps-4/ 2023-05-24T21:02:54.016 DEBUG:boto:auth_path=/qskrps-4/ 2023-05-24T21:02:54.017 DEBUG:boto:Method: HEAD 2023-05-24T21:02:54.017 DEBUG:boto:Path: /qskrps-4/ 2023-05-24T21:02:54.017 DEBUG:boto:Data: 2023-05-24T21:02:54.018 DEBUG:boto:Headers: {} 2023-05-24T21:02:54.018 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:02:54.018 DEBUG:boto:Port: 8001 2023-05-24T21:02:54.019 DEBUG:boto:Params: {} 2023-05-24T21:02:54.019 DEBUG:boto:Token: None 2023-05-24T21:02:54.020 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:02:54 GMT /qskrps-4/ 2023-05-24T21:02:54.020 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:GucNc39lwZ64sJBHNnOxDShHKgg= 2023-05-24T21:02:54.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:02:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:GucNc39lwZ64sJBHNnOxDShHKgg=', 'Content-Length': '0'} 2023-05-24T21:02:54.022 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e4bc2feceaef9c9e-00646e7b7e-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:02:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:54.023 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_recreate ... ok 2023-05-24T21:02:54.024 INFO:rgw_multi.tests:create bucket zone=a1 name=qskrps-5 2023-05-24T21:02:54.024 DEBUG:boto:path=/qskrps-5/ 2023-05-24T21:02:54.025 DEBUG:boto:auth_path=/qskrps-5/ 2023-05-24T21:02:54.025 DEBUG:boto:Method: PUT 2023-05-24T21:02:54.025 DEBUG:boto:Path: /qskrps-5/ 2023-05-24T21:02:54.026 DEBUG:boto:Data: 2023-05-24T21:02:54.026 DEBUG:boto:Headers: {} 2023-05-24T21:02:54.026 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:02:54.026 DEBUG:boto:Port: 8000 2023-05-24T21:02:54.027 DEBUG:boto:Params: {} 2023-05-24T21:02:54.027 DEBUG:boto:Token: None 2023-05-24T21:02:54.028 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:02:54 GMT /qskrps-5/ 2023-05-24T21:02:54.028 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:+b58xd5C7TTDzbCIpv8r539tKK4= 2023-05-24T21:02:54.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:02:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:+b58xd5C7TTDzbCIpv8r539tKK4=', 'Content-Length': '0'} 2023-05-24T21:02:54.040 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e111ab6e19d890cb-00646e7b7e-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:02:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:54.041 INFO:rgw_multi.tests:create bucket zone=a2 name=qskrps-6 2023-05-24T21:02:54.042 DEBUG:boto:path=/qskrps-6/ 2023-05-24T21:02:54.043 DEBUG:boto:auth_path=/qskrps-6/ 2023-05-24T21:02:54.043 DEBUG:boto:Method: PUT 2023-05-24T21:02:54.044 DEBUG:boto:Path: /qskrps-6/ 2023-05-24T21:02:54.044 DEBUG:boto:Data: 2023-05-24T21:02:54.044 DEBUG:boto:Headers: {} 2023-05-24T21:02:54.045 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:02:54.045 DEBUG:boto:Port: 8001 2023-05-24T21:02:54.046 DEBUG:boto:Params: {} 2023-05-24T21:02:54.046 DEBUG:boto:Token: None 2023-05-24T21:02:54.047 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:02:54 GMT /qskrps-6/ 2023-05-24T21:02:54.047 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:aPg17zRK4KnOUGtCnQC7yO5L3wg= 2023-05-24T21:02:54.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:02:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:aPg17zRK4KnOUGtCnQC7yO5L3wg=', 'Content-Length': '0'} 2023-05-24T21:02:54.077 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f3b3b6cce8818a5e-00646e7b7e-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:02:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:54.078 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:54.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:54.079 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:02:54.127 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:02:54.128 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:02:54.173 INFO:teuthology.orchestra.run.smithi071.stderr:No --period given, using current period=5c04936b-70ea-4415-b4eb-3aad3dc612e5 2023-05-24T21:02:54.191 INFO:teuthology.orchestra.run.smithi071.stdout:[ 2023-05-24T21:02:54.192 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:54.192 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962037.098901_69.1", 2023-05-24T21:02:54.192 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:00:37.098901Z" 2023-05-24T21:02:54.192 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:54.192 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:54.193 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962174.057718_61.1", 2023-05-24T21:02:54.193 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:02:54.057718Z" 2023-05-24T21:02:54.193 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:54.193 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:54.193 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:54.193 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:02:54.193 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:54.194 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:54.194 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:54.194 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:02:54.194 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:54.194 INFO:teuthology.orchestra.run.smithi071.stdout:] 2023-05-24T21:02:54.195 DEBUG:rgw_multi.tests:master meta markers={0: '1_1684962037.098901_69.1', 1: '1_1684962174.057718_61.1', 2: '', 3: ''} 2023-05-24T21:02:54.196 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-24T21:02:54.196 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:54.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:54.197 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:02:54.249 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:02:54.250 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:02:54.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:02:54.298+0000 7f02c6be0780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:02:54.305 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:02:54.299+0000 7f02c6be0780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:02:54.324 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:02:54.324 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:02:54.325 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:02:54.325 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:02:54.325 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:02:54.325 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:02:54.325 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:54.325 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:54.326 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:02:54.326 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:54.326 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:02:54.326 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:54.326 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:54.326 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962037.098901_69.1", 2023-05-24T21:02:54.326 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:54.327 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:02:54.327 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:54.327 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:00:37.098901Z", 2023-05-24T21:02:54.327 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:54.327 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:54.327 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:54.327 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:54.328 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:02:54.328 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:54.328 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:54.328 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962174.057718_61.1", 2023-05-24T21:02:54.328 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:54.328 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:54.329 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:54.329 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:02:54.057718Z", 2023-05-24T21:02:54.329 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:54.329 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:54.329 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:54.329 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:54.329 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:02:54.330 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:54.330 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:54.330 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:54.330 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:54.330 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:54.330 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:54.330 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:02:54.331 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:54.331 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:54.331 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:54.331 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:54.331 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:02:54.331 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:54.331 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:54.332 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:54.332 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:54.332 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:54.332 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:54.332 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:02:54.332 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:54.332 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:54.333 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:54.333 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:02:54.333 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:54.333 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:02:54.333 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:02:54.333 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:02:54.333 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:54.334 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:02:54Z" 2023-05-24T21:02:54.334 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:02:54.335 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962037.098901_69.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:00:37.098901Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962174.057718_61.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:02:54.057718Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:02:54Z" } 2023-05-24T21:02:54.336 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962037.098901_69.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:00:37.098901Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962174.057718_61.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:02:54.057718Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:02:54.336 DEBUG:rgw_multi.tests:log_status={0: '1_1684962037.098901_69.1', 1: '1_1684962174.057718_61.1', 2: '', 3: ''} 2023-05-24T21:02:54.337 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962037.098901_69.1', 1: '1_1684962174.057718_61.1', 2: '', 3: ''} 2023-05-24T21:02:54.337 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-24T21:02:54.337 DEBUG:boto:path=/qskrps-5/ 2023-05-24T21:02:54.338 DEBUG:boto:auth_path=/qskrps-5/ 2023-05-24T21:02:54.339 DEBUG:boto:Method: HEAD 2023-05-24T21:02:54.339 DEBUG:boto:Path: /qskrps-5/ 2023-05-24T21:02:54.339 DEBUG:boto:Data: 2023-05-24T21:02:54.340 DEBUG:boto:Headers: {} 2023-05-24T21:02:54.340 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:02:54.341 DEBUG:boto:Port: 8000 2023-05-24T21:02:54.341 DEBUG:boto:Params: {} 2023-05-24T21:02:54.342 DEBUG:boto:Token: None 2023-05-24T21:02:54.342 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:02:54 GMT /qskrps-5/ 2023-05-24T21:02:54.343 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:QhkAsSI9tOaZQYR+flwXM3XFNVk= 2023-05-24T21:02:54.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:02:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:QhkAsSI9tOaZQYR+flwXM3XFNVk=', 'Content-Length': '0'} 2023-05-24T21:02:54.346 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a0c99b12a4a3fcf6-00646e7b7e-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:02:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:54.346 DEBUG:boto:path=/qskrps-6/ 2023-05-24T21:02:54.347 DEBUG:boto:auth_path=/qskrps-6/ 2023-05-24T21:02:54.347 DEBUG:boto:Method: HEAD 2023-05-24T21:02:54.348 DEBUG:boto:Path: /qskrps-6/ 2023-05-24T21:02:54.348 DEBUG:boto:Data: 2023-05-24T21:02:54.348 DEBUG:boto:Headers: {} 2023-05-24T21:02:54.349 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:02:54.349 DEBUG:boto:Port: 8000 2023-05-24T21:02:54.349 DEBUG:boto:Params: {} 2023-05-24T21:02:54.350 DEBUG:boto:Token: None 2023-05-24T21:02:54.350 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:02:54 GMT /qskrps-6/ 2023-05-24T21:02:54.351 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:3Qi8oeRskII0q2+2t6eE8p1DSdY= 2023-05-24T21:02:54.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:02:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:3Qi8oeRskII0q2+2t6eE8p1DSdY=', 'Content-Length': '0'} 2023-05-24T21:02:54.353 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b0aac946f66045ed-00646e7b7e-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:02:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:54.354 DEBUG:boto:path=/qskrps-5/ 2023-05-24T21:02:54.354 DEBUG:boto:auth_path=/qskrps-5/ 2023-05-24T21:02:54.355 DEBUG:boto:Method: HEAD 2023-05-24T21:02:54.355 DEBUG:boto:Path: /qskrps-5/ 2023-05-24T21:02:54.355 DEBUG:boto:Data: 2023-05-24T21:02:54.356 DEBUG:boto:Headers: {} 2023-05-24T21:02:54.356 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:02:54.356 DEBUG:boto:Port: 8001 2023-05-24T21:02:54.357 DEBUG:boto:Params: {} 2023-05-24T21:02:54.357 DEBUG:boto:Token: None 2023-05-24T21:02:54.357 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:02:54 GMT /qskrps-5/ 2023-05-24T21:02:54.358 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:QhkAsSI9tOaZQYR+flwXM3XFNVk= 2023-05-24T21:02:54.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:02:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:QhkAsSI9tOaZQYR+flwXM3XFNVk=', 'Content-Length': '0'} 2023-05-24T21:02:54.360 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003f4f940285bf161b-00646e7b7e-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:02:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:54.361 DEBUG:boto:path=/qskrps-6/ 2023-05-24T21:02:54.361 DEBUG:boto:auth_path=/qskrps-6/ 2023-05-24T21:02:54.362 DEBUG:boto:Method: HEAD 2023-05-24T21:02:54.362 DEBUG:boto:Path: /qskrps-6/ 2023-05-24T21:02:54.363 DEBUG:boto:Data: 2023-05-24T21:02:54.363 DEBUG:boto:Headers: {} 2023-05-24T21:02:54.364 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:02:54.364 DEBUG:boto:Port: 8001 2023-05-24T21:02:54.365 DEBUG:boto:Params: {} 2023-05-24T21:02:54.365 DEBUG:boto:Token: None 2023-05-24T21:02:54.366 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:02:54 GMT /qskrps-6/ 2023-05-24T21:02:54.366 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:3Qi8oeRskII0q2+2t6eE8p1DSdY= 2023-05-24T21:02:54.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:02:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:3Qi8oeRskII0q2+2t6eE8p1DSdY=', 'Content-Length': '0'} 2023-05-24T21:02:54.368 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000601c18ea28e2680-00646e7b7e-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:02:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:54.369 DEBUG:boto:path=/qskrps-5/ 2023-05-24T21:02:54.369 DEBUG:boto:auth_path=/qskrps-5/ 2023-05-24T21:02:54.369 DEBUG:boto:Method: DELETE 2023-05-24T21:02:54.370 DEBUG:boto:Path: /qskrps-5/ 2023-05-24T21:02:54.370 DEBUG:boto:Data: 2023-05-24T21:02:54.371 DEBUG:boto:Headers: {} 2023-05-24T21:02:54.371 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:02:54.371 DEBUG:boto:Port: 8000 2023-05-24T21:02:54.371 DEBUG:boto:Params: {} 2023-05-24T21:02:54.372 DEBUG:boto:Token: None 2023-05-24T21:02:54.372 DEBUG:boto:StringToSign: DELETE Wed, 24 May 2023 21:02:54 GMT /qskrps-5/ 2023-05-24T21:02:54.373 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:7Yeumgp4YlgwRM3bh0l+NA2OcJ4= 2023-05-24T21:02:54.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:02:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:7Yeumgp4YlgwRM3bh0l+NA2OcJ4=', 'Content-Length': '0'} 2023-05-24T21:02:54.399 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000da54744dd3f0f8fc-00646e7b7e-110d-a1'), ('Date', 'Wed, 24 May 2023 21:02:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:54.399 DEBUG:boto:path=/qskrps-6/ 2023-05-24T21:02:54.400 DEBUG:boto:auth_path=/qskrps-6/ 2023-05-24T21:02:54.400 DEBUG:boto:Method: DELETE 2023-05-24T21:02:54.401 DEBUG:boto:Path: /qskrps-6/ 2023-05-24T21:02:54.401 DEBUG:boto:Data: 2023-05-24T21:02:54.401 DEBUG:boto:Headers: {} 2023-05-24T21:02:54.402 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:02:54.402 DEBUG:boto:Port: 8001 2023-05-24T21:02:54.402 DEBUG:boto:Params: {} 2023-05-24T21:02:54.403 DEBUG:boto:Token: None 2023-05-24T21:02:54.403 DEBUG:boto:StringToSign: DELETE Wed, 24 May 2023 21:02:54 GMT /qskrps-6/ 2023-05-24T21:02:54.403 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:EGE73XFk3zy8ZBcfC+ddydf2Vyg= 2023-05-24T21:02:54.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:02:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:EGE73XFk3zy8ZBcfC+ddydf2Vyg=', 'Content-Length': '0'} 2023-05-24T21:02:54.466 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000008a97614b868e9f5-00646e7b7e-1111-a2'), ('Date', 'Wed, 24 May 2023 21:02:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:54.467 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:54.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:54.468 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:02:54.516 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:02:54.517 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:02:54.564 INFO:teuthology.orchestra.run.smithi071.stderr:No --period given, using current period=5c04936b-70ea-4415-b4eb-3aad3dc612e5 2023-05-24T21:02:54.583 INFO:teuthology.orchestra.run.smithi071.stdout:[ 2023-05-24T21:02:54.584 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:54.584 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962037.098901_69.1", 2023-05-24T21:02:54.584 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:00:37.098901Z" 2023-05-24T21:02:54.584 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:54.584 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:54.585 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962174.427430_69.1", 2023-05-24T21:02:54.585 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:02:54.427430Z" 2023-05-24T21:02:54.585 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:54.585 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:54.585 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:54.585 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:02:54.585 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:54.586 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:54.586 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:54.586 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:02:54.586 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:54.586 INFO:teuthology.orchestra.run.smithi071.stdout:] 2023-05-24T21:02:54.587 DEBUG:rgw_multi.tests:master meta markers={0: '1_1684962037.098901_69.1', 1: '1_1684962174.427430_69.1', 2: '', 3: ''} 2023-05-24T21:02:54.588 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-24T21:02:54.588 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:54.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:54.589 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:02:54.638 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:02:54.638 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:02:54.676 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:02:54.670+0000 7f6ac1706780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:02:54.677 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:02:54.671+0000 7f6ac1706780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:02:54.688 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:02:54.689 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:02:54.689 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:02:54.689 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:02:54.689 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:02:54.689 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:02:54.690 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:54.690 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:54.690 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:02:54.690 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:54.690 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:02:54.690 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:54.690 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:54.691 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962037.098901_69.1", 2023-05-24T21:02:54.691 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:54.691 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:02:54.691 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:54.691 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:00:37.098901Z", 2023-05-24T21:02:54.691 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:54.691 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:54.692 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:54.692 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:54.692 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:02:54.692 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:54.692 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:54.692 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962174.427430_69.1", 2023-05-24T21:02:54.693 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:54.693 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:54.693 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:54.693 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:02:54.427430Z", 2023-05-24T21:02:54.694 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:54.694 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:54.694 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:54.694 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:54.694 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:02:54.694 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:54.694 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:54.695 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:54.695 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:54.695 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:54.695 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:54.695 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:02:54.695 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:54.695 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:54.696 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:54.696 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:54.696 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:02:54.696 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:54.696 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:54.696 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:54.696 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:54.697 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:54.697 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:54.697 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:02:54.697 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:54.697 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:54.697 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:54.697 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:02:54.698 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:54.698 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:02:54.698 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:02:54.698 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:02:54.698 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:54.698 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:02:54Z" 2023-05-24T21:02:54.698 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:02:54.699 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962037.098901_69.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:00:37.098901Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962174.427430_69.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:02:54.427430Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:02:54Z" } 2023-05-24T21:02:54.700 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962037.098901_69.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:00:37.098901Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962174.427430_69.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:02:54.427430Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:02:54.700 DEBUG:rgw_multi.tests:log_status={0: '1_1684962037.098901_69.1', 1: '1_1684962174.427430_69.1', 2: '', 3: ''} 2023-05-24T21:02:54.701 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962037.098901_69.1', 1: '1_1684962174.427430_69.1', 2: '', 3: ''} 2023-05-24T21:02:54.701 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-24T21:02:54.701 DEBUG:boto:path=/qskrps-5/ 2023-05-24T21:02:54.702 DEBUG:boto:auth_path=/qskrps-5/ 2023-05-24T21:02:54.702 DEBUG:boto:Method: HEAD 2023-05-24T21:02:54.702 DEBUG:boto:Path: /qskrps-5/ 2023-05-24T21:02:54.703 DEBUG:boto:Data: 2023-05-24T21:02:54.703 DEBUG:boto:Headers: {} 2023-05-24T21:02:54.703 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:02:54.704 DEBUG:boto:Port: 8000 2023-05-24T21:02:54.704 DEBUG:boto:Params: {} 2023-05-24T21:02:54.704 DEBUG:boto:Token: None 2023-05-24T21:02:54.705 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:02:54 GMT /qskrps-5/ 2023-05-24T21:02:54.705 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:QhkAsSI9tOaZQYR+flwXM3XFNVk= 2023-05-24T21:02:54.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:02:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:QhkAsSI9tOaZQYR+flwXM3XFNVk=', 'Content-Length': '0'} 2023-05-24T21:02:54.707 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx00000f426d439f1d6d356-00646e7b7e-110d-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:02:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:54.707 DEBUG:boto:path=/qskrps-6/ 2023-05-24T21:02:54.708 DEBUG:boto:auth_path=/qskrps-6/ 2023-05-24T21:02:54.708 DEBUG:boto:Method: HEAD 2023-05-24T21:02:54.708 DEBUG:boto:Path: /qskrps-6/ 2023-05-24T21:02:54.709 DEBUG:boto:Data: 2023-05-24T21:02:54.709 DEBUG:boto:Headers: {} 2023-05-24T21:02:54.709 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:02:54.710 DEBUG:boto:Port: 8000 2023-05-24T21:02:54.710 DEBUG:boto:Params: {} 2023-05-24T21:02:54.710 DEBUG:boto:Token: None 2023-05-24T21:02:54.711 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:02:54 GMT /qskrps-6/ 2023-05-24T21:02:54.711 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:3Qi8oeRskII0q2+2t6eE8p1DSdY= 2023-05-24T21:02:54.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:02:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:3Qi8oeRskII0q2+2t6eE8p1DSdY=', 'Content-Length': '0'} 2023-05-24T21:02:54.713 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx000004dc330bd5c5e65a3-00646e7b7e-110d-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:02:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:54.713 DEBUG:boto:path=/qskrps-5/ 2023-05-24T21:02:54.714 DEBUG:boto:auth_path=/qskrps-5/ 2023-05-24T21:02:54.714 DEBUG:boto:Method: HEAD 2023-05-24T21:02:54.714 DEBUG:boto:Path: /qskrps-5/ 2023-05-24T21:02:54.715 DEBUG:boto:Data: 2023-05-24T21:02:54.715 DEBUG:boto:Headers: {} 2023-05-24T21:02:54.715 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:02:54.716 DEBUG:boto:Port: 8001 2023-05-24T21:02:54.716 DEBUG:boto:Params: {} 2023-05-24T21:02:54.716 DEBUG:boto:Token: None 2023-05-24T21:02:54.717 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:02:54 GMT /qskrps-5/ 2023-05-24T21:02:54.717 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:QhkAsSI9tOaZQYR+flwXM3XFNVk= 2023-05-24T21:02:54.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:02:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:QhkAsSI9tOaZQYR+flwXM3XFNVk=', 'Content-Length': '0'} 2023-05-24T21:02:54.719 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx0000032c812760f486f7e-00646e7b7e-1111-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:02:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:54.719 DEBUG:boto:path=/qskrps-6/ 2023-05-24T21:02:54.720 DEBUG:boto:auth_path=/qskrps-6/ 2023-05-24T21:02:54.720 DEBUG:boto:Method: HEAD 2023-05-24T21:02:54.720 DEBUG:boto:Path: /qskrps-6/ 2023-05-24T21:02:54.721 DEBUG:boto:Data: 2023-05-24T21:02:54.721 DEBUG:boto:Headers: {} 2023-05-24T21:02:54.722 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:02:54.722 DEBUG:boto:Port: 8001 2023-05-24T21:02:54.722 DEBUG:boto:Params: {} 2023-05-24T21:02:54.723 DEBUG:boto:Token: None 2023-05-24T21:02:54.723 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:02:54 GMT /qskrps-6/ 2023-05-24T21:02:54.723 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:3Qi8oeRskII0q2+2t6eE8p1DSdY= 2023-05-24T21:02:54.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:02:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:3Qi8oeRskII0q2+2t6eE8p1DSdY=', 'Content-Length': '0'} 2023-05-24T21:02:54.725 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx000007e156c9868c53727-00646e7b7e-1111-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:02:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:54.726 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_remove ... ok 2023-05-24T21:02:54.726 INFO:rgw_multi.tests:create bucket zone=a1 name=qskrps-7 2023-05-24T21:02:54.727 DEBUG:boto:path=/qskrps-7/ 2023-05-24T21:02:54.727 DEBUG:boto:auth_path=/qskrps-7/ 2023-05-24T21:02:54.727 DEBUG:boto:Method: PUT 2023-05-24T21:02:54.728 DEBUG:boto:Path: /qskrps-7/ 2023-05-24T21:02:54.728 DEBUG:boto:Data: 2023-05-24T21:02:54.728 DEBUG:boto:Headers: {} 2023-05-24T21:02:54.729 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:02:54.729 DEBUG:boto:Port: 8000 2023-05-24T21:02:54.729 DEBUG:boto:Params: {} 2023-05-24T21:02:54.730 DEBUG:boto:Token: None 2023-05-24T21:02:54.730 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:02:54 GMT /qskrps-7/ 2023-05-24T21:02:54.731 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:UQjD/lYWCwAKJe66pNmdwoA3nVY= 2023-05-24T21:02:54.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:02:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:UQjD/lYWCwAKJe66pNmdwoA3nVY=', 'Content-Length': '0'} 2023-05-24T21:02:54.749 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000087068045db86faa1-00646e7b7e-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:02:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:54.750 INFO:rgw_multi.tests:create bucket zone=a2 name=qskrps-8 2023-05-24T21:02:54.750 DEBUG:boto:path=/qskrps-8/ 2023-05-24T21:02:54.751 DEBUG:boto:auth_path=/qskrps-8/ 2023-05-24T21:02:54.751 DEBUG:boto:Method: PUT 2023-05-24T21:02:54.752 DEBUG:boto:Path: /qskrps-8/ 2023-05-24T21:02:54.752 DEBUG:boto:Data: 2023-05-24T21:02:54.752 DEBUG:boto:Headers: {} 2023-05-24T21:02:54.753 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:02:54.753 DEBUG:boto:Port: 8001 2023-05-24T21:02:54.753 DEBUG:boto:Params: {} 2023-05-24T21:02:54.754 DEBUG:boto:Token: None 2023-05-24T21:02:54.754 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:02:54 GMT /qskrps-8/ 2023-05-24T21:02:54.754 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:3CxXK2oJzm1ciJJNV6/2jUQP8k0= 2023-05-24T21:02:54.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:02:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:3CxXK2oJzm1ciJJNV6/2jUQP8k0=', 'Content-Length': '0'} 2023-05-24T21:02:54.789 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000dbec9be2a62daab1-00646e7b7e-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:02:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:54.790 DEBUG:boto:path=/qskrps-7/ 2023-05-24T21:02:54.791 DEBUG:boto:auth_path=/qskrps-7/ 2023-05-24T21:02:54.791 DEBUG:boto:Method: HEAD 2023-05-24T21:02:54.791 DEBUG:boto:Path: /qskrps-7/ 2023-05-24T21:02:54.792 DEBUG:boto:Data: 2023-05-24T21:02:54.792 DEBUG:boto:Headers: {} 2023-05-24T21:02:54.792 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:02:54.793 DEBUG:boto:Port: 8000 2023-05-24T21:02:54.793 DEBUG:boto:Params: {} 2023-05-24T21:02:54.793 DEBUG:boto:Token: None 2023-05-24T21:02:54.794 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:02:54 GMT /qskrps-7/ 2023-05-24T21:02:54.794 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Do6FuTMPvribGbI+6LaTDTEeGKw= 2023-05-24T21:02:54.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:02:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Do6FuTMPvribGbI+6LaTDTEeGKw=', 'Content-Length': '0'} 2023-05-24T21:02:54.797 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c15b56b1a1d8691-00646e7b7e-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:02:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:54.797 DEBUG:boto:path=/qskrps-7/myobj 2023-05-24T21:02:54.850 DEBUG:boto:auth_path=/qskrps-7/myobj 2023-05-24T21:02:54.851 DEBUG:boto:Method: PUT 2023-05-24T21:02:54.851 DEBUG:boto:Path: /qskrps-7/myobj 2023-05-24T21:02:54.851 DEBUG:boto:Data: 2023-05-24T21:02:54.852 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-24T21:02:54.852 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:02:54.852 DEBUG:boto:Port: 8000 2023-05-24T21:02:54.853 DEBUG:boto:Params: {} 2023-05-24T21:02:54.853 DEBUG:boto:Token: None 2023-05-24T21:02:54.854 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 24 May 2023 21:02:54 GMT /qskrps-7/myobj 2023-05-24T21:02:54.854 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:dAD7tNJWlIuMyw6jxpnHQaExc2s= 2023-05-24T21:02:54.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:02:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:dAD7tNJWlIuMyw6jxpnHQaExc2s='} 2023-05-24T21:02:54.888 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f354bc8fb2fffd5b-00646e7b7e-110d-a1'), ('Date', 'Wed, 24 May 2023 21:02:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:54.888 DEBUG:boto:path=/qskrps-7/ 2023-05-24T21:02:54.889 DEBUG:boto:auth_path=/qskrps-7/ 2023-05-24T21:02:54.889 DEBUG:boto:Method: HEAD 2023-05-24T21:02:54.890 DEBUG:boto:Path: /qskrps-7/ 2023-05-24T21:02:54.890 DEBUG:boto:Data: 2023-05-24T21:02:54.890 DEBUG:boto:Headers: {} 2023-05-24T21:02:54.891 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:02:54.891 DEBUG:boto:Port: 8000 2023-05-24T21:02:54.891 DEBUG:boto:Params: {} 2023-05-24T21:02:54.892 DEBUG:boto:Token: None 2023-05-24T21:02:54.892 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:02:54 GMT /qskrps-7/ 2023-05-24T21:02:54.893 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Do6FuTMPvribGbI+6LaTDTEeGKw= 2023-05-24T21:02:54.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:02:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Do6FuTMPvribGbI+6LaTDTEeGKw=', 'Content-Length': '0'} 2023-05-24T21:02:54.895 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e5e4d4a48cf962ff-00646e7b7e-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:02:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:54.896 DEBUG:boto:path=/qskrps-7/_myobj 2023-05-24T21:02:54.896 DEBUG:boto:auth_path=/qskrps-7/_myobj 2023-05-24T21:02:54.896 DEBUG:boto:Method: PUT 2023-05-24T21:02:54.897 DEBUG:boto:Path: /qskrps-7/_myobj 2023-05-24T21:02:54.897 DEBUG:boto:Data: 2023-05-24T21:02:54.898 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-24T21:02:54.898 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:02:54.898 DEBUG:boto:Port: 8000 2023-05-24T21:02:54.899 DEBUG:boto:Params: {} 2023-05-24T21:02:54.899 DEBUG:boto:Token: None 2023-05-24T21:02:54.900 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 24 May 2023 21:02:54 GMT /qskrps-7/_myobj 2023-05-24T21:02:54.900 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ziIm+qconxni5k9y/WfzYQKjeBs= 2023-05-24T21:02:54.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:02:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ziIm+qconxni5k9y/WfzYQKjeBs='} 2023-05-24T21:02:54.948 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006b05016b7f2e2197-00646e7b7e-110d-a1'), ('Date', 'Wed, 24 May 2023 21:02:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:54.949 DEBUG:boto:path=/qskrps-7/ 2023-05-24T21:02:54.949 DEBUG:boto:auth_path=/qskrps-7/ 2023-05-24T21:02:54.950 DEBUG:boto:Method: HEAD 2023-05-24T21:02:54.950 DEBUG:boto:Path: /qskrps-7/ 2023-05-24T21:02:54.950 DEBUG:boto:Data: 2023-05-24T21:02:54.951 DEBUG:boto:Headers: {} 2023-05-24T21:02:54.951 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:02:54.951 DEBUG:boto:Port: 8000 2023-05-24T21:02:54.952 DEBUG:boto:Params: {} 2023-05-24T21:02:54.952 DEBUG:boto:Token: None 2023-05-24T21:02:54.953 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:02:54 GMT /qskrps-7/ 2023-05-24T21:02:54.953 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Do6FuTMPvribGbI+6LaTDTEeGKw= 2023-05-24T21:02:54.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:02:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Do6FuTMPvribGbI+6LaTDTEeGKw=', 'Content-Length': '0'} 2023-05-24T21:02:54.955 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a8b78fd333ab463d-00646e7b7e-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:02:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:54.956 DEBUG:boto:path=/qskrps-7/%3A 2023-05-24T21:02:54.957 DEBUG:boto:auth_path=/qskrps-7/%3A 2023-05-24T21:02:54.957 DEBUG:boto:Method: PUT 2023-05-24T21:02:54.957 DEBUG:boto:Path: /qskrps-7/%3A 2023-05-24T21:02:54.958 DEBUG:boto:Data: 2023-05-24T21:02:54.958 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-24T21:02:54.958 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:02:54.959 DEBUG:boto:Port: 8000 2023-05-24T21:02:54.959 DEBUG:boto:Params: {} 2023-05-24T21:02:54.959 DEBUG:boto:Token: None 2023-05-24T21:02:54.960 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 24 May 2023 21:02:54 GMT /qskrps-7/%3A 2023-05-24T21:02:54.960 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:DN6H4Y6fnLEuPEGOSdmimrPQ5SM= 2023-05-24T21:02:54.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:02:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:DN6H4Y6fnLEuPEGOSdmimrPQ5SM='} 2023-05-24T21:02:54.974 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000814e9b949a58469a-00646e7b7e-110d-a1'), ('Date', 'Wed, 24 May 2023 21:02:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:54.975 DEBUG:boto:path=/qskrps-7/ 2023-05-24T21:02:54.976 DEBUG:boto:auth_path=/qskrps-7/ 2023-05-24T21:02:54.976 DEBUG:boto:Method: HEAD 2023-05-24T21:02:54.976 DEBUG:boto:Path: /qskrps-7/ 2023-05-24T21:02:54.977 DEBUG:boto:Data: 2023-05-24T21:02:54.977 DEBUG:boto:Headers: {} 2023-05-24T21:02:54.977 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:02:54.978 DEBUG:boto:Port: 8000 2023-05-24T21:02:54.978 DEBUG:boto:Params: {} 2023-05-24T21:02:54.979 DEBUG:boto:Token: None 2023-05-24T21:02:54.979 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:02:54 GMT /qskrps-7/ 2023-05-24T21:02:54.979 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Do6FuTMPvribGbI+6LaTDTEeGKw= 2023-05-24T21:02:54.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:02:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Do6FuTMPvribGbI+6LaTDTEeGKw=', 'Content-Length': '0'} 2023-05-24T21:02:54.982 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b5efeb8ddce688db-00646e7b7e-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:02:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:54.983 DEBUG:boto:path=/qskrps-7/%26 2023-05-24T21:02:54.983 DEBUG:boto:auth_path=/qskrps-7/%26 2023-05-24T21:02:54.983 DEBUG:boto:Method: PUT 2023-05-24T21:02:54.984 DEBUG:boto:Path: /qskrps-7/%26 2023-05-24T21:02:54.984 DEBUG:boto:Data: 2023-05-24T21:02:54.984 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-24T21:02:54.985 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:02:54.985 DEBUG:boto:Port: 8000 2023-05-24T21:02:54.985 DEBUG:boto:Params: {} 2023-05-24T21:02:54.986 DEBUG:boto:Token: None 2023-05-24T21:02:54.986 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 24 May 2023 21:02:54 GMT /qskrps-7/%26 2023-05-24T21:02:54.987 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:yKMiGQSeEgX3ousugKuyga/sjSg= 2023-05-24T21:02:54.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:02:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:yKMiGQSeEgX3ousugKuyga/sjSg='} 2023-05-24T21:02:54.995 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000095beda09f2c08a08-00646e7b7e-110d-a1'), ('Date', 'Wed, 24 May 2023 21:02:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:54.996 DEBUG:boto:path=/qskrps-8/ 2023-05-24T21:02:54.996 DEBUG:boto:auth_path=/qskrps-8/ 2023-05-24T21:02:54.997 DEBUG:boto:Method: HEAD 2023-05-24T21:02:54.997 DEBUG:boto:Path: /qskrps-8/ 2023-05-24T21:02:54.997 DEBUG:boto:Data: 2023-05-24T21:02:54.998 DEBUG:boto:Headers: {} 2023-05-24T21:02:54.998 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:02:54.999 DEBUG:boto:Port: 8001 2023-05-24T21:02:55.000 DEBUG:boto:Params: {} 2023-05-24T21:02:55.000 DEBUG:boto:Token: None 2023-05-24T21:02:55.001 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:02:55 GMT /qskrps-8/ 2023-05-24T21:02:55.002 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:QdFFZLHu0gHm55sNF+C/ZQ1Idus= 2023-05-24T21:02:55.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:02:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:QdFFZLHu0gHm55sNF+C/ZQ1Idus=', 'Content-Length': '0'} 2023-05-24T21:02:55.005 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d80951fdb7aa1b82-00646e7b7e-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:02:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:55.006 DEBUG:boto:path=/qskrps-8/myobj 2023-05-24T21:02:55.007 DEBUG:boto:auth_path=/qskrps-8/myobj 2023-05-24T21:02:55.007 DEBUG:boto:Method: PUT 2023-05-24T21:02:55.008 DEBUG:boto:Path: /qskrps-8/myobj 2023-05-24T21:02:55.009 DEBUG:boto:Data: 2023-05-24T21:02:55.009 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-24T21:02:55.010 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:02:55.010 DEBUG:boto:Port: 8001 2023-05-24T21:02:55.011 DEBUG:boto:Params: {} 2023-05-24T21:02:55.012 DEBUG:boto:Token: None 2023-05-24T21:02:55.013 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 24 May 2023 21:02:55 GMT /qskrps-8/myobj 2023-05-24T21:02:55.014 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:n9R2E6QpobLQJHm1YqIW+s7tCVw= 2023-05-24T21:02:55.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:02:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:n9R2E6QpobLQJHm1YqIW+s7tCVw='} 2023-05-24T21:02:55.028 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fccd4fb5a327020e-00646e7b7f-1111-a2'), ('Date', 'Wed, 24 May 2023 21:02:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:55.029 DEBUG:boto:path=/qskrps-8/ 2023-05-24T21:02:55.029 DEBUG:boto:auth_path=/qskrps-8/ 2023-05-24T21:02:55.030 DEBUG:boto:Method: HEAD 2023-05-24T21:02:55.031 DEBUG:boto:Path: /qskrps-8/ 2023-05-24T21:02:55.031 DEBUG:boto:Data: 2023-05-24T21:02:55.032 DEBUG:boto:Headers: {} 2023-05-24T21:02:55.033 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:02:55.033 DEBUG:boto:Port: 8001 2023-05-24T21:02:55.034 DEBUG:boto:Params: {} 2023-05-24T21:02:55.035 DEBUG:boto:Token: None 2023-05-24T21:02:55.036 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:02:55 GMT /qskrps-8/ 2023-05-24T21:02:55.037 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:QdFFZLHu0gHm55sNF+C/ZQ1Idus= 2023-05-24T21:02:55.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:02:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:QdFFZLHu0gHm55sNF+C/ZQ1Idus=', 'Content-Length': '0'} 2023-05-24T21:02:55.040 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005d022c567f527a80-00646e7b7f-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:02:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:55.040 DEBUG:boto:path=/qskrps-8/_myobj 2023-05-24T21:02:55.041 DEBUG:boto:auth_path=/qskrps-8/_myobj 2023-05-24T21:02:55.042 DEBUG:boto:Method: PUT 2023-05-24T21:02:55.042 DEBUG:boto:Path: /qskrps-8/_myobj 2023-05-24T21:02:55.043 DEBUG:boto:Data: 2023-05-24T21:02:55.044 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-24T21:02:55.044 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:02:55.045 DEBUG:boto:Port: 8001 2023-05-24T21:02:55.046 DEBUG:boto:Params: {} 2023-05-24T21:02:55.046 DEBUG:boto:Token: None 2023-05-24T21:02:55.047 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 24 May 2023 21:02:55 GMT /qskrps-8/_myobj 2023-05-24T21:02:55.048 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:nkCVedKFjOQVXump+WjlkOaysKI= 2023-05-24T21:02:55.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:02:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:nkCVedKFjOQVXump+WjlkOaysKI='} 2023-05-24T21:02:55.066 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000054ab1f888a9b5466-00646e7b7f-1111-a2'), ('Date', 'Wed, 24 May 2023 21:02:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:55.067 DEBUG:boto:path=/qskrps-8/ 2023-05-24T21:02:55.067 DEBUG:boto:auth_path=/qskrps-8/ 2023-05-24T21:02:55.067 DEBUG:boto:Method: HEAD 2023-05-24T21:02:55.068 DEBUG:boto:Path: /qskrps-8/ 2023-05-24T21:02:55.068 DEBUG:boto:Data: 2023-05-24T21:02:55.068 DEBUG:boto:Headers: {} 2023-05-24T21:02:55.069 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:02:55.069 DEBUG:boto:Port: 8001 2023-05-24T21:02:55.069 DEBUG:boto:Params: {} 2023-05-24T21:02:55.070 DEBUG:boto:Token: None 2023-05-24T21:02:55.070 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:02:55 GMT /qskrps-8/ 2023-05-24T21:02:55.071 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:QdFFZLHu0gHm55sNF+C/ZQ1Idus= 2023-05-24T21:02:55.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:02:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:QdFFZLHu0gHm55sNF+C/ZQ1Idus=', 'Content-Length': '0'} 2023-05-24T21:02:55.073 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000024638377f10b114f-00646e7b7f-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:02:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:55.074 DEBUG:boto:path=/qskrps-8/%3A 2023-05-24T21:02:55.075 DEBUG:boto:auth_path=/qskrps-8/%3A 2023-05-24T21:02:55.075 DEBUG:boto:Method: PUT 2023-05-24T21:02:55.076 DEBUG:boto:Path: /qskrps-8/%3A 2023-05-24T21:02:55.076 DEBUG:boto:Data: 2023-05-24T21:02:55.077 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-24T21:02:55.077 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:02:55.077 DEBUG:boto:Port: 8001 2023-05-24T21:02:55.078 DEBUG:boto:Params: {} 2023-05-24T21:02:55.078 DEBUG:boto:Token: None 2023-05-24T21:02:55.079 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 24 May 2023 21:02:55 GMT /qskrps-8/%3A 2023-05-24T21:02:55.079 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:9iVpO+nx5sI/Ffwpkttk33ZUzRE= 2023-05-24T21:02:55.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:02:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:9iVpO+nx5sI/Ffwpkttk33ZUzRE='} 2023-05-24T21:02:55.124 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aac92bf44fa74ce6-00646e7b7f-1111-a2'), ('Date', 'Wed, 24 May 2023 21:02:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:55.125 DEBUG:boto:path=/qskrps-8/ 2023-05-24T21:02:55.126 DEBUG:boto:auth_path=/qskrps-8/ 2023-05-24T21:02:55.126 DEBUG:boto:Method: HEAD 2023-05-24T21:02:55.126 DEBUG:boto:Path: /qskrps-8/ 2023-05-24T21:02:55.127 DEBUG:boto:Data: 2023-05-24T21:02:55.127 DEBUG:boto:Headers: {} 2023-05-24T21:02:55.127 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:02:55.128 DEBUG:boto:Port: 8001 2023-05-24T21:02:55.128 DEBUG:boto:Params: {} 2023-05-24T21:02:55.128 DEBUG:boto:Token: None 2023-05-24T21:02:55.129 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:02:55 GMT /qskrps-8/ 2023-05-24T21:02:55.129 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:QdFFZLHu0gHm55sNF+C/ZQ1Idus= 2023-05-24T21:02:55.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:02:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:QdFFZLHu0gHm55sNF+C/ZQ1Idus=', 'Content-Length': '0'} 2023-05-24T21:02:55.132 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bcf546d9222a9662-00646e7b7f-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:02:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:55.132 DEBUG:boto:path=/qskrps-8/%26 2023-05-24T21:02:55.133 DEBUG:boto:auth_path=/qskrps-8/%26 2023-05-24T21:02:55.133 DEBUG:boto:Method: PUT 2023-05-24T21:02:55.134 DEBUG:boto:Path: /qskrps-8/%26 2023-05-24T21:02:55.134 DEBUG:boto:Data: 2023-05-24T21:02:55.134 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-24T21:02:55.135 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:02:55.135 DEBUG:boto:Port: 8001 2023-05-24T21:02:55.135 DEBUG:boto:Params: {} 2023-05-24T21:02:55.136 DEBUG:boto:Token: None 2023-05-24T21:02:55.136 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 24 May 2023 21:02:55 GMT /qskrps-8/%26 2023-05-24T21:02:55.136 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ypqFKHN5DRxY4zLjHhMKzlJ0iNY= 2023-05-24T21:02:55.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:02:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ypqFKHN5DRxY4zLjHhMKzlJ0iNY='} 2023-05-24T21:02:55.142 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000667b18e89d55de11-00646e7b7f-1111-a2'), ('Date', 'Wed, 24 May 2023 21:02:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:02:55.143 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:55.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:55.144 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:02:55.193 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:02:55.193 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:02:55.224 INFO:teuthology.orchestra.run.smithi071.stderr:No --period given, using current period=5c04936b-70ea-4415-b4eb-3aad3dc612e5 2023-05-24T21:02:55.242 INFO:teuthology.orchestra.run.smithi071.stdout:[ 2023-05-24T21:02:55.242 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:55.242 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962037.098901_69.1", 2023-05-24T21:02:55.243 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:00:37.098901Z" 2023-05-24T21:02:55.243 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:55.243 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:55.243 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962174.772148_77.1", 2023-05-24T21:02:55.243 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:02:54.772148Z" 2023-05-24T21:02:55.243 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:55.243 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:55.244 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:55.244 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:02:55.244 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:55.244 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:55.244 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:55.244 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:02:55.245 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:55.245 INFO:teuthology.orchestra.run.smithi071.stdout:] 2023-05-24T21:02:55.246 DEBUG:rgw_multi.tests:master meta markers={0: '1_1684962037.098901_69.1', 1: '1_1684962174.772148_77.1', 2: '', 3: ''} 2023-05-24T21:02:55.246 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-24T21:02:55.246 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:55.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:02:55.247 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:02:55.296 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:02:55.297 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:02:55.343 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:02:55.337+0000 7f3370de4780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:02:55.344 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:02:55.338+0000 7f3370de4780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:02:55.363 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:02:55.364 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:02:55.364 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:02:55.364 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:02:55.364 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:02:55.364 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:02:55.365 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:55.365 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:55.365 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:02:55.365 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:55.365 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:02:55.365 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:55.365 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:55.366 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962037.098901_69.1", 2023-05-24T21:02:55.366 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:55.366 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:02:55.366 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:55.366 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:00:37.098901Z", 2023-05-24T21:02:55.366 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:55.367 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:55.367 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:55.367 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:55.367 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:02:55.367 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:55.367 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:55.367 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962174.772148_77.1", 2023-05-24T21:02:55.368 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:55.368 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:55.368 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:55.368 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:02:54.772148Z", 2023-05-24T21:02:55.368 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:55.368 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:55.368 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:55.369 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:55.369 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:02:55.369 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:55.369 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:55.369 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:55.369 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:55.369 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:55.370 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:55.370 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:02:55.370 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:55.370 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:55.370 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:55.370 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:02:55.370 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:02:55.371 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:02:55.371 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:02:55.371 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:02:55.371 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:02:55.371 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:02:55.371 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:02:55.371 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:02:55.372 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:02:55.372 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:55.372 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:02:55.372 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:02:55.372 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:55.372 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:02:55.372 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:02:55.373 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:02:55.373 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:02:55.373 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:02:55Z" 2023-05-24T21:02:55.373 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:02:55.374 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962037.098901_69.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:00:37.098901Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962174.772148_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:02:54.772148Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:02:55Z" } 2023-05-24T21:02:55.375 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962037.098901_69.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:00:37.098901Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962174.772148_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:02:54.772148Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:02:55.375 DEBUG:rgw_multi.tests:log_status={0: '1_1684962037.098901_69.1', 1: '1_1684962174.772148_77.1', 2: '', 3: ''} 2023-05-24T21:02:55.375 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962037.098901_69.1', 1: '1_1684962174.772148_77.1', 2: '', 3: ''} 2023-05-24T21:02:55.376 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-24T21:02:55.376 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-7', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:02:55.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-7', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:02:55.377 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-7 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:02:55.432 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:02:55.433 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:02:55.482 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:02:55.476+0000 7f46fe82f780 1 waiting to reach incremental sync.. 2023-05-24T21:03:00.483 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:03:00.476+0000 7f46fe82f780 1 waiting to reach incremental sync.. 2023-05-24T21:03:05.483 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:03:05.477+0000 7f46fe82f780 1 waiting to reach incremental sync.. 2023-05-24T21:03:10.484 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:03:10.478+0000 7f46fe82f780 1 waiting to reach incremental sync.. 2023-05-24T21:03:15.484 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:03:15.477+0000 7f46fe82f780 1 waiting to reach incremental sync.. 2023-05-24T21:03:20.485 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:03:20.478+0000 7f46fe82f780 1 waiting to reach incremental sync.. 2023-05-24T21:03:25.486 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:03:25.479+0000 7f46fe82f780 1 waiting to reach incremental sync.. 2023-05-24T21:03:30.486 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:03:30.480+0000 7f46fe82f780 1 waiting to reach incremental sync.. 2023-05-24T21:03:35.487 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:03:35.481+0000 7f46fe82f780 1 waiting to reach incremental sync.. 2023-05-24T21:03:40.488 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:03:40.482+0000 7f46fe82f780 1 waiting to reach incremental sync.. 2023-05-24T21:03:45.489 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:03:45.483+0000 7f46fe82f780 1 waiting to reach incremental sync.. 2023-05-24T21:03:50.489 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:03:50.483+0000 7f46fe82f780 1 waiting to reach incremental sync.. 2023-05-24T21:03:55.490 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:03:55.484+0000 7f46fe82f780 1 waiting to reach incremental sync.. 2023-05-24T21:04:00.491 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:04:00.485+0000 7f46fe82f780 1 waiting to reach incremental sync.. 2023-05-24T21:04:05.491 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:04:05.485+0000 7f46fe82f780 1 waiting to reach incremental sync.. 2023-05-24T21:04:10.492 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:04:10.485+0000 7f46fe82f780 1 waiting to reach incremental sync.. 2023-05-24T21:04:15.492 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:04:15.486+0000 7f46fe82f780 1 waiting to reach incremental sync.. 2023-05-24T21:04:20.493 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:04:20.486+0000 7f46fe82f780 1 waiting to reach incremental sync.. 2023-05-24T21:04:25.494 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:04:25.487+0000 7f46fe82f780 1 waiting to reach incremental sync.. 2023-05-24T21:04:30.494 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:04:30.488+0000 7f46fe82f780 1 waiting to reach incremental sync.. 2023-05-24T21:04:35.495 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:04:35.489+0000 7f46fe82f780 1 waiting to reach incremental sync.. 2023-05-24T21:04:40.495 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:04:40.490+0000 7f46fe82f780 1 waiting to reach incremental sync.. 2023-05-24T21:04:45.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:04:45.490+0000 7f46fe82f780 1 waiting to reach incremental sync.. 2023-05-24T21:04:50.497 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:04:50.491+0000 7f46fe82f780 1 waiting to reach incremental sync.. 2023-05-24T21:04:55.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:04:55.492+0000 7f46fe82f780 1 waiting to reach incremental sync.. 2023-05-24T21:05:00.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:05:00.493+0000 7f46fe82f780 1 waiting to reach incremental sync.. 2023-05-24T21:05:05.499 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:05:05.492+0000 7f46fe82f780 1 waiting to reach incremental sync.. 2023-05-24T21:05:10.500 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:05:10.493+0000 7f46fe82f780 1 waiting to reach incremental sync.. 2023-05-24T21:05:15.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:05:15.494+0000 7f46fe82f780 1 bucket sync caught up with source: 2023-05-24T21:05:15.502 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000001.4.6, , 00000000001.6.6, , , , 00000000001.8.6, 00000000001.6.6, , ] 2023-05-24T21:05:15.502 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000001.4.6, , 00000000001.6.6, , , , 00000000001.8.6, 00000000001.6.6, , ] 2023-05-24T21:05:15.503 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:05:15.494+0000 7f46fe82f780 0 bucket checkpoint complete 2023-05-24T21:05:15.517 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qskrps-7 zones={a2, a1} 2023-05-24T21:05:15.518 DEBUG:boto:path=/qskrps-7/ 2023-05-24T21:05:15.518 DEBUG:boto:auth_path=/qskrps-7/ 2023-05-24T21:05:15.519 DEBUG:boto:Method: HEAD 2023-05-24T21:05:15.519 DEBUG:boto:Path: /qskrps-7/ 2023-05-24T21:05:15.520 DEBUG:boto:Data: 2023-05-24T21:05:15.520 DEBUG:boto:Headers: {} 2023-05-24T21:05:15.521 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:05:15.521 DEBUG:boto:Port: 8001 2023-05-24T21:05:15.521 DEBUG:boto:Params: {} 2023-05-24T21:05:15.522 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-24T21:05:15.523 DEBUG:boto:Token: None 2023-05-24T21:05:15.523 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:05:15 GMT /qskrps-7/ 2023-05-24T21:05:15.523 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:fHih8pMpOCKk+cuE7x1oojS+cQs= 2023-05-24T21:05:15.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:fHih8pMpOCKk+cuE7x1oojS+cQs=', 'Content-Length': '0'} 2023-05-24T21:05:15.528 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a34a802fd32410e9-00646e7c0b-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:15.528 DEBUG:boto:path=/qskrps-7/ 2023-05-24T21:05:15.529 DEBUG:boto:auth_path=/qskrps-7/ 2023-05-24T21:05:15.529 DEBUG:boto:Method: HEAD 2023-05-24T21:05:15.529 DEBUG:boto:Path: /qskrps-7/ 2023-05-24T21:05:15.530 DEBUG:boto:Data: 2023-05-24T21:05:15.531 DEBUG:boto:Headers: {} 2023-05-24T21:05:15.532 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:05:15.532 DEBUG:boto:Port: 8000 2023-05-24T21:05:15.533 DEBUG:boto:Params: {} 2023-05-24T21:05:15.534 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T21:05:15.535 DEBUG:boto:Token: None 2023-05-24T21:05:15.535 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:05:15 GMT /qskrps-7/ 2023-05-24T21:05:15.536 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:fHih8pMpOCKk+cuE7x1oojS+cQs= 2023-05-24T21:05:15.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:fHih8pMpOCKk+cuE7x1oojS+cQs=', 'Content-Length': '0'} 2023-05-24T21:05:15.540 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000066e09b692c75591e-00646e7c0b-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:15.540 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-24T21:05:15.541 DEBUG:boto:path=/qskrps-7/ 2023-05-24T21:05:15.628 DEBUG:boto:auth_path=/qskrps-7/ 2023-05-24T21:05:15.628 DEBUG:boto:path=/qskrps-7/?versions&max-keys=999 2023-05-24T21:05:15.629 DEBUG:boto:auth_path=/qskrps-7/?versions&max-keys=999 2023-05-24T21:05:15.629 DEBUG:boto:Method: GET 2023-05-24T21:05:15.630 DEBUG:boto:Path: /qskrps-7/?versions&max-keys=999 2023-05-24T21:05:15.630 DEBUG:boto:Data: 2023-05-24T21:05:15.631 DEBUG:boto:Headers: {} 2023-05-24T21:05:15.631 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:05:15.631 DEBUG:boto:Port: 8001 2023-05-24T21:05:15.632 DEBUG:boto:Params: {} 2023-05-24T21:05:15.632 DEBUG:boto:Token: None 2023-05-24T21:05:15.633 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:05:15 GMT /qskrps-7/?versions 2023-05-24T21:05:15.633 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4k/AY+kGU1w1g0wdmbYVz+10TJs= 2023-05-24T21:05:15.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4k/AY+kGU1w1g0wdmbYVz+10TJs=', 'Content-Length': '0'} 2023-05-24T21:05:15.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099c85fd8fcfa2902-00646e7c0b-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:15.637 DEBUG:boto:b'qskrps-7999false&nulltrue2023-05-24T21:02:54.984Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-05-24T21:02:54.958Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-05-24T21:02:54.897Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-05-24T21:02:54.852Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:05:15.639 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-05-24T21:05:15.639 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-05-24T21:05:15.640 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-05-24T21:05:15.640 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-05-24T21:05:15.641 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-24T21:05:15.641 DEBUG:boto:path=/qskrps-7/ 2023-05-24T21:05:15.641 DEBUG:boto:auth_path=/qskrps-7/ 2023-05-24T21:05:15.642 DEBUG:boto:path=/qskrps-7/?versions&max-keys=999 2023-05-24T21:05:15.642 DEBUG:boto:auth_path=/qskrps-7/?versions&max-keys=999 2023-05-24T21:05:15.642 DEBUG:boto:Method: GET 2023-05-24T21:05:15.643 DEBUG:boto:Path: /qskrps-7/?versions&max-keys=999 2023-05-24T21:05:15.643 DEBUG:boto:Data: 2023-05-24T21:05:15.644 DEBUG:boto:Headers: {} 2023-05-24T21:05:15.644 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:05:15.644 DEBUG:boto:Port: 8000 2023-05-24T21:05:15.645 DEBUG:boto:Params: {} 2023-05-24T21:05:15.645 DEBUG:boto:Token: None 2023-05-24T21:05:15.645 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:05:15 GMT /qskrps-7/?versions 2023-05-24T21:05:15.646 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4k/AY+kGU1w1g0wdmbYVz+10TJs= 2023-05-24T21:05:15.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4k/AY+kGU1w1g0wdmbYVz+10TJs=', 'Content-Length': '0'} 2023-05-24T21:05:15.649 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d4683314b8502f5-00646e7c0b-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:15.649 DEBUG:boto:b'qskrps-7999false&nulltrue2023-05-24T21:02:54.984Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-05-24T21:02:54.958Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-05-24T21:02:54.897Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-05-24T21:02:54.852Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:05:15.650 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-05-24T21:05:15.650 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-05-24T21:05:15.650 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-05-24T21:05:15.651 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-05-24T21:05:15.651 DEBUG:boto:path=/qskrps-7/ 2023-05-24T21:05:15.652 DEBUG:boto:auth_path=/qskrps-7/ 2023-05-24T21:05:15.652 DEBUG:boto:path=/qskrps-7/?versions&max-keys=999 2023-05-24T21:05:15.652 DEBUG:boto:auth_path=/qskrps-7/?versions&max-keys=999 2023-05-24T21:05:15.653 DEBUG:boto:Method: GET 2023-05-24T21:05:15.653 DEBUG:boto:Path: /qskrps-7/?versions&max-keys=999 2023-05-24T21:05:15.653 DEBUG:boto:Data: 2023-05-24T21:05:15.654 DEBUG:boto:Headers: {} 2023-05-24T21:05:15.654 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:05:15.654 DEBUG:boto:Port: 8001 2023-05-24T21:05:15.655 DEBUG:boto:Params: {} 2023-05-24T21:05:15.655 DEBUG:boto:Token: None 2023-05-24T21:05:15.655 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:05:15 GMT /qskrps-7/?versions 2023-05-24T21:05:15.656 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4k/AY+kGU1w1g0wdmbYVz+10TJs= 2023-05-24T21:05:15.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4k/AY+kGU1w1g0wdmbYVz+10TJs=', 'Content-Length': '0'} 2023-05-24T21:05:15.659 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000526878d0c8073a8a-00646e7c0b-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:15.660 DEBUG:boto:b'qskrps-7999false&nulltrue2023-05-24T21:02:54.984Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-05-24T21:02:54.958Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-05-24T21:02:54.897Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-05-24T21:02:54.852Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:05:15.660 DEBUG:boto:path=/qskrps-7/ 2023-05-24T21:05:15.661 DEBUG:boto:auth_path=/qskrps-7/ 2023-05-24T21:05:15.661 DEBUG:boto:path=/qskrps-7/?versions&max-keys=999 2023-05-24T21:05:15.661 DEBUG:boto:auth_path=/qskrps-7/?versions&max-keys=999 2023-05-24T21:05:15.662 DEBUG:boto:Method: GET 2023-05-24T21:05:15.662 DEBUG:boto:Path: /qskrps-7/?versions&max-keys=999 2023-05-24T21:05:15.662 DEBUG:boto:Data: 2023-05-24T21:05:15.663 DEBUG:boto:Headers: {} 2023-05-24T21:05:15.663 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:05:15.663 DEBUG:boto:Port: 8000 2023-05-24T21:05:15.664 DEBUG:boto:Params: {} 2023-05-24T21:05:15.664 DEBUG:boto:Token: None 2023-05-24T21:05:15.665 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:05:15 GMT /qskrps-7/?versions 2023-05-24T21:05:15.665 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4k/AY+kGU1w1g0wdmbYVz+10TJs= 2023-05-24T21:05:15.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4k/AY+kGU1w1g0wdmbYVz+10TJs=', 'Content-Length': '0'} 2023-05-24T21:05:15.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006849d6716e15c486-00646e7c0b-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:15.668 DEBUG:boto:b'qskrps-7999false&nulltrue2023-05-24T21:02:54.984Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-05-24T21:02:54.958Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-05-24T21:02:54.897Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-05-24T21:02:54.852Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:05:15.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-05-24T21:05:15.670 DEBUG:boto:path=/qskrps-7/%26 2023-05-24T21:05:15.670 DEBUG:boto:auth_path=/qskrps-7/%26 2023-05-24T21:05:15.670 DEBUG:boto:path=/qskrps-7/%26?versionId=null 2023-05-24T21:05:15.671 DEBUG:boto:auth_path=/qskrps-7/%26?versionId=null 2023-05-24T21:05:15.671 DEBUG:boto:Method: GET 2023-05-24T21:05:15.671 DEBUG:boto:Path: /qskrps-7/%26?versionId=null 2023-05-24T21:05:15.672 DEBUG:boto:Data: 2023-05-24T21:05:15.672 DEBUG:boto:Headers: {} 2023-05-24T21:05:15.673 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:05:15.673 DEBUG:boto:Port: 8001 2023-05-24T21:05:15.673 DEBUG:boto:Params: {} 2023-05-24T21:05:15.674 DEBUG:boto:Token: None 2023-05-24T21:05:15.674 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:05:15 GMT /qskrps-7/%26?versionId=null 2023-05-24T21:05:15.675 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:NhKkZ364GlONIbTghwO3l55fUEE= 2023-05-24T21:05:15.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:NhKkZ364GlONIbTghwO3l55fUEE=', 'Content-Length': '0'} 2023-05-24T21:05:15.677 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000481a78c8e37715a8-00646e7c0b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:15.678 DEBUG:boto:path=/qskrps-7/%26 2023-05-24T21:05:15.678 DEBUG:boto:auth_path=/qskrps-7/%26 2023-05-24T21:05:15.678 DEBUG:boto:path=/qskrps-7/%26?versionId=null 2023-05-24T21:05:15.679 DEBUG:boto:auth_path=/qskrps-7/%26?versionId=null 2023-05-24T21:05:15.679 DEBUG:boto:Method: GET 2023-05-24T21:05:15.679 DEBUG:boto:Path: /qskrps-7/%26?versionId=null 2023-05-24T21:05:15.680 DEBUG:boto:Data: 2023-05-24T21:05:15.680 DEBUG:boto:Headers: {} 2023-05-24T21:05:15.680 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:05:15.681 DEBUG:boto:Port: 8000 2023-05-24T21:05:15.681 DEBUG:boto:Params: {} 2023-05-24T21:05:15.682 DEBUG:boto:Token: None 2023-05-24T21:05:15.682 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:05:15 GMT /qskrps-7/%26?versionId=null 2023-05-24T21:05:15.682 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:NhKkZ364GlONIbTghwO3l55fUEE= 2023-05-24T21:05:15.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:NhKkZ364GlONIbTghwO3l55fUEE=', 'Content-Length': '0'} 2023-05-24T21:05:15.685 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000db4bfb44458e20bb-00646e7c0b-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:15.686 DEBUG:boto:path=/qskrps-7/%26 2023-05-24T21:05:15.686 DEBUG:boto:auth_path=/qskrps-7/%26 2023-05-24T21:05:15.687 DEBUG:boto:path=/qskrps-7/%26?versionId=null 2023-05-24T21:05:15.687 DEBUG:boto:auth_path=/qskrps-7/%26?versionId=null 2023-05-24T21:05:15.687 DEBUG:boto:Method: HEAD 2023-05-24T21:05:15.688 DEBUG:boto:Path: /qskrps-7/%26?versionId=null 2023-05-24T21:05:15.688 DEBUG:boto:Data: 2023-05-24T21:05:15.688 DEBUG:boto:Headers: {} 2023-05-24T21:05:15.689 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:05:15.689 DEBUG:boto:Port: 8001 2023-05-24T21:05:15.689 DEBUG:boto:Params: {} 2023-05-24T21:05:15.690 DEBUG:boto:Token: None 2023-05-24T21:05:15.690 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:05:15 GMT /qskrps-7/%26?versionId=null 2023-05-24T21:05:15.691 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:gfGPQv0bhEkMnXBlJpbwL0hTadQ= 2023-05-24T21:05:15.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:gfGPQv0bhEkMnXBlJpbwL0hTadQ=', 'Content-Length': '0'} 2023-05-24T21:05:15.693 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004434b0512d75b6f8-00646e7c0b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:15.693 DEBUG:boto:path=/qskrps-7/%26 2023-05-24T21:05:15.694 DEBUG:boto:auth_path=/qskrps-7/%26 2023-05-24T21:05:15.694 DEBUG:boto:path=/qskrps-7/%26?versionId=null 2023-05-24T21:05:15.694 DEBUG:boto:auth_path=/qskrps-7/%26?versionId=null 2023-05-24T21:05:15.695 DEBUG:boto:Method: HEAD 2023-05-24T21:05:15.695 DEBUG:boto:Path: /qskrps-7/%26?versionId=null 2023-05-24T21:05:15.695 DEBUG:boto:Data: 2023-05-24T21:05:15.696 DEBUG:boto:Headers: {} 2023-05-24T21:05:15.696 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:05:15.696 DEBUG:boto:Port: 8000 2023-05-24T21:05:15.697 DEBUG:boto:Params: {} 2023-05-24T21:05:15.697 DEBUG:boto:Token: None 2023-05-24T21:05:15.698 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:05:15 GMT /qskrps-7/%26?versionId=null 2023-05-24T21:05:15.698 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:gfGPQv0bhEkMnXBlJpbwL0hTadQ= 2023-05-24T21:05:15.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:gfGPQv0bhEkMnXBlJpbwL0hTadQ=', 'Content-Length': '0'} 2023-05-24T21:05:15.700 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000efd2f4f6c658fc88-00646e7c0b-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:15.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-05-24T21:05:15.701 DEBUG:boto:path=/qskrps-7/%26 2023-05-24T21:05:15.701 DEBUG:boto:auth_path=/qskrps-7/%26 2023-05-24T21:05:15.702 DEBUG:boto:path=/qskrps-7/%26?versionId=null 2023-05-24T21:05:15.702 DEBUG:boto:auth_path=/qskrps-7/%26?versionId=null 2023-05-24T21:05:15.703 DEBUG:boto:Method: GET 2023-05-24T21:05:15.703 DEBUG:boto:Path: /qskrps-7/%26?versionId=null 2023-05-24T21:05:15.703 DEBUG:boto:Data: 2023-05-24T21:05:15.704 DEBUG:boto:Headers: {} 2023-05-24T21:05:15.704 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:05:15.705 DEBUG:boto:Port: 8001 2023-05-24T21:05:15.705 DEBUG:boto:Params: {} 2023-05-24T21:05:15.705 DEBUG:boto:Token: None 2023-05-24T21:05:15.706 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:05:15 GMT /qskrps-7/%26?versionId=null 2023-05-24T21:05:15.706 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:NhKkZ364GlONIbTghwO3l55fUEE= 2023-05-24T21:05:15.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:NhKkZ364GlONIbTghwO3l55fUEE=', 'Content-Length': '0'} 2023-05-24T21:05:15.708 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005013bd183f7949ab-00646e7c0b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:15.752 DEBUG:boto:path=/qskrps-7/%26 2023-05-24T21:05:15.753 DEBUG:boto:auth_path=/qskrps-7/%26 2023-05-24T21:05:15.753 DEBUG:boto:path=/qskrps-7/%26?versionId=null 2023-05-24T21:05:15.754 DEBUG:boto:auth_path=/qskrps-7/%26?versionId=null 2023-05-24T21:05:15.754 DEBUG:boto:Method: GET 2023-05-24T21:05:15.754 DEBUG:boto:Path: /qskrps-7/%26?versionId=null 2023-05-24T21:05:15.755 DEBUG:boto:Data: 2023-05-24T21:05:15.755 DEBUG:boto:Headers: {} 2023-05-24T21:05:15.755 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:05:15.756 DEBUG:boto:Port: 8000 2023-05-24T21:05:15.756 DEBUG:boto:Params: {} 2023-05-24T21:05:15.757 DEBUG:boto:Token: None 2023-05-24T21:05:15.757 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:05:15 GMT /qskrps-7/%26?versionId=null 2023-05-24T21:05:15.758 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:NhKkZ364GlONIbTghwO3l55fUEE= 2023-05-24T21:05:15.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:NhKkZ364GlONIbTghwO3l55fUEE=', 'Content-Length': '0'} 2023-05-24T21:05:15.760 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000de895e119edbdbf5-00646e7c0b-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:15.761 DEBUG:boto:path=/qskrps-7/ 2023-05-24T21:05:15.761 DEBUG:boto:auth_path=/qskrps-7/ 2023-05-24T21:05:15.761 DEBUG:boto:path=/qskrps-7/?prefix=%26 2023-05-24T21:05:15.762 DEBUG:boto:auth_path=/qskrps-7/?prefix=%26 2023-05-24T21:05:15.762 DEBUG:boto:Method: GET 2023-05-24T21:05:15.762 DEBUG:boto:Path: /qskrps-7/?prefix=%26 2023-05-24T21:05:15.763 DEBUG:boto:Data: 2023-05-24T21:05:15.763 DEBUG:boto:Headers: {} 2023-05-24T21:05:15.764 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:05:15.764 DEBUG:boto:Port: 8001 2023-05-24T21:05:15.764 DEBUG:boto:Params: {} 2023-05-24T21:05:15.765 DEBUG:boto:Token: None 2023-05-24T21:05:15.765 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:05:15 GMT /qskrps-7/ 2023-05-24T21:05:15.766 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:B0Gp5KengGin8YzxiApQcwgOa8Q= 2023-05-24T21:05:15.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:B0Gp5KengGin8YzxiApQcwgOa8Q=', 'Content-Length': '0'} 2023-05-24T21:05:15.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000532831d89b218df9-00646e7c0b-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:15.770 DEBUG:boto:b'qskrps-7&1000false&2023-05-24T21:02:54.984Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:05:15.770 DEBUG:boto:path=/qskrps-7/ 2023-05-24T21:05:15.771 DEBUG:boto:auth_path=/qskrps-7/ 2023-05-24T21:05:15.771 DEBUG:boto:path=/qskrps-7/?prefix=%26 2023-05-24T21:05:15.771 DEBUG:boto:auth_path=/qskrps-7/?prefix=%26 2023-05-24T21:05:15.772 DEBUG:boto:Method: GET 2023-05-24T21:05:15.772 DEBUG:boto:Path: /qskrps-7/?prefix=%26 2023-05-24T21:05:15.773 DEBUG:boto:Data: 2023-05-24T21:05:15.773 DEBUG:boto:Headers: {} 2023-05-24T21:05:15.773 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:05:15.774 DEBUG:boto:Port: 8000 2023-05-24T21:05:15.774 DEBUG:boto:Params: {} 2023-05-24T21:05:15.774 DEBUG:boto:Token: None 2023-05-24T21:05:15.775 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:05:15 GMT /qskrps-7/ 2023-05-24T21:05:15.775 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:B0Gp5KengGin8YzxiApQcwgOa8Q= 2023-05-24T21:05:15.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:B0Gp5KengGin8YzxiApQcwgOa8Q=', 'Content-Length': '0'} 2023-05-24T21:05:15.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cf3dd5c88c5a7c4-00646e7c0b-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:15.779 DEBUG:boto:b'qskrps-7&1000false&2023-05-24T21:02:54.984Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:05:15.779 DEBUG:boto:path=/qskrps-7/%26 2023-05-24T21:05:15.780 DEBUG:boto:auth_path=/qskrps-7/%26 2023-05-24T21:05:15.780 DEBUG:boto:Method: HEAD 2023-05-24T21:05:15.781 DEBUG:boto:Path: /qskrps-7/%26 2023-05-24T21:05:15.781 DEBUG:boto:Data: 2023-05-24T21:05:15.781 DEBUG:boto:Headers: {} 2023-05-24T21:05:15.782 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:05:15.782 DEBUG:boto:Port: 8001 2023-05-24T21:05:15.782 DEBUG:boto:Params: {} 2023-05-24T21:05:15.783 DEBUG:boto:Token: None 2023-05-24T21:05:15.783 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:05:15 GMT /qskrps-7/%26 2023-05-24T21:05:15.783 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:oTKsBvd8RCK59GIHghLRJ68hOKM= 2023-05-24T21:05:15.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:oTKsBvd8RCK59GIHghLRJ68hOKM=', 'Content-Length': '0'} 2023-05-24T21:05:15.786 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000546cc0dc8e21f26e-00646e7c0b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:15.786 DEBUG:boto:path=/qskrps-7/%26 2023-05-24T21:05:15.787 DEBUG:boto:auth_path=/qskrps-7/%26 2023-05-24T21:05:15.787 DEBUG:boto:Method: HEAD 2023-05-24T21:05:15.787 DEBUG:boto:Path: /qskrps-7/%26 2023-05-24T21:05:15.788 DEBUG:boto:Data: 2023-05-24T21:05:15.788 DEBUG:boto:Headers: {} 2023-05-24T21:05:15.788 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:05:15.789 DEBUG:boto:Port: 8000 2023-05-24T21:05:15.789 DEBUG:boto:Params: {} 2023-05-24T21:05:15.789 DEBUG:boto:Token: None 2023-05-24T21:05:15.790 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:05:15 GMT /qskrps-7/%26 2023-05-24T21:05:15.790 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:oTKsBvd8RCK59GIHghLRJ68hOKM= 2023-05-24T21:05:15.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:oTKsBvd8RCK59GIHghLRJ68hOKM=', 'Content-Length': '0'} 2023-05-24T21:05:15.792 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b774d443d072398-00646e7c0b-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:15.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-05-24T21:05:15.793 DEBUG:boto:path=/qskrps-7/%26 2023-05-24T21:05:15.794 DEBUG:boto:auth_path=/qskrps-7/%26 2023-05-24T21:05:15.794 DEBUG:boto:Method: GET 2023-05-24T21:05:15.794 DEBUG:boto:Path: /qskrps-7/%26 2023-05-24T21:05:15.795 DEBUG:boto:Data: 2023-05-24T21:05:15.795 DEBUG:boto:Headers: {} 2023-05-24T21:05:15.795 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:05:15.796 DEBUG:boto:Port: 8001 2023-05-24T21:05:15.796 DEBUG:boto:Params: {} 2023-05-24T21:05:15.797 DEBUG:boto:Token: None 2023-05-24T21:05:15.797 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:05:15 GMT /qskrps-7/%26 2023-05-24T21:05:15.797 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:OaPdOoK9/lFwQ3DBhF8LkK0QkXM= 2023-05-24T21:05:15.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:OaPdOoK9/lFwQ3DBhF8LkK0QkXM=', 'Content-Length': '0'} 2023-05-24T21:05:15.800 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000879843e11cc218aa-00646e7c0b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:15.800 DEBUG:boto:path=/qskrps-7/%26 2023-05-24T21:05:15.801 DEBUG:boto:auth_path=/qskrps-7/%26 2023-05-24T21:05:15.801 DEBUG:boto:Method: GET 2023-05-24T21:05:15.801 DEBUG:boto:Path: /qskrps-7/%26 2023-05-24T21:05:15.802 DEBUG:boto:Data: 2023-05-24T21:05:15.802 DEBUG:boto:Headers: {} 2023-05-24T21:05:15.802 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:05:15.803 DEBUG:boto:Port: 8000 2023-05-24T21:05:15.803 DEBUG:boto:Params: {} 2023-05-24T21:05:15.804 DEBUG:boto:Token: None 2023-05-24T21:05:15.804 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:05:15 GMT /qskrps-7/%26 2023-05-24T21:05:15.804 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:OaPdOoK9/lFwQ3DBhF8LkK0QkXM= 2023-05-24T21:05:15.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:OaPdOoK9/lFwQ3DBhF8LkK0QkXM=', 'Content-Length': '0'} 2023-05-24T21:05:15.807 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008c664ede6ff1035f-00646e7c0b-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:15.848 DEBUG:boto:path=/qskrps-7/ 2023-05-24T21:05:15.849 DEBUG:boto:auth_path=/qskrps-7/ 2023-05-24T21:05:15.849 DEBUG:boto:path=/qskrps-7/?prefix=%26 2023-05-24T21:05:15.850 DEBUG:boto:auth_path=/qskrps-7/?prefix=%26 2023-05-24T21:05:15.850 DEBUG:boto:Method: GET 2023-05-24T21:05:15.850 DEBUG:boto:Path: /qskrps-7/?prefix=%26 2023-05-24T21:05:15.851 DEBUG:boto:Data: 2023-05-24T21:05:15.851 DEBUG:boto:Headers: {} 2023-05-24T21:05:15.851 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:05:15.852 DEBUG:boto:Port: 8001 2023-05-24T21:05:15.852 DEBUG:boto:Params: {} 2023-05-24T21:05:15.853 DEBUG:boto:Token: None 2023-05-24T21:05:15.853 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:05:15 GMT /qskrps-7/ 2023-05-24T21:05:15.854 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:B0Gp5KengGin8YzxiApQcwgOa8Q= 2023-05-24T21:05:15.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:B0Gp5KengGin8YzxiApQcwgOa8Q=', 'Content-Length': '0'} 2023-05-24T21:05:15.857 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d05db99bb1159faa-00646e7c0b-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:15.857 DEBUG:boto:b'qskrps-7&1000false&2023-05-24T21:02:54.984Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:05:15.858 DEBUG:boto:path=/qskrps-7/ 2023-05-24T21:05:15.859 DEBUG:boto:auth_path=/qskrps-7/ 2023-05-24T21:05:15.859 DEBUG:boto:path=/qskrps-7/?prefix=%26 2023-05-24T21:05:15.859 DEBUG:boto:auth_path=/qskrps-7/?prefix=%26 2023-05-24T21:05:15.860 DEBUG:boto:Method: GET 2023-05-24T21:05:15.860 DEBUG:boto:Path: /qskrps-7/?prefix=%26 2023-05-24T21:05:15.861 DEBUG:boto:Data: 2023-05-24T21:05:15.861 DEBUG:boto:Headers: {} 2023-05-24T21:05:15.861 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:05:15.862 DEBUG:boto:Port: 8000 2023-05-24T21:05:15.862 DEBUG:boto:Params: {} 2023-05-24T21:05:15.862 DEBUG:boto:Token: None 2023-05-24T21:05:15.863 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:05:15 GMT /qskrps-7/ 2023-05-24T21:05:15.863 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:B0Gp5KengGin8YzxiApQcwgOa8Q= 2023-05-24T21:05:15.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:B0Gp5KengGin8YzxiApQcwgOa8Q=', 'Content-Length': '0'} 2023-05-24T21:05:15.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e694b3c8beff501b-00646e7c0b-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:15.867 DEBUG:boto:b'qskrps-7&1000false&2023-05-24T21:02:54.984Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:05:15.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-05-24T21:05:15.868 DEBUG:boto:path=/qskrps-7/%3A 2023-05-24T21:05:15.868 DEBUG:boto:auth_path=/qskrps-7/%3A 2023-05-24T21:05:15.869 DEBUG:boto:path=/qskrps-7/%3A?versionId=null 2023-05-24T21:05:15.869 DEBUG:boto:auth_path=/qskrps-7/%3A?versionId=null 2023-05-24T21:05:15.870 DEBUG:boto:Method: GET 2023-05-24T21:05:15.870 DEBUG:boto:Path: /qskrps-7/%3A?versionId=null 2023-05-24T21:05:15.870 DEBUG:boto:Data: 2023-05-24T21:05:15.871 DEBUG:boto:Headers: {} 2023-05-24T21:05:15.871 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:05:15.872 DEBUG:boto:Port: 8001 2023-05-24T21:05:15.872 DEBUG:boto:Params: {} 2023-05-24T21:05:15.872 DEBUG:boto:Token: None 2023-05-24T21:05:15.873 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:05:15 GMT /qskrps-7/%3A?versionId=null 2023-05-24T21:05:15.873 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:uGfYR9QwX217Hoj46q5KNqiVvsE= 2023-05-24T21:05:15.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:uGfYR9QwX217Hoj46q5KNqiVvsE=', 'Content-Length': '0'} 2023-05-24T21:05:15.875 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aaa00c7138a427de-00646e7c0b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:15.916 DEBUG:boto:path=/qskrps-7/%3A 2023-05-24T21:05:15.917 DEBUG:boto:auth_path=/qskrps-7/%3A 2023-05-24T21:05:15.917 DEBUG:boto:path=/qskrps-7/%3A?versionId=null 2023-05-24T21:05:15.918 DEBUG:boto:auth_path=/qskrps-7/%3A?versionId=null 2023-05-24T21:05:15.918 DEBUG:boto:Method: GET 2023-05-24T21:05:15.919 DEBUG:boto:Path: /qskrps-7/%3A?versionId=null 2023-05-24T21:05:15.919 DEBUG:boto:Data: 2023-05-24T21:05:15.920 DEBUG:boto:Headers: {} 2023-05-24T21:05:15.920 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:05:15.921 DEBUG:boto:Port: 8000 2023-05-24T21:05:15.921 DEBUG:boto:Params: {} 2023-05-24T21:05:15.922 DEBUG:boto:Token: None 2023-05-24T21:05:15.922 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:05:15 GMT /qskrps-7/%3A?versionId=null 2023-05-24T21:05:15.923 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:uGfYR9QwX217Hoj46q5KNqiVvsE= 2023-05-24T21:05:15.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:uGfYR9QwX217Hoj46q5KNqiVvsE=', 'Content-Length': '0'} 2023-05-24T21:05:15.926 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008415e813e4196329-00646e7c0b-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:15.927 DEBUG:boto:path=/qskrps-7/%3A 2023-05-24T21:05:15.927 DEBUG:boto:auth_path=/qskrps-7/%3A 2023-05-24T21:05:15.928 DEBUG:boto:path=/qskrps-7/%3A?versionId=null 2023-05-24T21:05:15.929 DEBUG:boto:auth_path=/qskrps-7/%3A?versionId=null 2023-05-24T21:05:15.930 DEBUG:boto:Method: HEAD 2023-05-24T21:05:15.930 DEBUG:boto:Path: /qskrps-7/%3A?versionId=null 2023-05-24T21:05:15.931 DEBUG:boto:Data: 2023-05-24T21:05:15.932 DEBUG:boto:Headers: {} 2023-05-24T21:05:15.933 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:05:15.933 DEBUG:boto:Port: 8001 2023-05-24T21:05:15.933 DEBUG:boto:Params: {} 2023-05-24T21:05:15.934 DEBUG:boto:Token: None 2023-05-24T21:05:15.934 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:05:15 GMT /qskrps-7/%3A?versionId=null 2023-05-24T21:05:15.934 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:UWsCnBJoUNROtfISWvEo//HQSTs= 2023-05-24T21:05:15.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:UWsCnBJoUNROtfISWvEo//HQSTs=', 'Content-Length': '0'} 2023-05-24T21:05:15.937 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000947d282b77be42ed-00646e7c0b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:15.938 DEBUG:boto:path=/qskrps-7/%3A 2023-05-24T21:05:15.938 DEBUG:boto:auth_path=/qskrps-7/%3A 2023-05-24T21:05:15.938 DEBUG:boto:path=/qskrps-7/%3A?versionId=null 2023-05-24T21:05:15.939 DEBUG:boto:auth_path=/qskrps-7/%3A?versionId=null 2023-05-24T21:05:15.939 DEBUG:boto:Method: HEAD 2023-05-24T21:05:15.939 DEBUG:boto:Path: /qskrps-7/%3A?versionId=null 2023-05-24T21:05:15.940 DEBUG:boto:Data: 2023-05-24T21:05:15.940 DEBUG:boto:Headers: {} 2023-05-24T21:05:15.940 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:05:15.941 DEBUG:boto:Port: 8000 2023-05-24T21:05:15.941 DEBUG:boto:Params: {} 2023-05-24T21:05:15.941 DEBUG:boto:Token: None 2023-05-24T21:05:15.942 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:05:15 GMT /qskrps-7/%3A?versionId=null 2023-05-24T21:05:15.942 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:UWsCnBJoUNROtfISWvEo//HQSTs= 2023-05-24T21:05:15.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:UWsCnBJoUNROtfISWvEo//HQSTs=', 'Content-Length': '0'} 2023-05-24T21:05:15.944 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d507b9f708689166-00646e7c0b-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:15.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-05-24T21:05:15.945 DEBUG:boto:path=/qskrps-7/%3A 2023-05-24T21:05:15.946 DEBUG:boto:auth_path=/qskrps-7/%3A 2023-05-24T21:05:15.946 DEBUG:boto:path=/qskrps-7/%3A?versionId=null 2023-05-24T21:05:15.947 DEBUG:boto:auth_path=/qskrps-7/%3A?versionId=null 2023-05-24T21:05:15.947 DEBUG:boto:Method: GET 2023-05-24T21:05:15.947 DEBUG:boto:Path: /qskrps-7/%3A?versionId=null 2023-05-24T21:05:15.948 DEBUG:boto:Data: 2023-05-24T21:05:15.948 DEBUG:boto:Headers: {} 2023-05-24T21:05:15.949 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:05:15.949 DEBUG:boto:Port: 8001 2023-05-24T21:05:15.949 DEBUG:boto:Params: {} 2023-05-24T21:05:15.950 DEBUG:boto:Token: None 2023-05-24T21:05:15.950 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:05:15 GMT /qskrps-7/%3A?versionId=null 2023-05-24T21:05:15.951 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:uGfYR9QwX217Hoj46q5KNqiVvsE= 2023-05-24T21:05:15.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:uGfYR9QwX217Hoj46q5KNqiVvsE=', 'Content-Length': '0'} 2023-05-24T21:05:15.953 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c7c0a6ee531f682d-00646e7c0b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:15.954 DEBUG:boto:path=/qskrps-7/%3A 2023-05-24T21:05:15.954 DEBUG:boto:auth_path=/qskrps-7/%3A 2023-05-24T21:05:15.954 DEBUG:boto:path=/qskrps-7/%3A?versionId=null 2023-05-24T21:05:15.955 DEBUG:boto:auth_path=/qskrps-7/%3A?versionId=null 2023-05-24T21:05:15.955 DEBUG:boto:Method: GET 2023-05-24T21:05:15.956 DEBUG:boto:Path: /qskrps-7/%3A?versionId=null 2023-05-24T21:05:15.956 DEBUG:boto:Data: 2023-05-24T21:05:15.956 DEBUG:boto:Headers: {} 2023-05-24T21:05:15.957 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:05:15.957 DEBUG:boto:Port: 8000 2023-05-24T21:05:15.957 DEBUG:boto:Params: {} 2023-05-24T21:05:15.958 DEBUG:boto:Token: None 2023-05-24T21:05:15.958 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:05:15 GMT /qskrps-7/%3A?versionId=null 2023-05-24T21:05:15.959 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:uGfYR9QwX217Hoj46q5KNqiVvsE= 2023-05-24T21:05:15.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:uGfYR9QwX217Hoj46q5KNqiVvsE=', 'Content-Length': '0'} 2023-05-24T21:05:15.961 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000879a99edcf775dc9-00646e7c0b-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:15.962 DEBUG:boto:path=/qskrps-7/ 2023-05-24T21:05:15.962 DEBUG:boto:auth_path=/qskrps-7/ 2023-05-24T21:05:15.962 DEBUG:boto:path=/qskrps-7/?prefix=%3A 2023-05-24T21:05:15.963 DEBUG:boto:auth_path=/qskrps-7/?prefix=%3A 2023-05-24T21:05:15.963 DEBUG:boto:Method: GET 2023-05-24T21:05:15.963 DEBUG:boto:Path: /qskrps-7/?prefix=%3A 2023-05-24T21:05:15.964 DEBUG:boto:Data: 2023-05-24T21:05:15.964 DEBUG:boto:Headers: {} 2023-05-24T21:05:15.964 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:05:15.965 DEBUG:boto:Port: 8001 2023-05-24T21:05:15.965 DEBUG:boto:Params: {} 2023-05-24T21:05:15.966 DEBUG:boto:Token: None 2023-05-24T21:05:15.966 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:05:15 GMT /qskrps-7/ 2023-05-24T21:05:15.966 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:B0Gp5KengGin8YzxiApQcwgOa8Q= 2023-05-24T21:05:15.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:B0Gp5KengGin8YzxiApQcwgOa8Q=', 'Content-Length': '0'} 2023-05-24T21:05:15.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8ae18d4990b4942-00646e7c0b-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:16.012 DEBUG:boto:b'qskrps-7:1000false:2023-05-24T21:02:54.958Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:05:16.013 DEBUG:boto:path=/qskrps-7/ 2023-05-24T21:05:16.014 DEBUG:boto:auth_path=/qskrps-7/ 2023-05-24T21:05:16.014 DEBUG:boto:path=/qskrps-7/?prefix=%3A 2023-05-24T21:05:16.014 DEBUG:boto:auth_path=/qskrps-7/?prefix=%3A 2023-05-24T21:05:16.015 DEBUG:boto:Method: GET 2023-05-24T21:05:16.015 DEBUG:boto:Path: /qskrps-7/?prefix=%3A 2023-05-24T21:05:16.015 DEBUG:boto:Data: 2023-05-24T21:05:16.016 DEBUG:boto:Headers: {} 2023-05-24T21:05:16.016 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:05:16.017 DEBUG:boto:Port: 8000 2023-05-24T21:05:16.017 DEBUG:boto:Params: {} 2023-05-24T21:05:16.017 DEBUG:boto:Token: None 2023-05-24T21:05:16.018 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:05:16 GMT /qskrps-7/ 2023-05-24T21:05:16.018 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:wcFyPdIWu8scGBX1Gyp++PpcQB4= 2023-05-24T21:05:16.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:16 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wcFyPdIWu8scGBX1Gyp++PpcQB4=', 'Content-Length': '0'} 2023-05-24T21:05:16.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028ee9a7bee06c920-00646e7c0c-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:16.022 DEBUG:boto:b'qskrps-7:1000false:2023-05-24T21:02:54.958Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:05:16.022 DEBUG:boto:path=/qskrps-7/%3A 2023-05-24T21:05:16.023 DEBUG:boto:auth_path=/qskrps-7/%3A 2023-05-24T21:05:16.023 DEBUG:boto:Method: HEAD 2023-05-24T21:05:16.024 DEBUG:boto:Path: /qskrps-7/%3A 2023-05-24T21:05:16.024 DEBUG:boto:Data: 2023-05-24T21:05:16.024 DEBUG:boto:Headers: {} 2023-05-24T21:05:16.025 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:05:16.025 DEBUG:boto:Port: 8001 2023-05-24T21:05:16.025 DEBUG:boto:Params: {} 2023-05-24T21:05:16.026 DEBUG:boto:Token: None 2023-05-24T21:05:16.026 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:05:16 GMT /qskrps-7/%3A 2023-05-24T21:05:16.026 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ED+WZaktXyLerqmWLoH55/EOO9M= 2023-05-24T21:05:16.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:16 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ED+WZaktXyLerqmWLoH55/EOO9M=', 'Content-Length': '0'} 2023-05-24T21:05:16.029 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000547114f3466a0619-00646e7c0c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:16.029 DEBUG:boto:path=/qskrps-7/%3A 2023-05-24T21:05:16.030 DEBUG:boto:auth_path=/qskrps-7/%3A 2023-05-24T21:05:16.030 DEBUG:boto:Method: HEAD 2023-05-24T21:05:16.030 DEBUG:boto:Path: /qskrps-7/%3A 2023-05-24T21:05:16.031 DEBUG:boto:Data: 2023-05-24T21:05:16.031 DEBUG:boto:Headers: {} 2023-05-24T21:05:16.031 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:05:16.032 DEBUG:boto:Port: 8000 2023-05-24T21:05:16.032 DEBUG:boto:Params: {} 2023-05-24T21:05:16.032 DEBUG:boto:Token: None 2023-05-24T21:05:16.033 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:05:16 GMT /qskrps-7/%3A 2023-05-24T21:05:16.033 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ED+WZaktXyLerqmWLoH55/EOO9M= 2023-05-24T21:05:16.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:16 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ED+WZaktXyLerqmWLoH55/EOO9M=', 'Content-Length': '0'} 2023-05-24T21:05:16.036 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006fe7540db8e2ef6d-00646e7c0c-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:16.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-05-24T21:05:16.037 DEBUG:boto:path=/qskrps-7/%3A 2023-05-24T21:05:16.037 DEBUG:boto:auth_path=/qskrps-7/%3A 2023-05-24T21:05:16.038 DEBUG:boto:Method: GET 2023-05-24T21:05:16.038 DEBUG:boto:Path: /qskrps-7/%3A 2023-05-24T21:05:16.038 DEBUG:boto:Data: 2023-05-24T21:05:16.039 DEBUG:boto:Headers: {} 2023-05-24T21:05:16.039 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:05:16.039 DEBUG:boto:Port: 8001 2023-05-24T21:05:16.040 DEBUG:boto:Params: {} 2023-05-24T21:05:16.040 DEBUG:boto:Token: None 2023-05-24T21:05:16.040 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:05:16 GMT /qskrps-7/%3A 2023-05-24T21:05:16.041 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:+pwKxctTGJxgXS5PQ6JXmPcyFzI= 2023-05-24T21:05:16.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:16 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:+pwKxctTGJxgXS5PQ6JXmPcyFzI=', 'Content-Length': '0'} 2023-05-24T21:05:16.043 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000481639e9361048e3-00646e7c0c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:16.044 DEBUG:boto:path=/qskrps-7/%3A 2023-05-24T21:05:16.044 DEBUG:boto:auth_path=/qskrps-7/%3A 2023-05-24T21:05:16.045 DEBUG:boto:Method: GET 2023-05-24T21:05:16.045 DEBUG:boto:Path: /qskrps-7/%3A 2023-05-24T21:05:16.045 DEBUG:boto:Data: 2023-05-24T21:05:16.046 DEBUG:boto:Headers: {} 2023-05-24T21:05:16.046 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:05:16.046 DEBUG:boto:Port: 8000 2023-05-24T21:05:16.047 DEBUG:boto:Params: {} 2023-05-24T21:05:16.047 DEBUG:boto:Token: None 2023-05-24T21:05:16.048 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:05:16 GMT /qskrps-7/%3A 2023-05-24T21:05:16.048 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:+pwKxctTGJxgXS5PQ6JXmPcyFzI= 2023-05-24T21:05:16.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:16 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:+pwKxctTGJxgXS5PQ6JXmPcyFzI=', 'Content-Length': '0'} 2023-05-24T21:05:16.050 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f21cc51e35331f26-00646e7c0c-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:16.096 DEBUG:boto:path=/qskrps-7/ 2023-05-24T21:05:16.097 DEBUG:boto:auth_path=/qskrps-7/ 2023-05-24T21:05:16.097 DEBUG:boto:path=/qskrps-7/?prefix=%3A 2023-05-24T21:05:16.098 DEBUG:boto:auth_path=/qskrps-7/?prefix=%3A 2023-05-24T21:05:16.098 DEBUG:boto:Method: GET 2023-05-24T21:05:16.099 DEBUG:boto:Path: /qskrps-7/?prefix=%3A 2023-05-24T21:05:16.099 DEBUG:boto:Data: 2023-05-24T21:05:16.099 DEBUG:boto:Headers: {} 2023-05-24T21:05:16.100 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:05:16.100 DEBUG:boto:Port: 8001 2023-05-24T21:05:16.101 DEBUG:boto:Params: {} 2023-05-24T21:05:16.101 DEBUG:boto:Token: None 2023-05-24T21:05:16.102 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:05:16 GMT /qskrps-7/ 2023-05-24T21:05:16.102 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:wcFyPdIWu8scGBX1Gyp++PpcQB4= 2023-05-24T21:05:16.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:16 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wcFyPdIWu8scGBX1Gyp++PpcQB4=', 'Content-Length': '0'} 2023-05-24T21:05:16.106 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a5ebe099aa476c7-00646e7c0c-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:16.106 DEBUG:boto:b'qskrps-7:1000false:2023-05-24T21:02:54.958Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:05:16.107 DEBUG:boto:path=/qskrps-7/ 2023-05-24T21:05:16.107 DEBUG:boto:auth_path=/qskrps-7/ 2023-05-24T21:05:16.108 DEBUG:boto:path=/qskrps-7/?prefix=%3A 2023-05-24T21:05:16.108 DEBUG:boto:auth_path=/qskrps-7/?prefix=%3A 2023-05-24T21:05:16.108 DEBUG:boto:Method: GET 2023-05-24T21:05:16.109 DEBUG:boto:Path: /qskrps-7/?prefix=%3A 2023-05-24T21:05:16.109 DEBUG:boto:Data: 2023-05-24T21:05:16.109 DEBUG:boto:Headers: {} 2023-05-24T21:05:16.110 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:05:16.110 DEBUG:boto:Port: 8000 2023-05-24T21:05:16.111 DEBUG:boto:Params: {} 2023-05-24T21:05:16.111 DEBUG:boto:Token: None 2023-05-24T21:05:16.112 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:05:16 GMT /qskrps-7/ 2023-05-24T21:05:16.112 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:wcFyPdIWu8scGBX1Gyp++PpcQB4= 2023-05-24T21:05:16.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:16 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wcFyPdIWu8scGBX1Gyp++PpcQB4=', 'Content-Length': '0'} 2023-05-24T21:05:16.115 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5b298dfc9de4cd9-00646e7c0c-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:16.115 DEBUG:boto:b'qskrps-7:1000false:2023-05-24T21:02:54.958Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:05:16.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-05-24T21:05:16.117 DEBUG:boto:path=/qskrps-7/_myobj 2023-05-24T21:05:16.117 DEBUG:boto:auth_path=/qskrps-7/_myobj 2023-05-24T21:05:16.117 DEBUG:boto:path=/qskrps-7/_myobj?versionId=null 2023-05-24T21:05:16.118 DEBUG:boto:auth_path=/qskrps-7/_myobj?versionId=null 2023-05-24T21:05:16.118 DEBUG:boto:Method: GET 2023-05-24T21:05:16.118 DEBUG:boto:Path: /qskrps-7/_myobj?versionId=null 2023-05-24T21:05:16.119 DEBUG:boto:Data: 2023-05-24T21:05:16.119 DEBUG:boto:Headers: {} 2023-05-24T21:05:16.120 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:05:16.120 DEBUG:boto:Port: 8001 2023-05-24T21:05:16.120 DEBUG:boto:Params: {} 2023-05-24T21:05:16.121 DEBUG:boto:Token: None 2023-05-24T21:05:16.122 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:05:16 GMT /qskrps-7/_myobj?versionId=null 2023-05-24T21:05:16.122 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:CimUbNkGqRg1VF8n1zTH1Yt/zSY= 2023-05-24T21:05:16.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:16 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:CimUbNkGqRg1VF8n1zTH1Yt/zSY=', 'Content-Length': '0'} 2023-05-24T21:05:16.125 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000024954b04f88be744-00646e7c0c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:16.168 DEBUG:boto:path=/qskrps-7/_myobj 2023-05-24T21:05:16.169 DEBUG:boto:auth_path=/qskrps-7/_myobj 2023-05-24T21:05:16.170 DEBUG:boto:path=/qskrps-7/_myobj?versionId=null 2023-05-24T21:05:16.171 DEBUG:boto:auth_path=/qskrps-7/_myobj?versionId=null 2023-05-24T21:05:16.172 DEBUG:boto:Method: GET 2023-05-24T21:05:16.172 DEBUG:boto:Path: /qskrps-7/_myobj?versionId=null 2023-05-24T21:05:16.173 DEBUG:boto:Data: 2023-05-24T21:05:16.173 DEBUG:boto:Headers: {} 2023-05-24T21:05:16.174 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:05:16.174 DEBUG:boto:Port: 8000 2023-05-24T21:05:16.174 DEBUG:boto:Params: {} 2023-05-24T21:05:16.175 DEBUG:boto:Token: None 2023-05-24T21:05:16.175 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:05:16 GMT /qskrps-7/_myobj?versionId=null 2023-05-24T21:05:16.176 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:CimUbNkGqRg1VF8n1zTH1Yt/zSY= 2023-05-24T21:05:16.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:16 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:CimUbNkGqRg1VF8n1zTH1Yt/zSY=', 'Content-Length': '0'} 2023-05-24T21:05:16.179 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000482b05f7e07ef24f-00646e7c0c-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:16.180 DEBUG:boto:path=/qskrps-7/_myobj 2023-05-24T21:05:16.180 DEBUG:boto:auth_path=/qskrps-7/_myobj 2023-05-24T21:05:16.181 DEBUG:boto:path=/qskrps-7/_myobj?versionId=null 2023-05-24T21:05:16.182 DEBUG:boto:auth_path=/qskrps-7/_myobj?versionId=null 2023-05-24T21:05:16.182 DEBUG:boto:Method: HEAD 2023-05-24T21:05:16.183 DEBUG:boto:Path: /qskrps-7/_myobj?versionId=null 2023-05-24T21:05:16.184 DEBUG:boto:Data: 2023-05-24T21:05:16.184 DEBUG:boto:Headers: {} 2023-05-24T21:05:16.185 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:05:16.186 DEBUG:boto:Port: 8001 2023-05-24T21:05:16.186 DEBUG:boto:Params: {} 2023-05-24T21:05:16.187 DEBUG:boto:Token: None 2023-05-24T21:05:16.188 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:05:16 GMT /qskrps-7/_myobj?versionId=null 2023-05-24T21:05:16.188 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:yCIxhrHTvW5yHPxY5/AqBvkaTLk= 2023-05-24T21:05:16.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:16 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:yCIxhrHTvW5yHPxY5/AqBvkaTLk=', 'Content-Length': '0'} 2023-05-24T21:05:16.191 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c6736b856d65c123-00646e7c0c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:16.192 DEBUG:boto:path=/qskrps-7/_myobj 2023-05-24T21:05:16.193 DEBUG:boto:auth_path=/qskrps-7/_myobj 2023-05-24T21:05:16.194 DEBUG:boto:path=/qskrps-7/_myobj?versionId=null 2023-05-24T21:05:16.194 DEBUG:boto:auth_path=/qskrps-7/_myobj?versionId=null 2023-05-24T21:05:16.195 DEBUG:boto:Method: HEAD 2023-05-24T21:05:16.196 DEBUG:boto:Path: /qskrps-7/_myobj?versionId=null 2023-05-24T21:05:16.196 DEBUG:boto:Data: 2023-05-24T21:05:16.197 DEBUG:boto:Headers: {} 2023-05-24T21:05:16.198 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:05:16.198 DEBUG:boto:Port: 8000 2023-05-24T21:05:16.199 DEBUG:boto:Params: {} 2023-05-24T21:05:16.200 DEBUG:boto:Token: None 2023-05-24T21:05:16.200 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:05:16 GMT /qskrps-7/_myobj?versionId=null 2023-05-24T21:05:16.201 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:yCIxhrHTvW5yHPxY5/AqBvkaTLk= 2023-05-24T21:05:16.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:16 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:yCIxhrHTvW5yHPxY5/AqBvkaTLk=', 'Content-Length': '0'} 2023-05-24T21:05:16.204 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000020394fa2a451d0d2-00646e7c0c-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:16.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-05-24T21:05:16.206 DEBUG:boto:path=/qskrps-7/_myobj 2023-05-24T21:05:16.206 DEBUG:boto:auth_path=/qskrps-7/_myobj 2023-05-24T21:05:16.207 DEBUG:boto:path=/qskrps-7/_myobj?versionId=null 2023-05-24T21:05:16.208 DEBUG:boto:auth_path=/qskrps-7/_myobj?versionId=null 2023-05-24T21:05:16.208 DEBUG:boto:Method: GET 2023-05-24T21:05:16.209 DEBUG:boto:Path: /qskrps-7/_myobj?versionId=null 2023-05-24T21:05:16.210 DEBUG:boto:Data: 2023-05-24T21:05:16.210 DEBUG:boto:Headers: {} 2023-05-24T21:05:16.211 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:05:16.212 DEBUG:boto:Port: 8001 2023-05-24T21:05:16.212 DEBUG:boto:Params: {} 2023-05-24T21:05:16.213 DEBUG:boto:Token: None 2023-05-24T21:05:16.214 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:05:16 GMT /qskrps-7/_myobj?versionId=null 2023-05-24T21:05:16.215 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:CimUbNkGqRg1VF8n1zTH1Yt/zSY= 2023-05-24T21:05:16.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:16 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:CimUbNkGqRg1VF8n1zTH1Yt/zSY=', 'Content-Length': '0'} 2023-05-24T21:05:16.218 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e8f71a6bc9f666ab-00646e7c0c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:16.219 DEBUG:boto:path=/qskrps-7/_myobj 2023-05-24T21:05:16.219 DEBUG:boto:auth_path=/qskrps-7/_myobj 2023-05-24T21:05:16.220 DEBUG:boto:path=/qskrps-7/_myobj?versionId=null 2023-05-24T21:05:16.221 DEBUG:boto:auth_path=/qskrps-7/_myobj?versionId=null 2023-05-24T21:05:16.221 DEBUG:boto:Method: GET 2023-05-24T21:05:16.222 DEBUG:boto:Path: /qskrps-7/_myobj?versionId=null 2023-05-24T21:05:16.223 DEBUG:boto:Data: 2023-05-24T21:05:16.223 DEBUG:boto:Headers: {} 2023-05-24T21:05:16.224 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:05:16.225 DEBUG:boto:Port: 8000 2023-05-24T21:05:16.225 DEBUG:boto:Params: {} 2023-05-24T21:05:16.226 DEBUG:boto:Token: None 2023-05-24T21:05:16.227 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:05:16 GMT /qskrps-7/_myobj?versionId=null 2023-05-24T21:05:16.228 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:CimUbNkGqRg1VF8n1zTH1Yt/zSY= 2023-05-24T21:05:16.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:16 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:CimUbNkGqRg1VF8n1zTH1Yt/zSY=', 'Content-Length': '0'} 2023-05-24T21:05:16.231 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d899d682d5407988-00646e7c0c-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:16.232 DEBUG:boto:path=/qskrps-7/ 2023-05-24T21:05:16.232 DEBUG:boto:auth_path=/qskrps-7/ 2023-05-24T21:05:16.233 DEBUG:boto:path=/qskrps-7/?prefix=_myobj 2023-05-24T21:05:16.234 DEBUG:boto:auth_path=/qskrps-7/?prefix=_myobj 2023-05-24T21:05:16.234 DEBUG:boto:Method: GET 2023-05-24T21:05:16.235 DEBUG:boto:Path: /qskrps-7/?prefix=_myobj 2023-05-24T21:05:16.236 DEBUG:boto:Data: 2023-05-24T21:05:16.237 DEBUG:boto:Headers: {} 2023-05-24T21:05:16.237 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:05:16.238 DEBUG:boto:Port: 8001 2023-05-24T21:05:16.239 DEBUG:boto:Params: {} 2023-05-24T21:05:16.239 DEBUG:boto:Token: None 2023-05-24T21:05:16.240 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:05:16 GMT /qskrps-7/ 2023-05-24T21:05:16.241 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:wcFyPdIWu8scGBX1Gyp++PpcQB4= 2023-05-24T21:05:16.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:16 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wcFyPdIWu8scGBX1Gyp++PpcQB4=', 'Content-Length': '0'} 2023-05-24T21:05:16.245 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e372303c2db460c7-00646e7c0c-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:16.246 DEBUG:boto:b'qskrps-7_myobj1000false_myobj2023-05-24T21:02:54.897Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:05:16.246 DEBUG:boto:path=/qskrps-7/ 2023-05-24T21:05:16.247 DEBUG:boto:auth_path=/qskrps-7/ 2023-05-24T21:05:16.247 DEBUG:boto:path=/qskrps-7/?prefix=_myobj 2023-05-24T21:05:16.248 DEBUG:boto:auth_path=/qskrps-7/?prefix=_myobj 2023-05-24T21:05:16.248 DEBUG:boto:Method: GET 2023-05-24T21:05:16.248 DEBUG:boto:Path: /qskrps-7/?prefix=_myobj 2023-05-24T21:05:16.249 DEBUG:boto:Data: 2023-05-24T21:05:16.249 DEBUG:boto:Headers: {} 2023-05-24T21:05:16.249 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:05:16.250 DEBUG:boto:Port: 8000 2023-05-24T21:05:16.250 DEBUG:boto:Params: {} 2023-05-24T21:05:16.251 DEBUG:boto:Token: None 2023-05-24T21:05:16.251 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:05:16 GMT /qskrps-7/ 2023-05-24T21:05:16.251 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:wcFyPdIWu8scGBX1Gyp++PpcQB4= 2023-05-24T21:05:16.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:16 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wcFyPdIWu8scGBX1Gyp++PpcQB4=', 'Content-Length': '0'} 2023-05-24T21:05:16.254 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d98e962bda135c5d-00646e7c0c-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:16.255 DEBUG:boto:b'qskrps-7_myobj1000false_myobj2023-05-24T21:02:54.897Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:05:16.256 DEBUG:boto:path=/qskrps-7/_myobj 2023-05-24T21:05:16.256 DEBUG:boto:auth_path=/qskrps-7/_myobj 2023-05-24T21:05:16.256 DEBUG:boto:Method: HEAD 2023-05-24T21:05:16.257 DEBUG:boto:Path: /qskrps-7/_myobj 2023-05-24T21:05:16.257 DEBUG:boto:Data: 2023-05-24T21:05:16.257 DEBUG:boto:Headers: {} 2023-05-24T21:05:16.258 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:05:16.258 DEBUG:boto:Port: 8001 2023-05-24T21:05:16.258 DEBUG:boto:Params: {} 2023-05-24T21:05:16.259 DEBUG:boto:Token: None 2023-05-24T21:05:16.259 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:05:16 GMT /qskrps-7/_myobj 2023-05-24T21:05:16.260 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:JCsB6LZ9x0cS46M9jCI8Ei6r0xY= 2023-05-24T21:05:16.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:16 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:JCsB6LZ9x0cS46M9jCI8Ei6r0xY=', 'Content-Length': '0'} 2023-05-24T21:05:16.262 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000079ddb5602de894a4-00646e7c0c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:16.262 DEBUG:boto:path=/qskrps-7/_myobj 2023-05-24T21:05:16.263 DEBUG:boto:auth_path=/qskrps-7/_myobj 2023-05-24T21:05:16.263 DEBUG:boto:Method: HEAD 2023-05-24T21:05:16.263 DEBUG:boto:Path: /qskrps-7/_myobj 2023-05-24T21:05:16.264 DEBUG:boto:Data: 2023-05-24T21:05:16.264 DEBUG:boto:Headers: {} 2023-05-24T21:05:16.264 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:05:16.265 DEBUG:boto:Port: 8000 2023-05-24T21:05:16.265 DEBUG:boto:Params: {} 2023-05-24T21:05:16.266 DEBUG:boto:Token: None 2023-05-24T21:05:16.266 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:05:16 GMT /qskrps-7/_myobj 2023-05-24T21:05:16.266 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:JCsB6LZ9x0cS46M9jCI8Ei6r0xY= 2023-05-24T21:05:16.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:16 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:JCsB6LZ9x0cS46M9jCI8Ei6r0xY=', 'Content-Length': '0'} 2023-05-24T21:05:16.269 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b70952e82e21b9a5-00646e7c0c-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:16.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-05-24T21:05:16.270 DEBUG:boto:path=/qskrps-7/_myobj 2023-05-24T21:05:16.270 DEBUG:boto:auth_path=/qskrps-7/_myobj 2023-05-24T21:05:16.270 DEBUG:boto:Method: GET 2023-05-24T21:05:16.271 DEBUG:boto:Path: /qskrps-7/_myobj 2023-05-24T21:05:16.271 DEBUG:boto:Data: 2023-05-24T21:05:16.271 DEBUG:boto:Headers: {} 2023-05-24T21:05:16.272 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:05:16.272 DEBUG:boto:Port: 8001 2023-05-24T21:05:16.272 DEBUG:boto:Params: {} 2023-05-24T21:05:16.273 DEBUG:boto:Token: None 2023-05-24T21:05:16.273 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:05:16 GMT /qskrps-7/_myobj 2023-05-24T21:05:16.274 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:c+Vln5N0pIREzoHFMY6RcWCm3fU= 2023-05-24T21:05:16.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:16 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:c+Vln5N0pIREzoHFMY6RcWCm3fU=', 'Content-Length': '0'} 2023-05-24T21:05:16.276 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf43899578f0aacd-00646e7c0c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:16.276 DEBUG:boto:path=/qskrps-7/_myobj 2023-05-24T21:05:16.277 DEBUG:boto:auth_path=/qskrps-7/_myobj 2023-05-24T21:05:16.277 DEBUG:boto:Method: GET 2023-05-24T21:05:16.278 DEBUG:boto:Path: /qskrps-7/_myobj 2023-05-24T21:05:16.278 DEBUG:boto:Data: 2023-05-24T21:05:16.278 DEBUG:boto:Headers: {} 2023-05-24T21:05:16.279 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:05:16.279 DEBUG:boto:Port: 8000 2023-05-24T21:05:16.279 DEBUG:boto:Params: {} 2023-05-24T21:05:16.280 DEBUG:boto:Token: None 2023-05-24T21:05:16.280 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:05:16 GMT /qskrps-7/_myobj 2023-05-24T21:05:16.281 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:c+Vln5N0pIREzoHFMY6RcWCm3fU= 2023-05-24T21:05:16.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:16 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:c+Vln5N0pIREzoHFMY6RcWCm3fU=', 'Content-Length': '0'} 2023-05-24T21:05:16.283 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000059d84188220d2db6-00646e7c0c-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:16.283 DEBUG:boto:path=/qskrps-7/ 2023-05-24T21:05:16.284 DEBUG:boto:auth_path=/qskrps-7/ 2023-05-24T21:05:16.284 DEBUG:boto:path=/qskrps-7/?prefix=_myobj 2023-05-24T21:05:16.285 DEBUG:boto:auth_path=/qskrps-7/?prefix=_myobj 2023-05-24T21:05:16.285 DEBUG:boto:Method: GET 2023-05-24T21:05:16.285 DEBUG:boto:Path: /qskrps-7/?prefix=_myobj 2023-05-24T21:05:16.286 DEBUG:boto:Data: 2023-05-24T21:05:16.286 DEBUG:boto:Headers: {} 2023-05-24T21:05:16.286 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:05:16.287 DEBUG:boto:Port: 8001 2023-05-24T21:05:16.287 DEBUG:boto:Params: {} 2023-05-24T21:05:16.287 DEBUG:boto:Token: None 2023-05-24T21:05:16.288 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:05:16 GMT /qskrps-7/ 2023-05-24T21:05:16.288 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:wcFyPdIWu8scGBX1Gyp++PpcQB4= 2023-05-24T21:05:16.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:16 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wcFyPdIWu8scGBX1Gyp++PpcQB4=', 'Content-Length': '0'} 2023-05-24T21:05:16.291 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b293f4b95509de4-00646e7c0c-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:16.292 DEBUG:boto:b'qskrps-7_myobj1000false_myobj2023-05-24T21:02:54.897Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:05:16.292 DEBUG:boto:path=/qskrps-7/ 2023-05-24T21:05:16.293 DEBUG:boto:auth_path=/qskrps-7/ 2023-05-24T21:05:16.293 DEBUG:boto:path=/qskrps-7/?prefix=_myobj 2023-05-24T21:05:16.294 DEBUG:boto:auth_path=/qskrps-7/?prefix=_myobj 2023-05-24T21:05:16.294 DEBUG:boto:Method: GET 2023-05-24T21:05:16.294 DEBUG:boto:Path: /qskrps-7/?prefix=_myobj 2023-05-24T21:05:16.295 DEBUG:boto:Data: 2023-05-24T21:05:16.295 DEBUG:boto:Headers: {} 2023-05-24T21:05:16.295 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:05:16.296 DEBUG:boto:Port: 8000 2023-05-24T21:05:16.296 DEBUG:boto:Params: {} 2023-05-24T21:05:16.296 DEBUG:boto:Token: None 2023-05-24T21:05:16.297 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:05:16 GMT /qskrps-7/ 2023-05-24T21:05:16.297 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:wcFyPdIWu8scGBX1Gyp++PpcQB4= 2023-05-24T21:05:16.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:16 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wcFyPdIWu8scGBX1Gyp++PpcQB4=', 'Content-Length': '0'} 2023-05-24T21:05:16.300 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cca9e17e17820ad7-00646e7c0c-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:16.301 DEBUG:boto:b'qskrps-7_myobj1000false_myobj2023-05-24T21:02:54.897Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:05:16.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-05-24T21:05:16.302 DEBUG:boto:path=/qskrps-7/myobj 2023-05-24T21:05:16.302 DEBUG:boto:auth_path=/qskrps-7/myobj 2023-05-24T21:05:16.303 DEBUG:boto:path=/qskrps-7/myobj?versionId=null 2023-05-24T21:05:16.303 DEBUG:boto:auth_path=/qskrps-7/myobj?versionId=null 2023-05-24T21:05:16.303 DEBUG:boto:Method: GET 2023-05-24T21:05:16.304 DEBUG:boto:Path: /qskrps-7/myobj?versionId=null 2023-05-24T21:05:16.304 DEBUG:boto:Data: 2023-05-24T21:05:16.304 DEBUG:boto:Headers: {} 2023-05-24T21:05:16.305 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:05:16.305 DEBUG:boto:Port: 8001 2023-05-24T21:05:16.305 DEBUG:boto:Params: {} 2023-05-24T21:05:16.306 DEBUG:boto:Token: None 2023-05-24T21:05:16.306 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:05:16 GMT /qskrps-7/myobj?versionId=null 2023-05-24T21:05:16.307 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:MnBM09PNfyROkZbQLaaeXm0aopM= 2023-05-24T21:05:16.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:16 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:MnBM09PNfyROkZbQLaaeXm0aopM=', 'Content-Length': '0'} 2023-05-24T21:05:16.309 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000032bd02e0631c6dbc-00646e7c0c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:16.310 DEBUG:boto:path=/qskrps-7/myobj 2023-05-24T21:05:16.310 DEBUG:boto:auth_path=/qskrps-7/myobj 2023-05-24T21:05:16.310 DEBUG:boto:path=/qskrps-7/myobj?versionId=null 2023-05-24T21:05:16.311 DEBUG:boto:auth_path=/qskrps-7/myobj?versionId=null 2023-05-24T21:05:16.311 DEBUG:boto:Method: GET 2023-05-24T21:05:16.311 DEBUG:boto:Path: /qskrps-7/myobj?versionId=null 2023-05-24T21:05:16.312 DEBUG:boto:Data: 2023-05-24T21:05:16.312 DEBUG:boto:Headers: {} 2023-05-24T21:05:16.313 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:05:16.313 DEBUG:boto:Port: 8000 2023-05-24T21:05:16.313 DEBUG:boto:Params: {} 2023-05-24T21:05:16.314 DEBUG:boto:Token: None 2023-05-24T21:05:16.314 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:05:16 GMT /qskrps-7/myobj?versionId=null 2023-05-24T21:05:16.315 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:MnBM09PNfyROkZbQLaaeXm0aopM= 2023-05-24T21:05:16.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:16 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:MnBM09PNfyROkZbQLaaeXm0aopM=', 'Content-Length': '0'} 2023-05-24T21:05:16.317 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005b56226cfa851eca-00646e7c0c-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:16.364 DEBUG:boto:path=/qskrps-7/myobj 2023-05-24T21:05:16.365 DEBUG:boto:auth_path=/qskrps-7/myobj 2023-05-24T21:05:16.365 DEBUG:boto:path=/qskrps-7/myobj?versionId=null 2023-05-24T21:05:16.365 DEBUG:boto:auth_path=/qskrps-7/myobj?versionId=null 2023-05-24T21:05:16.366 DEBUG:boto:Method: HEAD 2023-05-24T21:05:16.366 DEBUG:boto:Path: /qskrps-7/myobj?versionId=null 2023-05-24T21:05:16.366 DEBUG:boto:Data: 2023-05-24T21:05:16.367 DEBUG:boto:Headers: {} 2023-05-24T21:05:16.367 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:05:16.367 DEBUG:boto:Port: 8001 2023-05-24T21:05:16.368 DEBUG:boto:Params: {} 2023-05-24T21:05:16.368 DEBUG:boto:Token: None 2023-05-24T21:05:16.368 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:05:16 GMT /qskrps-7/myobj?versionId=null 2023-05-24T21:05:16.369 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:eX9aRqBn5a7tXvRmddvMmCRwYR4= 2023-05-24T21:05:16.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:16 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:eX9aRqBn5a7tXvRmddvMmCRwYR4=', 'Content-Length': '0'} 2023-05-24T21:05:16.371 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000938a2ed05be89ca6-00646e7c0c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:16.372 DEBUG:boto:path=/qskrps-7/myobj 2023-05-24T21:05:16.372 DEBUG:boto:auth_path=/qskrps-7/myobj 2023-05-24T21:05:16.373 DEBUG:boto:path=/qskrps-7/myobj?versionId=null 2023-05-24T21:05:16.373 DEBUG:boto:auth_path=/qskrps-7/myobj?versionId=null 2023-05-24T21:05:16.373 DEBUG:boto:Method: HEAD 2023-05-24T21:05:16.374 DEBUG:boto:Path: /qskrps-7/myobj?versionId=null 2023-05-24T21:05:16.374 DEBUG:boto:Data: 2023-05-24T21:05:16.374 DEBUG:boto:Headers: {} 2023-05-24T21:05:16.375 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:05:16.375 DEBUG:boto:Port: 8000 2023-05-24T21:05:16.375 DEBUG:boto:Params: {} 2023-05-24T21:05:16.376 DEBUG:boto:Token: None 2023-05-24T21:05:16.376 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:05:16 GMT /qskrps-7/myobj?versionId=null 2023-05-24T21:05:16.376 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:eX9aRqBn5a7tXvRmddvMmCRwYR4= 2023-05-24T21:05:16.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:16 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:eX9aRqBn5a7tXvRmddvMmCRwYR4=', 'Content-Length': '0'} 2023-05-24T21:05:16.378 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ff4b55a1589c3d2f-00646e7c0c-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:16.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-05-24T21:05:16.379 DEBUG:boto:path=/qskrps-7/myobj 2023-05-24T21:05:16.380 DEBUG:boto:auth_path=/qskrps-7/myobj 2023-05-24T21:05:16.380 DEBUG:boto:path=/qskrps-7/myobj?versionId=null 2023-05-24T21:05:16.380 DEBUG:boto:auth_path=/qskrps-7/myobj?versionId=null 2023-05-24T21:05:16.381 DEBUG:boto:Method: GET 2023-05-24T21:05:16.381 DEBUG:boto:Path: /qskrps-7/myobj?versionId=null 2023-05-24T21:05:16.382 DEBUG:boto:Data: 2023-05-24T21:05:16.382 DEBUG:boto:Headers: {} 2023-05-24T21:05:16.382 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:05:16.383 DEBUG:boto:Port: 8001 2023-05-24T21:05:16.383 DEBUG:boto:Params: {} 2023-05-24T21:05:16.383 DEBUG:boto:Token: None 2023-05-24T21:05:16.384 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:05:16 GMT /qskrps-7/myobj?versionId=null 2023-05-24T21:05:16.384 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:MnBM09PNfyROkZbQLaaeXm0aopM= 2023-05-24T21:05:16.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:16 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:MnBM09PNfyROkZbQLaaeXm0aopM=', 'Content-Length': '0'} 2023-05-24T21:05:16.386 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b10e2e99eff50951-00646e7c0c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:16.387 DEBUG:boto:path=/qskrps-7/myobj 2023-05-24T21:05:16.388 DEBUG:boto:auth_path=/qskrps-7/myobj 2023-05-24T21:05:16.388 DEBUG:boto:path=/qskrps-7/myobj?versionId=null 2023-05-24T21:05:16.388 DEBUG:boto:auth_path=/qskrps-7/myobj?versionId=null 2023-05-24T21:05:16.389 DEBUG:boto:Method: GET 2023-05-24T21:05:16.389 DEBUG:boto:Path: /qskrps-7/myobj?versionId=null 2023-05-24T21:05:16.389 DEBUG:boto:Data: 2023-05-24T21:05:16.390 DEBUG:boto:Headers: {} 2023-05-24T21:05:16.390 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:05:16.390 DEBUG:boto:Port: 8000 2023-05-24T21:05:16.391 DEBUG:boto:Params: {} 2023-05-24T21:05:16.391 DEBUG:boto:Token: None 2023-05-24T21:05:16.392 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:05:16 GMT /qskrps-7/myobj?versionId=null 2023-05-24T21:05:16.392 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:MnBM09PNfyROkZbQLaaeXm0aopM= 2023-05-24T21:05:16.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:16 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:MnBM09PNfyROkZbQLaaeXm0aopM=', 'Content-Length': '0'} 2023-05-24T21:05:16.394 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001fd533c74d207078-00646e7c0c-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:16.395 DEBUG:boto:path=/qskrps-7/ 2023-05-24T21:05:16.395 DEBUG:boto:auth_path=/qskrps-7/ 2023-05-24T21:05:16.396 DEBUG:boto:path=/qskrps-7/?prefix=myobj 2023-05-24T21:05:16.396 DEBUG:boto:auth_path=/qskrps-7/?prefix=myobj 2023-05-24T21:05:16.396 DEBUG:boto:Method: GET 2023-05-24T21:05:16.397 DEBUG:boto:Path: /qskrps-7/?prefix=myobj 2023-05-24T21:05:16.397 DEBUG:boto:Data: 2023-05-24T21:05:16.397 DEBUG:boto:Headers: {} 2023-05-24T21:05:16.398 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:05:16.398 DEBUG:boto:Port: 8001 2023-05-24T21:05:16.398 DEBUG:boto:Params: {} 2023-05-24T21:05:16.399 DEBUG:boto:Token: None 2023-05-24T21:05:16.399 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:05:16 GMT /qskrps-7/ 2023-05-24T21:05:16.400 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:wcFyPdIWu8scGBX1Gyp++PpcQB4= 2023-05-24T21:05:16.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:16 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wcFyPdIWu8scGBX1Gyp++PpcQB4=', 'Content-Length': '0'} 2023-05-24T21:05:16.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be13851d6ec5632f-00646e7c0c-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:16.403 DEBUG:boto:b'qskrps-7myobj1000falsemyobj2023-05-24T21:02:54.852Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:05:16.404 DEBUG:boto:path=/qskrps-7/ 2023-05-24T21:05:16.404 DEBUG:boto:auth_path=/qskrps-7/ 2023-05-24T21:05:16.405 DEBUG:boto:path=/qskrps-7/?prefix=myobj 2023-05-24T21:05:16.405 DEBUG:boto:auth_path=/qskrps-7/?prefix=myobj 2023-05-24T21:05:16.405 DEBUG:boto:Method: GET 2023-05-24T21:05:16.406 DEBUG:boto:Path: /qskrps-7/?prefix=myobj 2023-05-24T21:05:16.406 DEBUG:boto:Data: 2023-05-24T21:05:16.406 DEBUG:boto:Headers: {} 2023-05-24T21:05:16.407 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:05:16.407 DEBUG:boto:Port: 8000 2023-05-24T21:05:16.407 DEBUG:boto:Params: {} 2023-05-24T21:05:16.408 DEBUG:boto:Token: None 2023-05-24T21:05:16.408 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:05:16 GMT /qskrps-7/ 2023-05-24T21:05:16.409 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:wcFyPdIWu8scGBX1Gyp++PpcQB4= 2023-05-24T21:05:16.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:16 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wcFyPdIWu8scGBX1Gyp++PpcQB4=', 'Content-Length': '0'} 2023-05-24T21:05:16.411 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b073aba1b246035d-00646e7c0c-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:16.412 DEBUG:boto:b'qskrps-7myobj1000falsemyobj2023-05-24T21:02:54.852Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:05:16.412 DEBUG:boto:path=/qskrps-7/myobj 2023-05-24T21:05:16.413 DEBUG:boto:auth_path=/qskrps-7/myobj 2023-05-24T21:05:16.413 DEBUG:boto:Method: HEAD 2023-05-24T21:05:16.413 DEBUG:boto:Path: /qskrps-7/myobj 2023-05-24T21:05:16.414 DEBUG:boto:Data: 2023-05-24T21:05:16.414 DEBUG:boto:Headers: {} 2023-05-24T21:05:16.414 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:05:16.415 DEBUG:boto:Port: 8001 2023-05-24T21:05:16.415 DEBUG:boto:Params: {} 2023-05-24T21:05:16.415 DEBUG:boto:Token: None 2023-05-24T21:05:16.416 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:05:16 GMT /qskrps-7/myobj 2023-05-24T21:05:16.416 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:zvdclmWOd3WRZxtmm+RJRK0l6FM= 2023-05-24T21:05:16.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:16 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:zvdclmWOd3WRZxtmm+RJRK0l6FM=', 'Content-Length': '0'} 2023-05-24T21:05:16.419 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000de463c823147b9b5-00646e7c0c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:16.420 DEBUG:boto:path=/qskrps-7/myobj 2023-05-24T21:05:16.420 DEBUG:boto:auth_path=/qskrps-7/myobj 2023-05-24T21:05:16.421 DEBUG:boto:Method: HEAD 2023-05-24T21:05:16.421 DEBUG:boto:Path: /qskrps-7/myobj 2023-05-24T21:05:16.421 DEBUG:boto:Data: 2023-05-24T21:05:16.422 DEBUG:boto:Headers: {} 2023-05-24T21:05:16.422 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:05:16.422 DEBUG:boto:Port: 8000 2023-05-24T21:05:16.423 DEBUG:boto:Params: {} 2023-05-24T21:05:16.424 DEBUG:boto:Token: None 2023-05-24T21:05:16.424 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:05:16 GMT /qskrps-7/myobj 2023-05-24T21:05:16.425 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:zvdclmWOd3WRZxtmm+RJRK0l6FM= 2023-05-24T21:05:16.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:16 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:zvdclmWOd3WRZxtmm+RJRK0l6FM=', 'Content-Length': '0'} 2023-05-24T21:05:16.428 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005fcea20ceea8e7a0-00646e7c0c-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:16.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-05-24T21:05:16.430 DEBUG:boto:path=/qskrps-7/myobj 2023-05-24T21:05:16.430 DEBUG:boto:auth_path=/qskrps-7/myobj 2023-05-24T21:05:16.431 DEBUG:boto:Method: GET 2023-05-24T21:05:16.431 DEBUG:boto:Path: /qskrps-7/myobj 2023-05-24T21:05:16.432 DEBUG:boto:Data: 2023-05-24T21:05:16.433 DEBUG:boto:Headers: {} 2023-05-24T21:05:16.433 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:05:16.434 DEBUG:boto:Port: 8001 2023-05-24T21:05:16.434 DEBUG:boto:Params: {} 2023-05-24T21:05:16.434 DEBUG:boto:Token: None 2023-05-24T21:05:16.435 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:05:16 GMT /qskrps-7/myobj 2023-05-24T21:05:16.435 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:CXZrvgQ6nY/DNzXRrv5ORftnLqY= 2023-05-24T21:05:16.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:16 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:CXZrvgQ6nY/DNzXRrv5ORftnLqY=', 'Content-Length': '0'} 2023-05-24T21:05:16.438 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003664c4c7c9d3f4e7-00646e7c0c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:16.439 DEBUG:boto:path=/qskrps-7/myobj 2023-05-24T21:05:16.439 DEBUG:boto:auth_path=/qskrps-7/myobj 2023-05-24T21:05:16.439 DEBUG:boto:Method: GET 2023-05-24T21:05:16.440 DEBUG:boto:Path: /qskrps-7/myobj 2023-05-24T21:05:16.440 DEBUG:boto:Data: 2023-05-24T21:05:16.440 DEBUG:boto:Headers: {} 2023-05-24T21:05:16.441 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:05:16.441 DEBUG:boto:Port: 8000 2023-05-24T21:05:16.442 DEBUG:boto:Params: {} 2023-05-24T21:05:16.442 DEBUG:boto:Token: None 2023-05-24T21:05:16.442 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:05:16 GMT /qskrps-7/myobj 2023-05-24T21:05:16.443 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:CXZrvgQ6nY/DNzXRrv5ORftnLqY= 2023-05-24T21:05:16.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:16 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:CXZrvgQ6nY/DNzXRrv5ORftnLqY=', 'Content-Length': '0'} 2023-05-24T21:05:16.445 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007716650de2312271-00646e7c0c-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:16.446 DEBUG:boto:path=/qskrps-7/ 2023-05-24T21:05:16.446 DEBUG:boto:auth_path=/qskrps-7/ 2023-05-24T21:05:16.446 DEBUG:boto:path=/qskrps-7/?prefix=myobj 2023-05-24T21:05:16.447 DEBUG:boto:auth_path=/qskrps-7/?prefix=myobj 2023-05-24T21:05:16.447 DEBUG:boto:Method: GET 2023-05-24T21:05:16.447 DEBUG:boto:Path: /qskrps-7/?prefix=myobj 2023-05-24T21:05:16.448 DEBUG:boto:Data: 2023-05-24T21:05:16.448 DEBUG:boto:Headers: {} 2023-05-24T21:05:16.449 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:05:16.449 DEBUG:boto:Port: 8001 2023-05-24T21:05:16.449 DEBUG:boto:Params: {} 2023-05-24T21:05:16.450 DEBUG:boto:Token: None 2023-05-24T21:05:16.450 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:05:16 GMT /qskrps-7/ 2023-05-24T21:05:16.451 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:wcFyPdIWu8scGBX1Gyp++PpcQB4= 2023-05-24T21:05:16.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:16 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wcFyPdIWu8scGBX1Gyp++PpcQB4=', 'Content-Length': '0'} 2023-05-24T21:05:16.453 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8a832d772a5c364-00646e7c0c-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:16.454 DEBUG:boto:b'qskrps-7myobj1000falsemyobj2023-05-24T21:02:54.852Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:05:16.455 DEBUG:boto:path=/qskrps-7/ 2023-05-24T21:05:16.455 DEBUG:boto:auth_path=/qskrps-7/ 2023-05-24T21:05:16.456 DEBUG:boto:path=/qskrps-7/?prefix=myobj 2023-05-24T21:05:16.456 DEBUG:boto:auth_path=/qskrps-7/?prefix=myobj 2023-05-24T21:05:16.456 DEBUG:boto:Method: GET 2023-05-24T21:05:16.457 DEBUG:boto:Path: /qskrps-7/?prefix=myobj 2023-05-24T21:05:16.457 DEBUG:boto:Data: 2023-05-24T21:05:16.458 DEBUG:boto:Headers: {} 2023-05-24T21:05:16.458 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:05:16.458 DEBUG:boto:Port: 8000 2023-05-24T21:05:16.459 DEBUG:boto:Params: {} 2023-05-24T21:05:16.459 DEBUG:boto:Token: None 2023-05-24T21:05:16.460 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:05:16 GMT /qskrps-7/ 2023-05-24T21:05:16.461 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:wcFyPdIWu8scGBX1Gyp++PpcQB4= 2023-05-24T21:05:16.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:05:16 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wcFyPdIWu8scGBX1Gyp++PpcQB4=', 'Content-Length': '0'} 2023-05-24T21:05:16.465 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007093b16c9e7d8a94-00646e7c0c-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:05:16.465 DEBUG:boto:b'qskrps-7myobj1000falsemyobj2023-05-24T21:02:54.852Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:05:16.466 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qskrps-7 zones={a2, a1} 2023-05-24T21:05:16.466 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-8', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:05:16.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-8', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:05:16.467 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-8 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:05:16.531 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:05:16.531 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:05:16.567 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:05:16.560+0000 7ff36b49f780 1 waiting to reach incremental sync.. 2023-05-24T21:05:21.567 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:05:21.561+0000 7ff36b49f780 1 waiting to reach incremental sync.. 2023-05-24T21:05:26.568 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:05:26.561+0000 7ff36b49f780 1 waiting to reach incremental sync.. 2023-05-24T21:05:31.568 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:05:31.562+0000 7ff36b49f780 1 waiting to reach incremental sync.. 2023-05-24T21:05:36.569 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:05:36.563+0000 7ff36b49f780 1 waiting to reach incremental sync.. 2023-05-24T21:05:41.570 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:05:41.564+0000 7ff36b49f780 1 waiting to reach incremental sync.. 2023-05-24T21:05:46.570 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:05:46.564+0000 7ff36b49f780 1 waiting to reach incremental sync.. 2023-05-24T21:05:51.571 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:05:51.565+0000 7ff36b49f780 1 waiting to reach incremental sync.. 2023-05-24T21:05:56.572 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:05:56.566+0000 7ff36b49f780 1 waiting to reach incremental sync.. 2023-05-24T21:06:01.572 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:06:01.565+0000 7ff36b49f780 1 waiting to reach incremental sync.. 2023-05-24T21:06:06.573 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:06:06.566+0000 7ff36b49f780 1 waiting to reach incremental sync.. 2023-05-24T21:06:11.574 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:06:11.567+0000 7ff36b49f780 1 waiting to reach incremental sync.. 2023-05-24T21:06:16.574 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:06:16.568+0000 7ff36b49f780 1 waiting to reach incremental sync.. 2023-05-24T21:06:21.575 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:06:21.568+0000 7ff36b49f780 1 waiting to reach incremental sync.. 2023-05-24T21:06:26.575 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:06:26.569+0000 7ff36b49f780 1 waiting to reach incremental sync.. 2023-05-24T21:06:31.577 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:06:31.571+0000 7ff36b49f780 1 bucket sync caught up with source: 2023-05-24T21:06:31.577 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000001.7.6, , 00000000001.9.6, , , , 00000000001.9.6, 00000000001.9.6, , ] 2023-05-24T21:06:31.578 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000001.7.6, , 00000000001.9.6, , , , 00000000001.9.6, 00000000001.9.6, , ] 2023-05-24T21:06:31.578 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:06:31.571+0000 7ff36b49f780 0 bucket checkpoint complete 2023-05-24T21:06:31.597 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qskrps-8 zones={a1, a2} 2023-05-24T21:06:31.598 DEBUG:boto:path=/qskrps-8/ 2023-05-24T21:06:31.598 DEBUG:boto:auth_path=/qskrps-8/ 2023-05-24T21:06:31.598 DEBUG:boto:Method: HEAD 2023-05-24T21:06:31.599 DEBUG:boto:Path: /qskrps-8/ 2023-05-24T21:06:31.599 DEBUG:boto:Data: 2023-05-24T21:06:31.599 DEBUG:boto:Headers: {} 2023-05-24T21:06:31.600 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:31.600 DEBUG:boto:Port: 8000 2023-05-24T21:06:31.601 DEBUG:boto:Params: {} 2023-05-24T21:06:31.601 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T21:06:31.602 DEBUG:boto:Token: None 2023-05-24T21:06:31.602 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:06:31 GMT /qskrps-8/ 2023-05-24T21:06:31.603 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:eGjLeblmGiJci+SE5KB+R7jljt8= 2023-05-24T21:06:31.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:31 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:eGjLeblmGiJci+SE5KB+R7jljt8=', 'Content-Length': '0'} 2023-05-24T21:06:31.607 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000008fd4bf43717e6f-00646e7c57-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:06:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:31.607 DEBUG:boto:path=/qskrps-8/ 2023-05-24T21:06:31.608 DEBUG:boto:auth_path=/qskrps-8/ 2023-05-24T21:06:31.608 DEBUG:boto:Method: HEAD 2023-05-24T21:06:31.608 DEBUG:boto:Path: /qskrps-8/ 2023-05-24T21:06:31.609 DEBUG:boto:Data: 2023-05-24T21:06:31.609 DEBUG:boto:Headers: {} 2023-05-24T21:06:31.609 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:31.610 DEBUG:boto:Port: 8001 2023-05-24T21:06:31.610 DEBUG:boto:Params: {} 2023-05-24T21:06:31.611 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-24T21:06:31.611 DEBUG:boto:Token: None 2023-05-24T21:06:31.611 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:06:31 GMT /qskrps-8/ 2023-05-24T21:06:31.612 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:eGjLeblmGiJci+SE5KB+R7jljt8= 2023-05-24T21:06:31.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:31 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:eGjLeblmGiJci+SE5KB+R7jljt8=', 'Content-Length': '0'} 2023-05-24T21:06:31.616 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002edde9799a9f5cfd-00646e7c57-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:06:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:31.616 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-24T21:06:31.617 DEBUG:boto:path=/qskrps-8/ 2023-05-24T21:06:31.617 DEBUG:boto:auth_path=/qskrps-8/ 2023-05-24T21:06:31.617 DEBUG:boto:path=/qskrps-8/?versions&max-keys=999 2023-05-24T21:06:31.618 DEBUG:boto:auth_path=/qskrps-8/?versions&max-keys=999 2023-05-24T21:06:31.618 DEBUG:boto:Method: GET 2023-05-24T21:06:31.618 DEBUG:boto:Path: /qskrps-8/?versions&max-keys=999 2023-05-24T21:06:31.619 DEBUG:boto:Data: 2023-05-24T21:06:31.619 DEBUG:boto:Headers: {} 2023-05-24T21:06:31.619 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:31.620 DEBUG:boto:Port: 8000 2023-05-24T21:06:31.620 DEBUG:boto:Params: {} 2023-05-24T21:06:31.620 DEBUG:boto:Token: None 2023-05-24T21:06:31.621 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:31 GMT /qskrps-8/?versions 2023-05-24T21:06:31.621 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:6k8kYMnUmc0ltoE6V/Ktl0ESC0Y= 2023-05-24T21:06:31.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:31 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:6k8kYMnUmc0ltoE6V/Ktl0ESC0Y=', 'Content-Length': '0'} 2023-05-24T21:06:31.624 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e522ffcea5085a5c-00646e7c57-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:06:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:31.625 DEBUG:boto:b'qskrps-8999false&nulltrue2023-05-24T21:02:55.134Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-05-24T21:02:55.076Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-05-24T21:02:55.045Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-05-24T21:02:55.012Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:06:31.625 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-05-24T21:06:31.626 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-05-24T21:06:31.626 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-05-24T21:06:31.626 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-05-24T21:06:31.627 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-24T21:06:31.627 DEBUG:boto:path=/qskrps-8/ 2023-05-24T21:06:31.627 DEBUG:boto:auth_path=/qskrps-8/ 2023-05-24T21:06:31.628 DEBUG:boto:path=/qskrps-8/?versions&max-keys=999 2023-05-24T21:06:31.628 DEBUG:boto:auth_path=/qskrps-8/?versions&max-keys=999 2023-05-24T21:06:31.629 DEBUG:boto:Method: GET 2023-05-24T21:06:31.629 DEBUG:boto:Path: /qskrps-8/?versions&max-keys=999 2023-05-24T21:06:31.629 DEBUG:boto:Data: 2023-05-24T21:06:31.630 DEBUG:boto:Headers: {} 2023-05-24T21:06:31.630 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:31.630 DEBUG:boto:Port: 8001 2023-05-24T21:06:31.631 DEBUG:boto:Params: {} 2023-05-24T21:06:31.631 DEBUG:boto:Token: None 2023-05-24T21:06:31.631 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:31 GMT /qskrps-8/?versions 2023-05-24T21:06:31.632 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:6k8kYMnUmc0ltoE6V/Ktl0ESC0Y= 2023-05-24T21:06:31.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:31 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:6k8kYMnUmc0ltoE6V/Ktl0ESC0Y=', 'Content-Length': '0'} 2023-05-24T21:06:31.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008279db9b129c1d2e-00646e7c57-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:06:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:31.635 DEBUG:boto:b'qskrps-8999false&nulltrue2023-05-24T21:02:55.134Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-05-24T21:02:55.076Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-05-24T21:02:55.045Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-05-24T21:02:55.012Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:06:31.636 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-05-24T21:06:31.636 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-05-24T21:06:31.637 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-05-24T21:06:31.637 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-05-24T21:06:31.637 DEBUG:boto:path=/qskrps-8/ 2023-05-24T21:06:31.638 DEBUG:boto:auth_path=/qskrps-8/ 2023-05-24T21:06:31.638 DEBUG:boto:path=/qskrps-8/?versions&max-keys=999 2023-05-24T21:06:31.638 DEBUG:boto:auth_path=/qskrps-8/?versions&max-keys=999 2023-05-24T21:06:31.639 DEBUG:boto:Method: GET 2023-05-24T21:06:31.639 DEBUG:boto:Path: /qskrps-8/?versions&max-keys=999 2023-05-24T21:06:31.639 DEBUG:boto:Data: 2023-05-24T21:06:31.640 DEBUG:boto:Headers: {} 2023-05-24T21:06:31.640 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:31.641 DEBUG:boto:Port: 8000 2023-05-24T21:06:31.641 DEBUG:boto:Params: {} 2023-05-24T21:06:31.641 DEBUG:boto:Token: None 2023-05-24T21:06:31.642 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:31 GMT /qskrps-8/?versions 2023-05-24T21:06:31.642 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:6k8kYMnUmc0ltoE6V/Ktl0ESC0Y= 2023-05-24T21:06:31.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:31 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:6k8kYMnUmc0ltoE6V/Ktl0ESC0Y=', 'Content-Length': '0'} 2023-05-24T21:06:31.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000776607dbb9859c4f-00646e7c57-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:06:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:31.646 DEBUG:boto:b'qskrps-8999false&nulltrue2023-05-24T21:02:55.134Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-05-24T21:02:55.076Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-05-24T21:02:55.045Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-05-24T21:02:55.012Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:06:31.646 DEBUG:boto:path=/qskrps-8/ 2023-05-24T21:06:31.647 DEBUG:boto:auth_path=/qskrps-8/ 2023-05-24T21:06:31.647 DEBUG:boto:path=/qskrps-8/?versions&max-keys=999 2023-05-24T21:06:31.647 DEBUG:boto:auth_path=/qskrps-8/?versions&max-keys=999 2023-05-24T21:06:31.648 DEBUG:boto:Method: GET 2023-05-24T21:06:31.648 DEBUG:boto:Path: /qskrps-8/?versions&max-keys=999 2023-05-24T21:06:31.648 DEBUG:boto:Data: 2023-05-24T21:06:31.649 DEBUG:boto:Headers: {} 2023-05-24T21:06:31.649 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:31.650 DEBUG:boto:Port: 8001 2023-05-24T21:06:31.650 DEBUG:boto:Params: {} 2023-05-24T21:06:31.650 DEBUG:boto:Token: None 2023-05-24T21:06:31.651 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:31 GMT /qskrps-8/?versions 2023-05-24T21:06:31.651 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:6k8kYMnUmc0ltoE6V/Ktl0ESC0Y= 2023-05-24T21:06:31.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:31 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:6k8kYMnUmc0ltoE6V/Ktl0ESC0Y=', 'Content-Length': '0'} 2023-05-24T21:06:31.654 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003c74b39931a70a5-00646e7c57-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:06:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:31.654 DEBUG:boto:b'qskrps-8999false&nulltrue2023-05-24T21:02:55.134Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-05-24T21:02:55.076Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-05-24T21:02:55.045Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-05-24T21:02:55.012Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:06:31.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-05-24T21:06:31.656 DEBUG:boto:path=/qskrps-8/%26 2023-05-24T21:06:31.656 DEBUG:boto:auth_path=/qskrps-8/%26 2023-05-24T21:06:31.656 DEBUG:boto:path=/qskrps-8/%26?versionId=null 2023-05-24T21:06:31.657 DEBUG:boto:auth_path=/qskrps-8/%26?versionId=null 2023-05-24T21:06:31.657 DEBUG:boto:Method: GET 2023-05-24T21:06:31.657 DEBUG:boto:Path: /qskrps-8/%26?versionId=null 2023-05-24T21:06:31.658 DEBUG:boto:Data: 2023-05-24T21:06:31.658 DEBUG:boto:Headers: {} 2023-05-24T21:06:31.659 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:31.659 DEBUG:boto:Port: 8000 2023-05-24T21:06:31.659 DEBUG:boto:Params: {} 2023-05-24T21:06:31.660 DEBUG:boto:Token: None 2023-05-24T21:06:31.660 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:31 GMT /qskrps-8/%26?versionId=null 2023-05-24T21:06:31.660 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5o7atONnXFxf8vhYCwEY68l7KD0= 2023-05-24T21:06:31.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:31 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5o7atONnXFxf8vhYCwEY68l7KD0=', 'Content-Length': '0'} 2023-05-24T21:06:31.663 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000016cb4b949e712f74-00646e7c57-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:31.663 DEBUG:boto:path=/qskrps-8/%26 2023-05-24T21:06:31.664 DEBUG:boto:auth_path=/qskrps-8/%26 2023-05-24T21:06:31.664 DEBUG:boto:path=/qskrps-8/%26?versionId=null 2023-05-24T21:06:31.665 DEBUG:boto:auth_path=/qskrps-8/%26?versionId=null 2023-05-24T21:06:31.665 DEBUG:boto:Method: GET 2023-05-24T21:06:31.665 DEBUG:boto:Path: /qskrps-8/%26?versionId=null 2023-05-24T21:06:31.666 DEBUG:boto:Data: 2023-05-24T21:06:31.666 DEBUG:boto:Headers: {} 2023-05-24T21:06:31.666 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:31.667 DEBUG:boto:Port: 8001 2023-05-24T21:06:31.667 DEBUG:boto:Params: {} 2023-05-24T21:06:31.667 DEBUG:boto:Token: None 2023-05-24T21:06:31.668 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:31 GMT /qskrps-8/%26?versionId=null 2023-05-24T21:06:31.668 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5o7atONnXFxf8vhYCwEY68l7KD0= 2023-05-24T21:06:31.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:31 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5o7atONnXFxf8vhYCwEY68l7KD0=', 'Content-Length': '0'} 2023-05-24T21:06:31.671 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006a3187a88abe2560-00646e7c57-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:31.671 DEBUG:boto:path=/qskrps-8/%26 2023-05-24T21:06:31.672 DEBUG:boto:auth_path=/qskrps-8/%26 2023-05-24T21:06:31.672 DEBUG:boto:path=/qskrps-8/%26?versionId=null 2023-05-24T21:06:31.672 DEBUG:boto:auth_path=/qskrps-8/%26?versionId=null 2023-05-24T21:06:31.673 DEBUG:boto:Method: HEAD 2023-05-24T21:06:31.673 DEBUG:boto:Path: /qskrps-8/%26?versionId=null 2023-05-24T21:06:31.674 DEBUG:boto:Data: 2023-05-24T21:06:31.674 DEBUG:boto:Headers: {} 2023-05-24T21:06:31.674 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:31.674 DEBUG:boto:Port: 8000 2023-05-24T21:06:31.675 DEBUG:boto:Params: {} 2023-05-24T21:06:31.675 DEBUG:boto:Token: None 2023-05-24T21:06:31.676 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:06:31 GMT /qskrps-8/%26?versionId=null 2023-05-24T21:06:31.676 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:YT5q0IjWgv2t3S4qfr4gvrFjQnw= 2023-05-24T21:06:31.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:31 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:YT5q0IjWgv2t3S4qfr4gvrFjQnw=', 'Content-Length': '0'} 2023-05-24T21:06:31.678 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f00f2195a292fe16-00646e7c57-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:31.679 DEBUG:boto:path=/qskrps-8/%26 2023-05-24T21:06:31.679 DEBUG:boto:auth_path=/qskrps-8/%26 2023-05-24T21:06:31.679 DEBUG:boto:path=/qskrps-8/%26?versionId=null 2023-05-24T21:06:31.680 DEBUG:boto:auth_path=/qskrps-8/%26?versionId=null 2023-05-24T21:06:31.680 DEBUG:boto:Method: HEAD 2023-05-24T21:06:31.680 DEBUG:boto:Path: /qskrps-8/%26?versionId=null 2023-05-24T21:06:31.681 DEBUG:boto:Data: 2023-05-24T21:06:31.681 DEBUG:boto:Headers: {} 2023-05-24T21:06:31.682 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:31.682 DEBUG:boto:Port: 8001 2023-05-24T21:06:31.682 DEBUG:boto:Params: {} 2023-05-24T21:06:31.683 DEBUG:boto:Token: None 2023-05-24T21:06:31.683 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:06:31 GMT /qskrps-8/%26?versionId=null 2023-05-24T21:06:31.683 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:YT5q0IjWgv2t3S4qfr4gvrFjQnw= 2023-05-24T21:06:31.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:31 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:YT5q0IjWgv2t3S4qfr4gvrFjQnw=', 'Content-Length': '0'} 2023-05-24T21:06:31.685 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000078aa538f8a0059fe-00646e7c57-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:31.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-05-24T21:06:31.686 DEBUG:boto:path=/qskrps-8/%26 2023-05-24T21:06:31.687 DEBUG:boto:auth_path=/qskrps-8/%26 2023-05-24T21:06:31.687 DEBUG:boto:path=/qskrps-8/%26?versionId=null 2023-05-24T21:06:31.687 DEBUG:boto:auth_path=/qskrps-8/%26?versionId=null 2023-05-24T21:06:31.688 DEBUG:boto:Method: GET 2023-05-24T21:06:31.688 DEBUG:boto:Path: /qskrps-8/%26?versionId=null 2023-05-24T21:06:31.688 DEBUG:boto:Data: 2023-05-24T21:06:31.689 DEBUG:boto:Headers: {} 2023-05-24T21:06:31.689 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:31.690 DEBUG:boto:Port: 8000 2023-05-24T21:06:31.690 DEBUG:boto:Params: {} 2023-05-24T21:06:31.690 DEBUG:boto:Token: None 2023-05-24T21:06:31.691 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:31 GMT /qskrps-8/%26?versionId=null 2023-05-24T21:06:31.691 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5o7atONnXFxf8vhYCwEY68l7KD0= 2023-05-24T21:06:31.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:31 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5o7atONnXFxf8vhYCwEY68l7KD0=', 'Content-Length': '0'} 2023-05-24T21:06:31.693 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006f126e7409db03b4-00646e7c57-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:31.694 DEBUG:boto:path=/qskrps-8/%26 2023-05-24T21:06:31.694 DEBUG:boto:auth_path=/qskrps-8/%26 2023-05-24T21:06:31.695 DEBUG:boto:path=/qskrps-8/%26?versionId=null 2023-05-24T21:06:31.695 DEBUG:boto:auth_path=/qskrps-8/%26?versionId=null 2023-05-24T21:06:31.695 DEBUG:boto:Method: GET 2023-05-24T21:06:31.696 DEBUG:boto:Path: /qskrps-8/%26?versionId=null 2023-05-24T21:06:31.696 DEBUG:boto:Data: 2023-05-24T21:06:31.697 DEBUG:boto:Headers: {} 2023-05-24T21:06:31.697 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:31.697 DEBUG:boto:Port: 8001 2023-05-24T21:06:31.698 DEBUG:boto:Params: {} 2023-05-24T21:06:31.698 DEBUG:boto:Token: None 2023-05-24T21:06:31.698 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:31 GMT /qskrps-8/%26?versionId=null 2023-05-24T21:06:31.699 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5o7atONnXFxf8vhYCwEY68l7KD0= 2023-05-24T21:06:31.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:31 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5o7atONnXFxf8vhYCwEY68l7KD0=', 'Content-Length': '0'} 2023-05-24T21:06:31.701 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000057520365b5738da6-00646e7c57-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:31.702 DEBUG:boto:path=/qskrps-8/ 2023-05-24T21:06:31.702 DEBUG:boto:auth_path=/qskrps-8/ 2023-05-24T21:06:31.703 DEBUG:boto:path=/qskrps-8/?prefix=%26 2023-05-24T21:06:31.703 DEBUG:boto:auth_path=/qskrps-8/?prefix=%26 2023-05-24T21:06:31.704 DEBUG:boto:Method: GET 2023-05-24T21:06:31.704 DEBUG:boto:Path: /qskrps-8/?prefix=%26 2023-05-24T21:06:31.704 DEBUG:boto:Data: 2023-05-24T21:06:31.705 DEBUG:boto:Headers: {} 2023-05-24T21:06:31.705 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:31.705 DEBUG:boto:Port: 8000 2023-05-24T21:06:31.706 DEBUG:boto:Params: {} 2023-05-24T21:06:31.706 DEBUG:boto:Token: None 2023-05-24T21:06:31.707 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:31 GMT /qskrps-8/ 2023-05-24T21:06:31.707 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:PAqDwJL89OgODZlP71f4mhIO22A= 2023-05-24T21:06:31.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:31 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PAqDwJL89OgODZlP71f4mhIO22A=', 'Content-Length': '0'} 2023-05-24T21:06:31.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098469345a1c5cbf9-00646e7c57-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:06:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:31.710 DEBUG:boto:b'qskrps-8&1000false&2023-05-24T21:02:55.134Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:06:31.711 DEBUG:boto:path=/qskrps-8/ 2023-05-24T21:06:31.711 DEBUG:boto:auth_path=/qskrps-8/ 2023-05-24T21:06:31.712 DEBUG:boto:path=/qskrps-8/?prefix=%26 2023-05-24T21:06:31.712 DEBUG:boto:auth_path=/qskrps-8/?prefix=%26 2023-05-24T21:06:31.713 DEBUG:boto:Method: GET 2023-05-24T21:06:31.713 DEBUG:boto:Path: /qskrps-8/?prefix=%26 2023-05-24T21:06:31.713 DEBUG:boto:Data: 2023-05-24T21:06:31.714 DEBUG:boto:Headers: {} 2023-05-24T21:06:31.714 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:31.714 DEBUG:boto:Port: 8001 2023-05-24T21:06:31.715 DEBUG:boto:Params: {} 2023-05-24T21:06:31.715 DEBUG:boto:Token: None 2023-05-24T21:06:31.715 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:31 GMT /qskrps-8/ 2023-05-24T21:06:31.716 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:PAqDwJL89OgODZlP71f4mhIO22A= 2023-05-24T21:06:31.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:31 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PAqDwJL89OgODZlP71f4mhIO22A=', 'Content-Length': '0'} 2023-05-24T21:06:31.719 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e4fcafe97f2920c-00646e7c57-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:06:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:31.719 DEBUG:boto:b'qskrps-8&1000false&2023-05-24T21:02:55.134Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:06:31.720 DEBUG:boto:path=/qskrps-8/%26 2023-05-24T21:06:31.720 DEBUG:boto:auth_path=/qskrps-8/%26 2023-05-24T21:06:31.721 DEBUG:boto:Method: HEAD 2023-05-24T21:06:31.721 DEBUG:boto:Path: /qskrps-8/%26 2023-05-24T21:06:31.721 DEBUG:boto:Data: 2023-05-24T21:06:31.722 DEBUG:boto:Headers: {} 2023-05-24T21:06:31.722 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:31.722 DEBUG:boto:Port: 8000 2023-05-24T21:06:31.723 DEBUG:boto:Params: {} 2023-05-24T21:06:31.723 DEBUG:boto:Token: None 2023-05-24T21:06:31.724 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:06:31 GMT /qskrps-8/%26 2023-05-24T21:06:31.724 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ibn7AEW4ebCDEhRgC66eRKA/b7w= 2023-05-24T21:06:31.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:31 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ibn7AEW4ebCDEhRgC66eRKA/b7w=', 'Content-Length': '0'} 2023-05-24T21:06:31.726 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009d1291cde7fcfc34-00646e7c57-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:31.726 DEBUG:boto:path=/qskrps-8/%26 2023-05-24T21:06:31.727 DEBUG:boto:auth_path=/qskrps-8/%26 2023-05-24T21:06:31.727 DEBUG:boto:Method: HEAD 2023-05-24T21:06:31.728 DEBUG:boto:Path: /qskrps-8/%26 2023-05-24T21:06:31.728 DEBUG:boto:Data: 2023-05-24T21:06:31.728 DEBUG:boto:Headers: {} 2023-05-24T21:06:31.729 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:31.729 DEBUG:boto:Port: 8001 2023-05-24T21:06:31.729 DEBUG:boto:Params: {} 2023-05-24T21:06:31.730 DEBUG:boto:Token: None 2023-05-24T21:06:31.730 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:06:31 GMT /qskrps-8/%26 2023-05-24T21:06:31.730 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ibn7AEW4ebCDEhRgC66eRKA/b7w= 2023-05-24T21:06:31.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:31 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ibn7AEW4ebCDEhRgC66eRKA/b7w=', 'Content-Length': '0'} 2023-05-24T21:06:31.732 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000240ee5748bb99975-00646e7c57-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:31.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-05-24T21:06:31.733 DEBUG:boto:path=/qskrps-8/%26 2023-05-24T21:06:31.734 DEBUG:boto:auth_path=/qskrps-8/%26 2023-05-24T21:06:31.734 DEBUG:boto:Method: GET 2023-05-24T21:06:31.734 DEBUG:boto:Path: /qskrps-8/%26 2023-05-24T21:06:31.735 DEBUG:boto:Data: 2023-05-24T21:06:31.735 DEBUG:boto:Headers: {} 2023-05-24T21:06:31.736 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:31.736 DEBUG:boto:Port: 8000 2023-05-24T21:06:31.736 DEBUG:boto:Params: {} 2023-05-24T21:06:31.737 DEBUG:boto:Token: None 2023-05-24T21:06:31.737 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:31 GMT /qskrps-8/%26 2023-05-24T21:06:31.738 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:W9Zc67flJTt1HniJOizXZzNO6VI= 2023-05-24T21:06:31.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:31 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:W9Zc67flJTt1HniJOizXZzNO6VI=', 'Content-Length': '0'} 2023-05-24T21:06:31.740 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000195e7c72af017cc-00646e7c57-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:31.740 DEBUG:boto:path=/qskrps-8/%26 2023-05-24T21:06:31.741 DEBUG:boto:auth_path=/qskrps-8/%26 2023-05-24T21:06:31.741 DEBUG:boto:Method: GET 2023-05-24T21:06:31.741 DEBUG:boto:Path: /qskrps-8/%26 2023-05-24T21:06:31.742 DEBUG:boto:Data: 2023-05-24T21:06:31.742 DEBUG:boto:Headers: {} 2023-05-24T21:06:31.742 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:31.743 DEBUG:boto:Port: 8001 2023-05-24T21:06:31.743 DEBUG:boto:Params: {} 2023-05-24T21:06:31.744 DEBUG:boto:Token: None 2023-05-24T21:06:31.744 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:31 GMT /qskrps-8/%26 2023-05-24T21:06:31.744 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:W9Zc67flJTt1HniJOizXZzNO6VI= 2023-05-24T21:06:31.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:31 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:W9Zc67flJTt1HniJOizXZzNO6VI=', 'Content-Length': '0'} 2023-05-24T21:06:31.746 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006434ff6516043139-00646e7c57-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:31.747 DEBUG:boto:path=/qskrps-8/ 2023-05-24T21:06:31.747 DEBUG:boto:auth_path=/qskrps-8/ 2023-05-24T21:06:31.748 DEBUG:boto:path=/qskrps-8/?prefix=%26 2023-05-24T21:06:31.748 DEBUG:boto:auth_path=/qskrps-8/?prefix=%26 2023-05-24T21:06:31.749 DEBUG:boto:Method: GET 2023-05-24T21:06:31.749 DEBUG:boto:Path: /qskrps-8/?prefix=%26 2023-05-24T21:06:31.749 DEBUG:boto:Data: 2023-05-24T21:06:31.750 DEBUG:boto:Headers: {} 2023-05-24T21:06:31.750 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:31.750 DEBUG:boto:Port: 8000 2023-05-24T21:06:31.751 DEBUG:boto:Params: {} 2023-05-24T21:06:31.751 DEBUG:boto:Token: None 2023-05-24T21:06:31.751 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:31 GMT /qskrps-8/ 2023-05-24T21:06:31.752 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:PAqDwJL89OgODZlP71f4mhIO22A= 2023-05-24T21:06:31.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:31 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PAqDwJL89OgODZlP71f4mhIO22A=', 'Content-Length': '0'} 2023-05-24T21:06:31.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db4b1095cd32fec9-00646e7c57-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:06:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:31.755 DEBUG:boto:b'qskrps-8&1000false&2023-05-24T21:02:55.134Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:06:31.756 DEBUG:boto:path=/qskrps-8/ 2023-05-24T21:06:31.756 DEBUG:boto:auth_path=/qskrps-8/ 2023-05-24T21:06:31.756 DEBUG:boto:path=/qskrps-8/?prefix=%26 2023-05-24T21:06:31.757 DEBUG:boto:auth_path=/qskrps-8/?prefix=%26 2023-05-24T21:06:31.757 DEBUG:boto:Method: GET 2023-05-24T21:06:31.758 DEBUG:boto:Path: /qskrps-8/?prefix=%26 2023-05-24T21:06:31.758 DEBUG:boto:Data: 2023-05-24T21:06:31.758 DEBUG:boto:Headers: {} 2023-05-24T21:06:31.759 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:31.759 DEBUG:boto:Port: 8001 2023-05-24T21:06:31.759 DEBUG:boto:Params: {} 2023-05-24T21:06:31.760 DEBUG:boto:Token: None 2023-05-24T21:06:31.760 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:31 GMT /qskrps-8/ 2023-05-24T21:06:31.761 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:PAqDwJL89OgODZlP71f4mhIO22A= 2023-05-24T21:06:31.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:31 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PAqDwJL89OgODZlP71f4mhIO22A=', 'Content-Length': '0'} 2023-05-24T21:06:31.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056a8340d1f43123d-00646e7c57-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:06:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:31.765 DEBUG:boto:b'qskrps-8&1000false&2023-05-24T21:02:55.134Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:06:31.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-05-24T21:06:31.766 DEBUG:boto:path=/qskrps-8/%3A 2023-05-24T21:06:31.766 DEBUG:boto:auth_path=/qskrps-8/%3A 2023-05-24T21:06:31.767 DEBUG:boto:path=/qskrps-8/%3A?versionId=null 2023-05-24T21:06:31.767 DEBUG:boto:auth_path=/qskrps-8/%3A?versionId=null 2023-05-24T21:06:31.767 DEBUG:boto:Method: GET 2023-05-24T21:06:31.768 DEBUG:boto:Path: /qskrps-8/%3A?versionId=null 2023-05-24T21:06:31.768 DEBUG:boto:Data: 2023-05-24T21:06:31.768 DEBUG:boto:Headers: {} 2023-05-24T21:06:31.769 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:31.769 DEBUG:boto:Port: 8000 2023-05-24T21:06:31.770 DEBUG:boto:Params: {} 2023-05-24T21:06:31.770 DEBUG:boto:Token: None 2023-05-24T21:06:31.770 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:31 GMT /qskrps-8/%3A?versionId=null 2023-05-24T21:06:31.771 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:UqQpYeTlty+RhQWHqmQ86d9hytI= 2023-05-24T21:06:31.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:31 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:UqQpYeTlty+RhQWHqmQ86d9hytI=', 'Content-Length': '0'} 2023-05-24T21:06:31.773 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001a1ca11bb2b84287-00646e7c57-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:31.774 DEBUG:boto:path=/qskrps-8/%3A 2023-05-24T21:06:31.774 DEBUG:boto:auth_path=/qskrps-8/%3A 2023-05-24T21:06:31.775 DEBUG:boto:path=/qskrps-8/%3A?versionId=null 2023-05-24T21:06:31.775 DEBUG:boto:auth_path=/qskrps-8/%3A?versionId=null 2023-05-24T21:06:31.775 DEBUG:boto:Method: GET 2023-05-24T21:06:31.776 DEBUG:boto:Path: /qskrps-8/%3A?versionId=null 2023-05-24T21:06:31.776 DEBUG:boto:Data: 2023-05-24T21:06:31.776 DEBUG:boto:Headers: {} 2023-05-24T21:06:31.777 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:31.777 DEBUG:boto:Port: 8001 2023-05-24T21:06:31.777 DEBUG:boto:Params: {} 2023-05-24T21:06:31.778 DEBUG:boto:Token: None 2023-05-24T21:06:31.778 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:31 GMT /qskrps-8/%3A?versionId=null 2023-05-24T21:06:31.779 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:UqQpYeTlty+RhQWHqmQ86d9hytI= 2023-05-24T21:06:31.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:31 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:UqQpYeTlty+RhQWHqmQ86d9hytI=', 'Content-Length': '0'} 2023-05-24T21:06:31.781 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e1d4af409d2c63ec-00646e7c57-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:31.824 DEBUG:boto:path=/qskrps-8/%3A 2023-05-24T21:06:31.825 DEBUG:boto:auth_path=/qskrps-8/%3A 2023-05-24T21:06:31.825 DEBUG:boto:path=/qskrps-8/%3A?versionId=null 2023-05-24T21:06:31.825 DEBUG:boto:auth_path=/qskrps-8/%3A?versionId=null 2023-05-24T21:06:31.826 DEBUG:boto:Method: HEAD 2023-05-24T21:06:31.826 DEBUG:boto:Path: /qskrps-8/%3A?versionId=null 2023-05-24T21:06:31.826 DEBUG:boto:Data: 2023-05-24T21:06:31.827 DEBUG:boto:Headers: {} 2023-05-24T21:06:31.827 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:31.827 DEBUG:boto:Port: 8000 2023-05-24T21:06:31.828 DEBUG:boto:Params: {} 2023-05-24T21:06:31.828 DEBUG:boto:Token: None 2023-05-24T21:06:31.828 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:06:31 GMT /qskrps-8/%3A?versionId=null 2023-05-24T21:06:31.829 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ybYMn1u6gqgWUV4LtZtMLizaEi0= 2023-05-24T21:06:31.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:31 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ybYMn1u6gqgWUV4LtZtMLizaEi0=', 'Content-Length': '0'} 2023-05-24T21:06:31.831 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000157d6c1764919cfc-00646e7c57-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:31.832 DEBUG:boto:path=/qskrps-8/%3A 2023-05-24T21:06:31.832 DEBUG:boto:auth_path=/qskrps-8/%3A 2023-05-24T21:06:31.832 DEBUG:boto:path=/qskrps-8/%3A?versionId=null 2023-05-24T21:06:31.833 DEBUG:boto:auth_path=/qskrps-8/%3A?versionId=null 2023-05-24T21:06:31.833 DEBUG:boto:Method: HEAD 2023-05-24T21:06:31.833 DEBUG:boto:Path: /qskrps-8/%3A?versionId=null 2023-05-24T21:06:31.834 DEBUG:boto:Data: 2023-05-24T21:06:31.834 DEBUG:boto:Headers: {} 2023-05-24T21:06:31.834 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:31.835 DEBUG:boto:Port: 8001 2023-05-24T21:06:31.835 DEBUG:boto:Params: {} 2023-05-24T21:06:31.835 DEBUG:boto:Token: None 2023-05-24T21:06:31.836 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:06:31 GMT /qskrps-8/%3A?versionId=null 2023-05-24T21:06:31.836 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ybYMn1u6gqgWUV4LtZtMLizaEi0= 2023-05-24T21:06:31.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:31 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ybYMn1u6gqgWUV4LtZtMLizaEi0=', 'Content-Length': '0'} 2023-05-24T21:06:31.838 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ff3dcfca693687fb-00646e7c57-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:31.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-05-24T21:06:31.839 DEBUG:boto:path=/qskrps-8/%3A 2023-05-24T21:06:31.840 DEBUG:boto:auth_path=/qskrps-8/%3A 2023-05-24T21:06:31.840 DEBUG:boto:path=/qskrps-8/%3A?versionId=null 2023-05-24T21:06:31.840 DEBUG:boto:auth_path=/qskrps-8/%3A?versionId=null 2023-05-24T21:06:31.841 DEBUG:boto:Method: GET 2023-05-24T21:06:31.841 DEBUG:boto:Path: /qskrps-8/%3A?versionId=null 2023-05-24T21:06:31.841 DEBUG:boto:Data: 2023-05-24T21:06:31.842 DEBUG:boto:Headers: {} 2023-05-24T21:06:31.842 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:31.842 DEBUG:boto:Port: 8000 2023-05-24T21:06:31.843 DEBUG:boto:Params: {} 2023-05-24T21:06:31.843 DEBUG:boto:Token: None 2023-05-24T21:06:31.844 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:31 GMT /qskrps-8/%3A?versionId=null 2023-05-24T21:06:31.844 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:UqQpYeTlty+RhQWHqmQ86d9hytI= 2023-05-24T21:06:31.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:31 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:UqQpYeTlty+RhQWHqmQ86d9hytI=', 'Content-Length': '0'} 2023-05-24T21:06:31.846 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a092daac105beb9d-00646e7c57-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:31.847 DEBUG:boto:path=/qskrps-8/%3A 2023-05-24T21:06:31.847 DEBUG:boto:auth_path=/qskrps-8/%3A 2023-05-24T21:06:31.848 DEBUG:boto:path=/qskrps-8/%3A?versionId=null 2023-05-24T21:06:31.848 DEBUG:boto:auth_path=/qskrps-8/%3A?versionId=null 2023-05-24T21:06:31.848 DEBUG:boto:Method: GET 2023-05-24T21:06:31.849 DEBUG:boto:Path: /qskrps-8/%3A?versionId=null 2023-05-24T21:06:31.849 DEBUG:boto:Data: 2023-05-24T21:06:31.849 DEBUG:boto:Headers: {} 2023-05-24T21:06:31.850 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:31.850 DEBUG:boto:Port: 8001 2023-05-24T21:06:31.850 DEBUG:boto:Params: {} 2023-05-24T21:06:31.851 DEBUG:boto:Token: None 2023-05-24T21:06:31.851 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:31 GMT /qskrps-8/%3A?versionId=null 2023-05-24T21:06:31.852 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:UqQpYeTlty+RhQWHqmQ86d9hytI= 2023-05-24T21:06:31.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:31 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:UqQpYeTlty+RhQWHqmQ86d9hytI=', 'Content-Length': '0'} 2023-05-24T21:06:31.854 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b708efd87b53e704-00646e7c57-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:31.854 DEBUG:boto:path=/qskrps-8/ 2023-05-24T21:06:31.855 DEBUG:boto:auth_path=/qskrps-8/ 2023-05-24T21:06:31.855 DEBUG:boto:path=/qskrps-8/?prefix=%3A 2023-05-24T21:06:31.855 DEBUG:boto:auth_path=/qskrps-8/?prefix=%3A 2023-05-24T21:06:31.856 DEBUG:boto:Method: GET 2023-05-24T21:06:31.856 DEBUG:boto:Path: /qskrps-8/?prefix=%3A 2023-05-24T21:06:31.857 DEBUG:boto:Data: 2023-05-24T21:06:31.857 DEBUG:boto:Headers: {} 2023-05-24T21:06:31.857 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:31.858 DEBUG:boto:Port: 8000 2023-05-24T21:06:31.858 DEBUG:boto:Params: {} 2023-05-24T21:06:31.858 DEBUG:boto:Token: None 2023-05-24T21:06:31.859 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:31 GMT /qskrps-8/ 2023-05-24T21:06:31.859 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:PAqDwJL89OgODZlP71f4mhIO22A= 2023-05-24T21:06:31.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:31 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PAqDwJL89OgODZlP71f4mhIO22A=', 'Content-Length': '0'} 2023-05-24T21:06:31.862 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea113f68fc72d194-00646e7c57-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:06:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:31.862 DEBUG:boto:b'qskrps-8:1000false:2023-05-24T21:02:55.076Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:06:31.863 DEBUG:boto:path=/qskrps-8/ 2023-05-24T21:06:31.863 DEBUG:boto:auth_path=/qskrps-8/ 2023-05-24T21:06:31.864 DEBUG:boto:path=/qskrps-8/?prefix=%3A 2023-05-24T21:06:31.864 DEBUG:boto:auth_path=/qskrps-8/?prefix=%3A 2023-05-24T21:06:31.864 DEBUG:boto:Method: GET 2023-05-24T21:06:31.865 DEBUG:boto:Path: /qskrps-8/?prefix=%3A 2023-05-24T21:06:31.865 DEBUG:boto:Data: 2023-05-24T21:06:31.866 DEBUG:boto:Headers: {} 2023-05-24T21:06:31.866 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:31.866 DEBUG:boto:Port: 8001 2023-05-24T21:06:31.867 DEBUG:boto:Params: {} 2023-05-24T21:06:31.867 DEBUG:boto:Token: None 2023-05-24T21:06:31.867 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:31 GMT /qskrps-8/ 2023-05-24T21:06:31.868 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:PAqDwJL89OgODZlP71f4mhIO22A= 2023-05-24T21:06:31.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:31 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PAqDwJL89OgODZlP71f4mhIO22A=', 'Content-Length': '0'} 2023-05-24T21:06:31.870 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000074c16010b41286c-00646e7c57-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:06:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:31.871 DEBUG:boto:b'qskrps-8:1000false:2023-05-24T21:02:55.076Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:06:31.871 DEBUG:boto:path=/qskrps-8/%3A 2023-05-24T21:06:31.872 DEBUG:boto:auth_path=/qskrps-8/%3A 2023-05-24T21:06:31.872 DEBUG:boto:Method: HEAD 2023-05-24T21:06:31.873 DEBUG:boto:Path: /qskrps-8/%3A 2023-05-24T21:06:31.873 DEBUG:boto:Data: 2023-05-24T21:06:31.873 DEBUG:boto:Headers: {} 2023-05-24T21:06:31.874 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:31.874 DEBUG:boto:Port: 8000 2023-05-24T21:06:31.874 DEBUG:boto:Params: {} 2023-05-24T21:06:31.875 DEBUG:boto:Token: None 2023-05-24T21:06:31.875 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:06:31 GMT /qskrps-8/%3A 2023-05-24T21:06:31.875 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Pg28Rr9vo61Aedxpuv36zkdt72I= 2023-05-24T21:06:31.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:31 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Pg28Rr9vo61Aedxpuv36zkdt72I=', 'Content-Length': '0'} 2023-05-24T21:06:31.877 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000062e7ae0ea50b6962-00646e7c57-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:31.878 DEBUG:boto:path=/qskrps-8/%3A 2023-05-24T21:06:31.878 DEBUG:boto:auth_path=/qskrps-8/%3A 2023-05-24T21:06:31.879 DEBUG:boto:Method: HEAD 2023-05-24T21:06:31.879 DEBUG:boto:Path: /qskrps-8/%3A 2023-05-24T21:06:31.879 DEBUG:boto:Data: 2023-05-24T21:06:31.880 DEBUG:boto:Headers: {} 2023-05-24T21:06:31.880 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:31.880 DEBUG:boto:Port: 8001 2023-05-24T21:06:31.881 DEBUG:boto:Params: {} 2023-05-24T21:06:31.881 DEBUG:boto:Token: None 2023-05-24T21:06:31.881 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:06:31 GMT /qskrps-8/%3A 2023-05-24T21:06:31.882 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Pg28Rr9vo61Aedxpuv36zkdt72I= 2023-05-24T21:06:31.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:31 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Pg28Rr9vo61Aedxpuv36zkdt72I=', 'Content-Length': '0'} 2023-05-24T21:06:31.884 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf7f7bc4b732b9a0-00646e7c57-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:31.884 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-05-24T21:06:31.885 DEBUG:boto:path=/qskrps-8/%3A 2023-05-24T21:06:31.885 DEBUG:boto:auth_path=/qskrps-8/%3A 2023-05-24T21:06:31.886 DEBUG:boto:Method: GET 2023-05-24T21:06:31.886 DEBUG:boto:Path: /qskrps-8/%3A 2023-05-24T21:06:31.886 DEBUG:boto:Data: 2023-05-24T21:06:31.887 DEBUG:boto:Headers: {} 2023-05-24T21:06:31.887 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:31.887 DEBUG:boto:Port: 8000 2023-05-24T21:06:31.888 DEBUG:boto:Params: {} 2023-05-24T21:06:31.888 DEBUG:boto:Token: None 2023-05-24T21:06:31.889 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:31 GMT /qskrps-8/%3A 2023-05-24T21:06:31.889 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:+gFjgq47eNX9J6LgA4igvQaUp6Y= 2023-05-24T21:06:31.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:31 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:+gFjgq47eNX9J6LgA4igvQaUp6Y=', 'Content-Length': '0'} 2023-05-24T21:06:31.891 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f542c3140f236d7d-00646e7c57-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:31.892 DEBUG:boto:path=/qskrps-8/%3A 2023-05-24T21:06:31.892 DEBUG:boto:auth_path=/qskrps-8/%3A 2023-05-24T21:06:31.893 DEBUG:boto:Method: GET 2023-05-24T21:06:31.893 DEBUG:boto:Path: /qskrps-8/%3A 2023-05-24T21:06:31.894 DEBUG:boto:Data: 2023-05-24T21:06:31.894 DEBUG:boto:Headers: {} 2023-05-24T21:06:31.894 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:31.895 DEBUG:boto:Port: 8001 2023-05-24T21:06:31.895 DEBUG:boto:Params: {} 2023-05-24T21:06:31.896 DEBUG:boto:Token: None 2023-05-24T21:06:31.896 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:31 GMT /qskrps-8/%3A 2023-05-24T21:06:31.896 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:+gFjgq47eNX9J6LgA4igvQaUp6Y= 2023-05-24T21:06:31.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:31 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:+gFjgq47eNX9J6LgA4igvQaUp6Y=', 'Content-Length': '0'} 2023-05-24T21:06:31.899 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000046903fd2365c37ad-00646e7c57-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:31.899 DEBUG:boto:path=/qskrps-8/ 2023-05-24T21:06:31.900 DEBUG:boto:auth_path=/qskrps-8/ 2023-05-24T21:06:31.900 DEBUG:boto:path=/qskrps-8/?prefix=%3A 2023-05-24T21:06:31.900 DEBUG:boto:auth_path=/qskrps-8/?prefix=%3A 2023-05-24T21:06:31.901 DEBUG:boto:Method: GET 2023-05-24T21:06:31.901 DEBUG:boto:Path: /qskrps-8/?prefix=%3A 2023-05-24T21:06:31.901 DEBUG:boto:Data: 2023-05-24T21:06:31.902 DEBUG:boto:Headers: {} 2023-05-24T21:06:31.902 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:31.902 DEBUG:boto:Port: 8000 2023-05-24T21:06:31.903 DEBUG:boto:Params: {} 2023-05-24T21:06:31.903 DEBUG:boto:Token: None 2023-05-24T21:06:31.904 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:31 GMT /qskrps-8/ 2023-05-24T21:06:31.904 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:PAqDwJL89OgODZlP71f4mhIO22A= 2023-05-24T21:06:31.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:31 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PAqDwJL89OgODZlP71f4mhIO22A=', 'Content-Length': '0'} 2023-05-24T21:06:31.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000053adb612146540d-00646e7c57-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:06:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:31.907 DEBUG:boto:b'qskrps-8:1000false:2023-05-24T21:02:55.076Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:06:31.908 DEBUG:boto:path=/qskrps-8/ 2023-05-24T21:06:31.908 DEBUG:boto:auth_path=/qskrps-8/ 2023-05-24T21:06:31.908 DEBUG:boto:path=/qskrps-8/?prefix=%3A 2023-05-24T21:06:31.909 DEBUG:boto:auth_path=/qskrps-8/?prefix=%3A 2023-05-24T21:06:31.909 DEBUG:boto:Method: GET 2023-05-24T21:06:31.910 DEBUG:boto:Path: /qskrps-8/?prefix=%3A 2023-05-24T21:06:31.910 DEBUG:boto:Data: 2023-05-24T21:06:31.910 DEBUG:boto:Headers: {} 2023-05-24T21:06:31.911 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:31.911 DEBUG:boto:Port: 8001 2023-05-24T21:06:31.911 DEBUG:boto:Params: {} 2023-05-24T21:06:31.912 DEBUG:boto:Token: None 2023-05-24T21:06:31.912 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:31 GMT /qskrps-8/ 2023-05-24T21:06:31.913 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:PAqDwJL89OgODZlP71f4mhIO22A= 2023-05-24T21:06:31.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:31 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PAqDwJL89OgODZlP71f4mhIO22A=', 'Content-Length': '0'} 2023-05-24T21:06:31.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8793ec95e300d28-00646e7c57-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:06:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:31.916 DEBUG:boto:b'qskrps-8:1000false:2023-05-24T21:02:55.076Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:06:31.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-05-24T21:06:31.917 DEBUG:boto:path=/qskrps-8/_myobj 2023-05-24T21:06:31.917 DEBUG:boto:auth_path=/qskrps-8/_myobj 2023-05-24T21:06:31.918 DEBUG:boto:path=/qskrps-8/_myobj?versionId=null 2023-05-24T21:06:31.918 DEBUG:boto:auth_path=/qskrps-8/_myobj?versionId=null 2023-05-24T21:06:31.918 DEBUG:boto:Method: GET 2023-05-24T21:06:31.919 DEBUG:boto:Path: /qskrps-8/_myobj?versionId=null 2023-05-24T21:06:31.919 DEBUG:boto:Data: 2023-05-24T21:06:31.919 DEBUG:boto:Headers: {} 2023-05-24T21:06:31.920 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:31.920 DEBUG:boto:Port: 8000 2023-05-24T21:06:31.921 DEBUG:boto:Params: {} 2023-05-24T21:06:31.921 DEBUG:boto:Token: None 2023-05-24T21:06:31.922 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:31 GMT /qskrps-8/_myobj?versionId=null 2023-05-24T21:06:31.922 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:/6BshBUa7lEKWGYhGy0Q+A4vH2w= 2023-05-24T21:06:31.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:31 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:/6BshBUa7lEKWGYhGy0Q+A4vH2w=', 'Content-Length': '0'} 2023-05-24T21:06:31.924 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e869965fb67c16a2-00646e7c57-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:31.925 DEBUG:boto:path=/qskrps-8/_myobj 2023-05-24T21:06:31.925 DEBUG:boto:auth_path=/qskrps-8/_myobj 2023-05-24T21:06:31.926 DEBUG:boto:path=/qskrps-8/_myobj?versionId=null 2023-05-24T21:06:31.926 DEBUG:boto:auth_path=/qskrps-8/_myobj?versionId=null 2023-05-24T21:06:31.926 DEBUG:boto:Method: GET 2023-05-24T21:06:31.927 DEBUG:boto:Path: /qskrps-8/_myobj?versionId=null 2023-05-24T21:06:31.927 DEBUG:boto:Data: 2023-05-24T21:06:31.928 DEBUG:boto:Headers: {} 2023-05-24T21:06:31.928 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:31.928 DEBUG:boto:Port: 8001 2023-05-24T21:06:31.929 DEBUG:boto:Params: {} 2023-05-24T21:06:31.929 DEBUG:boto:Token: None 2023-05-24T21:06:31.929 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:31 GMT /qskrps-8/_myobj?versionId=null 2023-05-24T21:06:31.930 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:/6BshBUa7lEKWGYhGy0Q+A4vH2w= 2023-05-24T21:06:31.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:31 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:/6BshBUa7lEKWGYhGy0Q+A4vH2w=', 'Content-Length': '0'} 2023-05-24T21:06:31.932 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000088fa4938abd36be0-00646e7c57-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:31.933 DEBUG:boto:path=/qskrps-8/_myobj 2023-05-24T21:06:31.933 DEBUG:boto:auth_path=/qskrps-8/_myobj 2023-05-24T21:06:31.933 DEBUG:boto:path=/qskrps-8/_myobj?versionId=null 2023-05-24T21:06:31.934 DEBUG:boto:auth_path=/qskrps-8/_myobj?versionId=null 2023-05-24T21:06:31.934 DEBUG:boto:Method: HEAD 2023-05-24T21:06:31.934 DEBUG:boto:Path: /qskrps-8/_myobj?versionId=null 2023-05-24T21:06:31.935 DEBUG:boto:Data: 2023-05-24T21:06:31.935 DEBUG:boto:Headers: {} 2023-05-24T21:06:31.935 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:31.936 DEBUG:boto:Port: 8000 2023-05-24T21:06:31.936 DEBUG:boto:Params: {} 2023-05-24T21:06:31.936 DEBUG:boto:Token: None 2023-05-24T21:06:31.937 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:06:31 GMT /qskrps-8/_myobj?versionId=null 2023-05-24T21:06:31.937 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:nSUrlvMVX1JpYwaPMlXIOEo1xLA= 2023-05-24T21:06:31.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:31 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:nSUrlvMVX1JpYwaPMlXIOEo1xLA=', 'Content-Length': '0'} 2023-05-24T21:06:31.939 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001b99f45c428c5633-00646e7c57-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:31.940 DEBUG:boto:path=/qskrps-8/_myobj 2023-05-24T21:06:31.940 DEBUG:boto:auth_path=/qskrps-8/_myobj 2023-05-24T21:06:31.940 DEBUG:boto:path=/qskrps-8/_myobj?versionId=null 2023-05-24T21:06:31.941 DEBUG:boto:auth_path=/qskrps-8/_myobj?versionId=null 2023-05-24T21:06:31.941 DEBUG:boto:Method: HEAD 2023-05-24T21:06:31.941 DEBUG:boto:Path: /qskrps-8/_myobj?versionId=null 2023-05-24T21:06:31.942 DEBUG:boto:Data: 2023-05-24T21:06:31.942 DEBUG:boto:Headers: {} 2023-05-24T21:06:31.942 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:31.943 DEBUG:boto:Port: 8001 2023-05-24T21:06:31.943 DEBUG:boto:Params: {} 2023-05-24T21:06:31.944 DEBUG:boto:Token: None 2023-05-24T21:06:31.944 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:06:31 GMT /qskrps-8/_myobj?versionId=null 2023-05-24T21:06:31.944 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:nSUrlvMVX1JpYwaPMlXIOEo1xLA= 2023-05-24T21:06:31.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:31 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:nSUrlvMVX1JpYwaPMlXIOEo1xLA=', 'Content-Length': '0'} 2023-05-24T21:06:31.946 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006df0b1aa727f53cc-00646e7c57-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:31.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-05-24T21:06:31.947 DEBUG:boto:path=/qskrps-8/_myobj 2023-05-24T21:06:31.947 DEBUG:boto:auth_path=/qskrps-8/_myobj 2023-05-24T21:06:31.948 DEBUG:boto:path=/qskrps-8/_myobj?versionId=null 2023-05-24T21:06:31.948 DEBUG:boto:auth_path=/qskrps-8/_myobj?versionId=null 2023-05-24T21:06:31.949 DEBUG:boto:Method: GET 2023-05-24T21:06:31.949 DEBUG:boto:Path: /qskrps-8/_myobj?versionId=null 2023-05-24T21:06:31.949 DEBUG:boto:Data: 2023-05-24T21:06:31.950 DEBUG:boto:Headers: {} 2023-05-24T21:06:31.950 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:31.950 DEBUG:boto:Port: 8000 2023-05-24T21:06:31.951 DEBUG:boto:Params: {} 2023-05-24T21:06:31.951 DEBUG:boto:Token: None 2023-05-24T21:06:31.952 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:31 GMT /qskrps-8/_myobj?versionId=null 2023-05-24T21:06:31.952 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:/6BshBUa7lEKWGYhGy0Q+A4vH2w= 2023-05-24T21:06:31.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:31 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:/6BshBUa7lEKWGYhGy0Q+A4vH2w=', 'Content-Length': '0'} 2023-05-24T21:06:31.957 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000847b315e55e1b29a-00646e7c57-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:32.004 DEBUG:boto:path=/qskrps-8/_myobj 2023-05-24T21:06:32.005 DEBUG:boto:auth_path=/qskrps-8/_myobj 2023-05-24T21:06:32.005 DEBUG:boto:path=/qskrps-8/_myobj?versionId=null 2023-05-24T21:06:32.005 DEBUG:boto:auth_path=/qskrps-8/_myobj?versionId=null 2023-05-24T21:06:32.006 DEBUG:boto:Method: GET 2023-05-24T21:06:32.006 DEBUG:boto:Path: /qskrps-8/_myobj?versionId=null 2023-05-24T21:06:32.006 DEBUG:boto:Data: 2023-05-24T21:06:32.007 DEBUG:boto:Headers: {} 2023-05-24T21:06:32.007 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:32.008 DEBUG:boto:Port: 8001 2023-05-24T21:06:32.008 DEBUG:boto:Params: {} 2023-05-24T21:06:32.008 DEBUG:boto:Token: None 2023-05-24T21:06:32.009 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:32 GMT /qskrps-8/_myobj?versionId=null 2023-05-24T21:06:32.009 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:uAxk/6da6vdX2zwfWbcXJwhmYGY= 2023-05-24T21:06:32.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:32 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:uAxk/6da6vdX2zwfWbcXJwhmYGY=', 'Content-Length': '0'} 2023-05-24T21:06:32.011 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b355b842b4145182-00646e7c58-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:32.012 DEBUG:boto:path=/qskrps-8/ 2023-05-24T21:06:32.012 DEBUG:boto:auth_path=/qskrps-8/ 2023-05-24T21:06:32.013 DEBUG:boto:path=/qskrps-8/?prefix=_myobj 2023-05-24T21:06:32.013 DEBUG:boto:auth_path=/qskrps-8/?prefix=_myobj 2023-05-24T21:06:32.014 DEBUG:boto:Method: GET 2023-05-24T21:06:32.014 DEBUG:boto:Path: /qskrps-8/?prefix=_myobj 2023-05-24T21:06:32.014 DEBUG:boto:Data: 2023-05-24T21:06:32.015 DEBUG:boto:Headers: {} 2023-05-24T21:06:32.015 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:32.015 DEBUG:boto:Port: 8000 2023-05-24T21:06:32.016 DEBUG:boto:Params: {} 2023-05-24T21:06:32.016 DEBUG:boto:Token: None 2023-05-24T21:06:32.017 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:32 GMT /qskrps-8/ 2023-05-24T21:06:32.017 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:OcwObQ0DFuo/5vHy3Oe5+iBKi/g= 2023-05-24T21:06:32.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:32 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:OcwObQ0DFuo/5vHy3Oe5+iBKi/g=', 'Content-Length': '0'} 2023-05-24T21:06:32.020 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d044c75865e74f33-00646e7c58-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:06:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:32.020 DEBUG:boto:b'qskrps-8_myobj1000false_myobj2023-05-24T21:02:55.045Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:06:32.021 DEBUG:boto:path=/qskrps-8/ 2023-05-24T21:06:32.021 DEBUG:boto:auth_path=/qskrps-8/ 2023-05-24T21:06:32.022 DEBUG:boto:path=/qskrps-8/?prefix=_myobj 2023-05-24T21:06:32.022 DEBUG:boto:auth_path=/qskrps-8/?prefix=_myobj 2023-05-24T21:06:32.022 DEBUG:boto:Method: GET 2023-05-24T21:06:32.023 DEBUG:boto:Path: /qskrps-8/?prefix=_myobj 2023-05-24T21:06:32.023 DEBUG:boto:Data: 2023-05-24T21:06:32.023 DEBUG:boto:Headers: {} 2023-05-24T21:06:32.024 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:32.024 DEBUG:boto:Port: 8001 2023-05-24T21:06:32.025 DEBUG:boto:Params: {} 2023-05-24T21:06:32.025 DEBUG:boto:Token: None 2023-05-24T21:06:32.025 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:32 GMT /qskrps-8/ 2023-05-24T21:06:32.026 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:OcwObQ0DFuo/5vHy3Oe5+iBKi/g= 2023-05-24T21:06:32.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:32 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:OcwObQ0DFuo/5vHy3Oe5+iBKi/g=', 'Content-Length': '0'} 2023-05-24T21:06:32.029 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000488199d3c72f029a-00646e7c58-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:06:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:32.029 DEBUG:boto:b'qskrps-8_myobj1000false_myobj2023-05-24T21:02:55.045Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:06:32.030 DEBUG:boto:path=/qskrps-8/_myobj 2023-05-24T21:06:32.030 DEBUG:boto:auth_path=/qskrps-8/_myobj 2023-05-24T21:06:32.030 DEBUG:boto:Method: HEAD 2023-05-24T21:06:32.031 DEBUG:boto:Path: /qskrps-8/_myobj 2023-05-24T21:06:32.031 DEBUG:boto:Data: 2023-05-24T21:06:32.031 DEBUG:boto:Headers: {} 2023-05-24T21:06:32.032 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:32.032 DEBUG:boto:Port: 8000 2023-05-24T21:06:32.033 DEBUG:boto:Params: {} 2023-05-24T21:06:32.033 DEBUG:boto:Token: None 2023-05-24T21:06:32.033 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:06:32 GMT /qskrps-8/_myobj 2023-05-24T21:06:32.034 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:AIoMCpfG/VOE0ZPsPI1sqqhLUPE= 2023-05-24T21:06:32.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:32 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:AIoMCpfG/VOE0ZPsPI1sqqhLUPE=', 'Content-Length': '0'} 2023-05-24T21:06:32.036 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d3ecdce77785bb49-00646e7c58-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:32.036 DEBUG:boto:path=/qskrps-8/_myobj 2023-05-24T21:06:32.037 DEBUG:boto:auth_path=/qskrps-8/_myobj 2023-05-24T21:06:32.037 DEBUG:boto:Method: HEAD 2023-05-24T21:06:32.037 DEBUG:boto:Path: /qskrps-8/_myobj 2023-05-24T21:06:32.038 DEBUG:boto:Data: 2023-05-24T21:06:32.038 DEBUG:boto:Headers: {} 2023-05-24T21:06:32.038 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:32.039 DEBUG:boto:Port: 8001 2023-05-24T21:06:32.039 DEBUG:boto:Params: {} 2023-05-24T21:06:32.040 DEBUG:boto:Token: None 2023-05-24T21:06:32.040 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:06:32 GMT /qskrps-8/_myobj 2023-05-24T21:06:32.040 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:AIoMCpfG/VOE0ZPsPI1sqqhLUPE= 2023-05-24T21:06:32.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:32 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:AIoMCpfG/VOE0ZPsPI1sqqhLUPE=', 'Content-Length': '0'} 2023-05-24T21:06:32.042 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ebcaa3e5da75d67b-00646e7c58-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:32.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-05-24T21:06:32.043 DEBUG:boto:path=/qskrps-8/_myobj 2023-05-24T21:06:32.044 DEBUG:boto:auth_path=/qskrps-8/_myobj 2023-05-24T21:06:32.044 DEBUG:boto:Method: GET 2023-05-24T21:06:32.045 DEBUG:boto:Path: /qskrps-8/_myobj 2023-05-24T21:06:32.045 DEBUG:boto:Data: 2023-05-24T21:06:32.045 DEBUG:boto:Headers: {} 2023-05-24T21:06:32.046 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:32.046 DEBUG:boto:Port: 8000 2023-05-24T21:06:32.046 DEBUG:boto:Params: {} 2023-05-24T21:06:32.047 DEBUG:boto:Token: None 2023-05-24T21:06:32.047 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:32 GMT /qskrps-8/_myobj 2023-05-24T21:06:32.048 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:q5Vwr9iLC8EXC2ghHx1ZQNMbxR4= 2023-05-24T21:06:32.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:32 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:q5Vwr9iLC8EXC2ghHx1ZQNMbxR4=', 'Content-Length': '0'} 2023-05-24T21:06:32.050 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e5377b06a601a0a1-00646e7c58-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:32.050 DEBUG:boto:path=/qskrps-8/_myobj 2023-05-24T21:06:32.051 DEBUG:boto:auth_path=/qskrps-8/_myobj 2023-05-24T21:06:32.051 DEBUG:boto:Method: GET 2023-05-24T21:06:32.051 DEBUG:boto:Path: /qskrps-8/_myobj 2023-05-24T21:06:32.052 DEBUG:boto:Data: 2023-05-24T21:06:32.052 DEBUG:boto:Headers: {} 2023-05-24T21:06:32.052 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:32.053 DEBUG:boto:Port: 8001 2023-05-24T21:06:32.053 DEBUG:boto:Params: {} 2023-05-24T21:06:32.053 DEBUG:boto:Token: None 2023-05-24T21:06:32.054 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:32 GMT /qskrps-8/_myobj 2023-05-24T21:06:32.054 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:q5Vwr9iLC8EXC2ghHx1ZQNMbxR4= 2023-05-24T21:06:32.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:32 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:q5Vwr9iLC8EXC2ghHx1ZQNMbxR4=', 'Content-Length': '0'} 2023-05-24T21:06:32.056 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000037147aaee6b3bf61-00646e7c58-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:32.100 DEBUG:boto:path=/qskrps-8/ 2023-05-24T21:06:32.101 DEBUG:boto:auth_path=/qskrps-8/ 2023-05-24T21:06:32.102 DEBUG:boto:path=/qskrps-8/?prefix=_myobj 2023-05-24T21:06:32.102 DEBUG:boto:auth_path=/qskrps-8/?prefix=_myobj 2023-05-24T21:06:32.102 DEBUG:boto:Method: GET 2023-05-24T21:06:32.103 DEBUG:boto:Path: /qskrps-8/?prefix=_myobj 2023-05-24T21:06:32.103 DEBUG:boto:Data: 2023-05-24T21:06:32.103 DEBUG:boto:Headers: {} 2023-05-24T21:06:32.104 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:32.104 DEBUG:boto:Port: 8000 2023-05-24T21:06:32.104 DEBUG:boto:Params: {} 2023-05-24T21:06:32.105 DEBUG:boto:Token: None 2023-05-24T21:06:32.105 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:32 GMT /qskrps-8/ 2023-05-24T21:06:32.106 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:OcwObQ0DFuo/5vHy3Oe5+iBKi/g= 2023-05-24T21:06:32.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:32 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:OcwObQ0DFuo/5vHy3Oe5+iBKi/g=', 'Content-Length': '0'} 2023-05-24T21:06:32.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000464414c7a65166d6-00646e7c58-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:06:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:32.109 DEBUG:boto:b'qskrps-8_myobj1000false_myobj2023-05-24T21:02:55.045Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:06:32.110 DEBUG:boto:path=/qskrps-8/ 2023-05-24T21:06:32.110 DEBUG:boto:auth_path=/qskrps-8/ 2023-05-24T21:06:32.111 DEBUG:boto:path=/qskrps-8/?prefix=_myobj 2023-05-24T21:06:32.111 DEBUG:boto:auth_path=/qskrps-8/?prefix=_myobj 2023-05-24T21:06:32.112 DEBUG:boto:Method: GET 2023-05-24T21:06:32.112 DEBUG:boto:Path: /qskrps-8/?prefix=_myobj 2023-05-24T21:06:32.112 DEBUG:boto:Data: 2023-05-24T21:06:32.113 DEBUG:boto:Headers: {} 2023-05-24T21:06:32.113 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:32.113 DEBUG:boto:Port: 8001 2023-05-24T21:06:32.114 DEBUG:boto:Params: {} 2023-05-24T21:06:32.114 DEBUG:boto:Token: None 2023-05-24T21:06:32.115 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:32 GMT /qskrps-8/ 2023-05-24T21:06:32.115 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:OcwObQ0DFuo/5vHy3Oe5+iBKi/g= 2023-05-24T21:06:32.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:32 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:OcwObQ0DFuo/5vHy3Oe5+iBKi/g=', 'Content-Length': '0'} 2023-05-24T21:06:32.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002a16543d979145b-00646e7c58-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:06:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:32.118 DEBUG:boto:b'qskrps-8_myobj1000false_myobj2023-05-24T21:02:55.045Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:06:32.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-05-24T21:06:32.119 DEBUG:boto:path=/qskrps-8/myobj 2023-05-24T21:06:32.119 DEBUG:boto:auth_path=/qskrps-8/myobj 2023-05-24T21:06:32.120 DEBUG:boto:path=/qskrps-8/myobj?versionId=null 2023-05-24T21:06:32.120 DEBUG:boto:auth_path=/qskrps-8/myobj?versionId=null 2023-05-24T21:06:32.121 DEBUG:boto:Method: GET 2023-05-24T21:06:32.121 DEBUG:boto:Path: /qskrps-8/myobj?versionId=null 2023-05-24T21:06:32.121 DEBUG:boto:Data: 2023-05-24T21:06:32.122 DEBUG:boto:Headers: {} 2023-05-24T21:06:32.122 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:32.122 DEBUG:boto:Port: 8000 2023-05-24T21:06:32.123 DEBUG:boto:Params: {} 2023-05-24T21:06:32.123 DEBUG:boto:Token: None 2023-05-24T21:06:32.124 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:32 GMT /qskrps-8/myobj?versionId=null 2023-05-24T21:06:32.124 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:vtyALsOPai8iZ49ipE5Rqm1yRbo= 2023-05-24T21:06:32.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:32 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:vtyALsOPai8iZ49ipE5Rqm1yRbo=', 'Content-Length': '0'} 2023-05-24T21:06:32.126 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cef44b6955382dc4-00646e7c58-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:32.127 DEBUG:boto:path=/qskrps-8/myobj 2023-05-24T21:06:32.127 DEBUG:boto:auth_path=/qskrps-8/myobj 2023-05-24T21:06:32.128 DEBUG:boto:path=/qskrps-8/myobj?versionId=null 2023-05-24T21:06:32.128 DEBUG:boto:auth_path=/qskrps-8/myobj?versionId=null 2023-05-24T21:06:32.128 DEBUG:boto:Method: GET 2023-05-24T21:06:32.129 DEBUG:boto:Path: /qskrps-8/myobj?versionId=null 2023-05-24T21:06:32.129 DEBUG:boto:Data: 2023-05-24T21:06:32.130 DEBUG:boto:Headers: {} 2023-05-24T21:06:32.130 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:32.130 DEBUG:boto:Port: 8001 2023-05-24T21:06:32.131 DEBUG:boto:Params: {} 2023-05-24T21:06:32.131 DEBUG:boto:Token: None 2023-05-24T21:06:32.132 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:32 GMT /qskrps-8/myobj?versionId=null 2023-05-24T21:06:32.132 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:vtyALsOPai8iZ49ipE5Rqm1yRbo= 2023-05-24T21:06:32.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:32 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:vtyALsOPai8iZ49ipE5Rqm1yRbo=', 'Content-Length': '0'} 2023-05-24T21:06:32.134 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d6d8f84aa1c951e6-00646e7c58-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:32.135 DEBUG:boto:path=/qskrps-8/myobj 2023-05-24T21:06:32.135 DEBUG:boto:auth_path=/qskrps-8/myobj 2023-05-24T21:06:32.136 DEBUG:boto:path=/qskrps-8/myobj?versionId=null 2023-05-24T21:06:32.136 DEBUG:boto:auth_path=/qskrps-8/myobj?versionId=null 2023-05-24T21:06:32.137 DEBUG:boto:Method: HEAD 2023-05-24T21:06:32.137 DEBUG:boto:Path: /qskrps-8/myobj?versionId=null 2023-05-24T21:06:32.137 DEBUG:boto:Data: 2023-05-24T21:06:32.138 DEBUG:boto:Headers: {} 2023-05-24T21:06:32.138 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:32.138 DEBUG:boto:Port: 8000 2023-05-24T21:06:32.139 DEBUG:boto:Params: {} 2023-05-24T21:06:32.139 DEBUG:boto:Token: None 2023-05-24T21:06:32.139 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:06:32 GMT /qskrps-8/myobj?versionId=null 2023-05-24T21:06:32.140 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:gGGrG3F5Xj2a3SghtNlWjXVwxQE= 2023-05-24T21:06:32.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:32 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:gGGrG3F5Xj2a3SghtNlWjXVwxQE=', 'Content-Length': '0'} 2023-05-24T21:06:32.142 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bc1a665362011cc5-00646e7c58-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:32.142 DEBUG:boto:path=/qskrps-8/myobj 2023-05-24T21:06:32.143 DEBUG:boto:auth_path=/qskrps-8/myobj 2023-05-24T21:06:32.143 DEBUG:boto:path=/qskrps-8/myobj?versionId=null 2023-05-24T21:06:32.143 DEBUG:boto:auth_path=/qskrps-8/myobj?versionId=null 2023-05-24T21:06:32.144 DEBUG:boto:Method: HEAD 2023-05-24T21:06:32.144 DEBUG:boto:Path: /qskrps-8/myobj?versionId=null 2023-05-24T21:06:32.144 DEBUG:boto:Data: 2023-05-24T21:06:32.145 DEBUG:boto:Headers: {} 2023-05-24T21:06:32.145 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:32.145 DEBUG:boto:Port: 8001 2023-05-24T21:06:32.146 DEBUG:boto:Params: {} 2023-05-24T21:06:32.146 DEBUG:boto:Token: None 2023-05-24T21:06:32.146 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:06:32 GMT /qskrps-8/myobj?versionId=null 2023-05-24T21:06:32.147 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:gGGrG3F5Xj2a3SghtNlWjXVwxQE= 2023-05-24T21:06:32.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:32 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:gGGrG3F5Xj2a3SghtNlWjXVwxQE=', 'Content-Length': '0'} 2023-05-24T21:06:32.149 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000081d78c3fd5155994-00646e7c58-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:32.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-05-24T21:06:32.150 DEBUG:boto:path=/qskrps-8/myobj 2023-05-24T21:06:32.150 DEBUG:boto:auth_path=/qskrps-8/myobj 2023-05-24T21:06:32.150 DEBUG:boto:path=/qskrps-8/myobj?versionId=null 2023-05-24T21:06:32.151 DEBUG:boto:auth_path=/qskrps-8/myobj?versionId=null 2023-05-24T21:06:32.151 DEBUG:boto:Method: GET 2023-05-24T21:06:32.151 DEBUG:boto:Path: /qskrps-8/myobj?versionId=null 2023-05-24T21:06:32.152 DEBUG:boto:Data: 2023-05-24T21:06:32.152 DEBUG:boto:Headers: {} 2023-05-24T21:06:32.153 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:32.153 DEBUG:boto:Port: 8000 2023-05-24T21:06:32.153 DEBUG:boto:Params: {} 2023-05-24T21:06:32.154 DEBUG:boto:Token: None 2023-05-24T21:06:32.154 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:32 GMT /qskrps-8/myobj?versionId=null 2023-05-24T21:06:32.154 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:vtyALsOPai8iZ49ipE5Rqm1yRbo= 2023-05-24T21:06:32.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:32 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:vtyALsOPai8iZ49ipE5Rqm1yRbo=', 'Content-Length': '0'} 2023-05-24T21:06:32.157 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000816f97c804ef881f-00646e7c58-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:32.157 DEBUG:boto:path=/qskrps-8/myobj 2023-05-24T21:06:32.158 DEBUG:boto:auth_path=/qskrps-8/myobj 2023-05-24T21:06:32.158 DEBUG:boto:path=/qskrps-8/myobj?versionId=null 2023-05-24T21:06:32.159 DEBUG:boto:auth_path=/qskrps-8/myobj?versionId=null 2023-05-24T21:06:32.159 DEBUG:boto:Method: GET 2023-05-24T21:06:32.159 DEBUG:boto:Path: /qskrps-8/myobj?versionId=null 2023-05-24T21:06:32.160 DEBUG:boto:Data: 2023-05-24T21:06:32.160 DEBUG:boto:Headers: {} 2023-05-24T21:06:32.161 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:32.161 DEBUG:boto:Port: 8001 2023-05-24T21:06:32.161 DEBUG:boto:Params: {} 2023-05-24T21:06:32.162 DEBUG:boto:Token: None 2023-05-24T21:06:32.162 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:32 GMT /qskrps-8/myobj?versionId=null 2023-05-24T21:06:32.162 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:vtyALsOPai8iZ49ipE5Rqm1yRbo= 2023-05-24T21:06:32.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:32 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:vtyALsOPai8iZ49ipE5Rqm1yRbo=', 'Content-Length': '0'} 2023-05-24T21:06:32.165 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000707a839834c28d8d-00646e7c58-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:32.165 DEBUG:boto:path=/qskrps-8/ 2023-05-24T21:06:32.166 DEBUG:boto:auth_path=/qskrps-8/ 2023-05-24T21:06:32.166 DEBUG:boto:path=/qskrps-8/?prefix=myobj 2023-05-24T21:06:32.167 DEBUG:boto:auth_path=/qskrps-8/?prefix=myobj 2023-05-24T21:06:32.167 DEBUG:boto:Method: GET 2023-05-24T21:06:32.167 DEBUG:boto:Path: /qskrps-8/?prefix=myobj 2023-05-24T21:06:32.168 DEBUG:boto:Data: 2023-05-24T21:06:32.168 DEBUG:boto:Headers: {} 2023-05-24T21:06:32.168 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:32.169 DEBUG:boto:Port: 8000 2023-05-24T21:06:32.169 DEBUG:boto:Params: {} 2023-05-24T21:06:32.169 DEBUG:boto:Token: None 2023-05-24T21:06:32.170 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:32 GMT /qskrps-8/ 2023-05-24T21:06:32.170 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:OcwObQ0DFuo/5vHy3Oe5+iBKi/g= 2023-05-24T21:06:32.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:32 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:OcwObQ0DFuo/5vHy3Oe5+iBKi/g=', 'Content-Length': '0'} 2023-05-24T21:06:32.173 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d10735b49c944a92-00646e7c58-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:06:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:32.216 DEBUG:boto:b'qskrps-8myobj1000falsemyobj2023-05-24T21:02:55.012Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:06:32.217 DEBUG:boto:path=/qskrps-8/ 2023-05-24T21:06:32.217 DEBUG:boto:auth_path=/qskrps-8/ 2023-05-24T21:06:32.218 DEBUG:boto:path=/qskrps-8/?prefix=myobj 2023-05-24T21:06:32.218 DEBUG:boto:auth_path=/qskrps-8/?prefix=myobj 2023-05-24T21:06:32.218 DEBUG:boto:Method: GET 2023-05-24T21:06:32.219 DEBUG:boto:Path: /qskrps-8/?prefix=myobj 2023-05-24T21:06:32.219 DEBUG:boto:Data: 2023-05-24T21:06:32.219 DEBUG:boto:Headers: {} 2023-05-24T21:06:32.220 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:32.220 DEBUG:boto:Port: 8001 2023-05-24T21:06:32.221 DEBUG:boto:Params: {} 2023-05-24T21:06:32.221 DEBUG:boto:Token: None 2023-05-24T21:06:32.222 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:32 GMT /qskrps-8/ 2023-05-24T21:06:32.222 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:OcwObQ0DFuo/5vHy3Oe5+iBKi/g= 2023-05-24T21:06:32.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:32 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:OcwObQ0DFuo/5vHy3Oe5+iBKi/g=', 'Content-Length': '0'} 2023-05-24T21:06:32.225 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005542eafae397bb9b-00646e7c58-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:06:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:32.226 DEBUG:boto:b'qskrps-8myobj1000falsemyobj2023-05-24T21:02:55.012Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:06:32.227 DEBUG:boto:path=/qskrps-8/myobj 2023-05-24T21:06:32.227 DEBUG:boto:auth_path=/qskrps-8/myobj 2023-05-24T21:06:32.227 DEBUG:boto:Method: HEAD 2023-05-24T21:06:32.228 DEBUG:boto:Path: /qskrps-8/myobj 2023-05-24T21:06:32.228 DEBUG:boto:Data: 2023-05-24T21:06:32.228 DEBUG:boto:Headers: {} 2023-05-24T21:06:32.229 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:32.229 DEBUG:boto:Port: 8000 2023-05-24T21:06:32.229 DEBUG:boto:Params: {} 2023-05-24T21:06:32.230 DEBUG:boto:Token: None 2023-05-24T21:06:32.230 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:06:32 GMT /qskrps-8/myobj 2023-05-24T21:06:32.230 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:oOaB+COsm/Vchfb0rUqyl5qO0eY= 2023-05-24T21:06:32.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:32 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:oOaB+COsm/Vchfb0rUqyl5qO0eY=', 'Content-Length': '0'} 2023-05-24T21:06:32.232 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b38eb759ef1e018-00646e7c58-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:32.233 DEBUG:boto:path=/qskrps-8/myobj 2023-05-24T21:06:32.233 DEBUG:boto:auth_path=/qskrps-8/myobj 2023-05-24T21:06:32.234 DEBUG:boto:Method: HEAD 2023-05-24T21:06:32.234 DEBUG:boto:Path: /qskrps-8/myobj 2023-05-24T21:06:32.234 DEBUG:boto:Data: 2023-05-24T21:06:32.235 DEBUG:boto:Headers: {} 2023-05-24T21:06:32.235 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:32.236 DEBUG:boto:Port: 8001 2023-05-24T21:06:32.236 DEBUG:boto:Params: {} 2023-05-24T21:06:32.236 DEBUG:boto:Token: None 2023-05-24T21:06:32.237 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:06:32 GMT /qskrps-8/myobj 2023-05-24T21:06:32.237 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:oOaB+COsm/Vchfb0rUqyl5qO0eY= 2023-05-24T21:06:32.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:32 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:oOaB+COsm/Vchfb0rUqyl5qO0eY=', 'Content-Length': '0'} 2023-05-24T21:06:32.239 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000287dc9d212a37988-00646e7c58-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:32.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-05-24T21:06:32.240 DEBUG:boto:path=/qskrps-8/myobj 2023-05-24T21:06:32.240 DEBUG:boto:auth_path=/qskrps-8/myobj 2023-05-24T21:06:32.241 DEBUG:boto:Method: GET 2023-05-24T21:06:32.241 DEBUG:boto:Path: /qskrps-8/myobj 2023-05-24T21:06:32.242 DEBUG:boto:Data: 2023-05-24T21:06:32.242 DEBUG:boto:Headers: {} 2023-05-24T21:06:32.242 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:32.243 DEBUG:boto:Port: 8000 2023-05-24T21:06:32.243 DEBUG:boto:Params: {} 2023-05-24T21:06:32.244 DEBUG:boto:Token: None 2023-05-24T21:06:32.244 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:32 GMT /qskrps-8/myobj 2023-05-24T21:06:32.244 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:NukB2FmT4XcR7NVs0dwjgwvfTVQ= 2023-05-24T21:06:32.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:32 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:NukB2FmT4XcR7NVs0dwjgwvfTVQ=', 'Content-Length': '0'} 2023-05-24T21:06:32.247 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000078e38c8ae2b3856c-00646e7c58-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:32.247 DEBUG:boto:path=/qskrps-8/myobj 2023-05-24T21:06:32.248 DEBUG:boto:auth_path=/qskrps-8/myobj 2023-05-24T21:06:32.248 DEBUG:boto:Method: GET 2023-05-24T21:06:32.249 DEBUG:boto:Path: /qskrps-8/myobj 2023-05-24T21:06:32.249 DEBUG:boto:Data: 2023-05-24T21:06:32.249 DEBUG:boto:Headers: {} 2023-05-24T21:06:32.250 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:32.250 DEBUG:boto:Port: 8001 2023-05-24T21:06:32.250 DEBUG:boto:Params: {} 2023-05-24T21:06:32.251 DEBUG:boto:Token: None 2023-05-24T21:06:32.251 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:32 GMT /qskrps-8/myobj 2023-05-24T21:06:32.252 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:NukB2FmT4XcR7NVs0dwjgwvfTVQ= 2023-05-24T21:06:32.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:32 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:NukB2FmT4XcR7NVs0dwjgwvfTVQ=', 'Content-Length': '0'} 2023-05-24T21:06:32.254 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:02:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009ae5df1d67a10174-00646e7c58-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:32.296 DEBUG:boto:path=/qskrps-8/ 2023-05-24T21:06:32.297 DEBUG:boto:auth_path=/qskrps-8/ 2023-05-24T21:06:32.297 DEBUG:boto:path=/qskrps-8/?prefix=myobj 2023-05-24T21:06:32.297 DEBUG:boto:auth_path=/qskrps-8/?prefix=myobj 2023-05-24T21:06:32.298 DEBUG:boto:Method: GET 2023-05-24T21:06:32.298 DEBUG:boto:Path: /qskrps-8/?prefix=myobj 2023-05-24T21:06:32.299 DEBUG:boto:Data: 2023-05-24T21:06:32.299 DEBUG:boto:Headers: {} 2023-05-24T21:06:32.299 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:32.300 DEBUG:boto:Port: 8000 2023-05-24T21:06:32.300 DEBUG:boto:Params: {} 2023-05-24T21:06:32.301 DEBUG:boto:Token: None 2023-05-24T21:06:32.301 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:32 GMT /qskrps-8/ 2023-05-24T21:06:32.301 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:OcwObQ0DFuo/5vHy3Oe5+iBKi/g= 2023-05-24T21:06:32.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:32 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:OcwObQ0DFuo/5vHy3Oe5+iBKi/g=', 'Content-Length': '0'} 2023-05-24T21:06:32.305 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020d46b6eb0613e69-00646e7c58-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:06:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:32.305 DEBUG:boto:b'qskrps-8myobj1000falsemyobj2023-05-24T21:02:55.012Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:06:32.306 DEBUG:boto:path=/qskrps-8/ 2023-05-24T21:06:32.306 DEBUG:boto:auth_path=/qskrps-8/ 2023-05-24T21:06:32.307 DEBUG:boto:path=/qskrps-8/?prefix=myobj 2023-05-24T21:06:32.307 DEBUG:boto:auth_path=/qskrps-8/?prefix=myobj 2023-05-24T21:06:32.307 DEBUG:boto:Method: GET 2023-05-24T21:06:32.308 DEBUG:boto:Path: /qskrps-8/?prefix=myobj 2023-05-24T21:06:32.308 DEBUG:boto:Data: 2023-05-24T21:06:32.309 DEBUG:boto:Headers: {} 2023-05-24T21:06:32.309 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:32.309 DEBUG:boto:Port: 8001 2023-05-24T21:06:32.310 DEBUG:boto:Params: {} 2023-05-24T21:06:32.310 DEBUG:boto:Token: None 2023-05-24T21:06:32.310 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:32 GMT /qskrps-8/ 2023-05-24T21:06:32.311 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:OcwObQ0DFuo/5vHy3Oe5+iBKi/g= 2023-05-24T21:06:32.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:32 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:OcwObQ0DFuo/5vHy3Oe5+iBKi/g=', 'Content-Length': '0'} 2023-05-24T21:06:32.314 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058cf2a8c2ea993ce-00646e7c58-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:06:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:32.314 DEBUG:boto:b'qskrps-8myobj1000falsemyobj2023-05-24T21:02:55.012Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:06:32.315 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qskrps-8 zones={a1, a2} 2023-05-24T21:06:32.316 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_sync ... ok 2023-05-24T21:06:32.317 INFO:rgw_multi.tests:create bucket zone=a1 name=qskrps-9 2023-05-24T21:06:32.318 DEBUG:boto:path=/qskrps-9/ 2023-05-24T21:06:32.318 DEBUG:boto:auth_path=/qskrps-9/ 2023-05-24T21:06:32.319 DEBUG:boto:Method: PUT 2023-05-24T21:06:32.319 DEBUG:boto:Path: /qskrps-9/ 2023-05-24T21:06:32.319 DEBUG:boto:Data: 2023-05-24T21:06:32.320 DEBUG:boto:Headers: {} 2023-05-24T21:06:32.320 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:32.320 DEBUG:boto:Port: 8000 2023-05-24T21:06:32.321 DEBUG:boto:Params: {} 2023-05-24T21:06:32.321 DEBUG:boto:Token: None 2023-05-24T21:06:32.321 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:06:32 GMT /qskrps-9/ 2023-05-24T21:06:32.322 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ddZToupVtiUsFfXLppuV0HKegBY= 2023-05-24T21:06:32.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:32 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ddZToupVtiUsFfXLppuV0HKegBY=', 'Content-Length': '0'} 2023-05-24T21:06:32.342 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001fa7445f3e570439-00646e7c58-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:06:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:32.342 INFO:rgw_multi.tests:create bucket zone=a2 name=qskrps-10 2023-05-24T21:06:32.343 DEBUG:boto:path=/qskrps-10/ 2023-05-24T21:06:32.343 DEBUG:boto:auth_path=/qskrps-10/ 2023-05-24T21:06:32.344 DEBUG:boto:Method: PUT 2023-05-24T21:06:32.344 DEBUG:boto:Path: /qskrps-10/ 2023-05-24T21:06:32.344 DEBUG:boto:Data: 2023-05-24T21:06:32.345 DEBUG:boto:Headers: {} 2023-05-24T21:06:32.345 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:32.345 DEBUG:boto:Port: 8001 2023-05-24T21:06:32.346 DEBUG:boto:Params: {} 2023-05-24T21:06:32.346 DEBUG:boto:Token: None 2023-05-24T21:06:32.346 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:06:32 GMT /qskrps-10/ 2023-05-24T21:06:32.347 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:MEmdHdFcFlZ/PKdNtSe/MFYwPac= 2023-05-24T21:06:32.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:32 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:MEmdHdFcFlZ/PKdNtSe/MFYwPac=', 'Content-Length': '0'} 2023-05-24T21:06:32.373 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d550a7106ed12790-00646e7c58-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:06:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:32.374 DEBUG:boto:path=/qskrps-9/ 2023-05-24T21:06:32.374 DEBUG:boto:auth_path=/qskrps-9/ 2023-05-24T21:06:32.374 DEBUG:boto:Method: HEAD 2023-05-24T21:06:32.375 DEBUG:boto:Path: /qskrps-9/ 2023-05-24T21:06:32.375 DEBUG:boto:Data: 2023-05-24T21:06:32.375 DEBUG:boto:Headers: {} 2023-05-24T21:06:32.376 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:32.376 DEBUG:boto:Port: 8000 2023-05-24T21:06:32.376 DEBUG:boto:Params: {} 2023-05-24T21:06:32.377 DEBUG:boto:Token: None 2023-05-24T21:06:32.377 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:06:32 GMT /qskrps-9/ 2023-05-24T21:06:32.378 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:1O3zC5kX8gKmvEegtkrZuKFTcVg= 2023-05-24T21:06:32.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:32 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:1O3zC5kX8gKmvEegtkrZuKFTcVg=', 'Content-Length': '0'} 2023-05-24T21:06:32.380 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004a1d9e46948f267f-00646e7c58-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:06:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:32.381 DEBUG:boto:path=/qskrps-9/myobj 2023-05-24T21:06:32.381 DEBUG:boto:auth_path=/qskrps-9/myobj 2023-05-24T21:06:32.382 DEBUG:boto:Method: PUT 2023-05-24T21:06:32.382 DEBUG:boto:Path: /qskrps-9/myobj 2023-05-24T21:06:32.382 DEBUG:boto:Data: 2023-05-24T21:06:32.383 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-24T21:06:32.383 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:32.383 DEBUG:boto:Port: 8000 2023-05-24T21:06:32.384 DEBUG:boto:Params: {} 2023-05-24T21:06:32.384 DEBUG:boto:Token: None 2023-05-24T21:06:32.384 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 24 May 2023 21:06:32 GMT /qskrps-9/myobj 2023-05-24T21:06:32.385 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:FE4xrJKS7rImccDMpV6ycRrdflg= 2023-05-24T21:06:32.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:06:32 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:FE4xrJKS7rImccDMpV6ycRrdflg='} 2023-05-24T21:06:32.428 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000283ff977df273145-00646e7c58-110d-a1'), ('Date', 'Wed, 24 May 2023 21:06:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:32.429 DEBUG:boto:path=/qskrps-10/ 2023-05-24T21:06:32.429 DEBUG:boto:auth_path=/qskrps-10/ 2023-05-24T21:06:32.430 DEBUG:boto:Method: HEAD 2023-05-24T21:06:32.430 DEBUG:boto:Path: /qskrps-10/ 2023-05-24T21:06:32.430 DEBUG:boto:Data: 2023-05-24T21:06:32.431 DEBUG:boto:Headers: {} 2023-05-24T21:06:32.431 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:32.431 DEBUG:boto:Port: 8001 2023-05-24T21:06:32.432 DEBUG:boto:Params: {} 2023-05-24T21:06:32.432 DEBUG:boto:Token: None 2023-05-24T21:06:32.433 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:06:32 GMT /qskrps-10/ 2023-05-24T21:06:32.433 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:x/xD6cONHPAy1rWAXClWf6cJiS0= 2023-05-24T21:06:32.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:32 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:x/xD6cONHPAy1rWAXClWf6cJiS0=', 'Content-Length': '0'} 2023-05-24T21:06:32.436 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e04525a8364592bd-00646e7c58-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:06:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:32.436 DEBUG:boto:path=/qskrps-10/myobj 2023-05-24T21:06:32.437 DEBUG:boto:auth_path=/qskrps-10/myobj 2023-05-24T21:06:32.437 DEBUG:boto:Method: PUT 2023-05-24T21:06:32.437 DEBUG:boto:Path: /qskrps-10/myobj 2023-05-24T21:06:32.438 DEBUG:boto:Data: 2023-05-24T21:06:32.438 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-24T21:06:32.438 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:32.439 DEBUG:boto:Port: 8001 2023-05-24T21:06:32.439 DEBUG:boto:Params: {} 2023-05-24T21:06:32.439 DEBUG:boto:Token: None 2023-05-24T21:06:32.440 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 24 May 2023 21:06:32 GMT /qskrps-10/myobj 2023-05-24T21:06:32.440 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:sxazkR7BzP7CfQqXDOIi6R0JXq0= 2023-05-24T21:06:32.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:06:32 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:sxazkR7BzP7CfQqXDOIi6R0JXq0='} 2023-05-24T21:06:32.450 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c8e59a9c9080254b-00646e7c58-1111-a2'), ('Date', 'Wed, 24 May 2023 21:06:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:32.451 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:06:32.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:06:32.452 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:06:32.502 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:06:32.502 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:06:32.543 INFO:teuthology.orchestra.run.smithi071.stderr:No --period given, using current period=5c04936b-70ea-4415-b4eb-3aad3dc612e5 2023-05-24T21:06:32.562 INFO:teuthology.orchestra.run.smithi071.stdout:[ 2023-05-24T21:06:32.563 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:06:32.563 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.358207_90.1", 2023-05-24T21:06:32.563 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:06:32.358207Z" 2023-05-24T21:06:32.564 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:06:32.564 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:06:32.564 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:06:32.564 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:06:32.333844Z" 2023-05-24T21:06:32.564 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:06:32.564 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:06:32.565 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:06:32.565 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:06:32.565 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:06:32.565 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:06:32.565 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:06:32.565 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:06:32.565 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:06:32.566 INFO:teuthology.orchestra.run.smithi071.stdout:] 2023-05-24T21:06:32.566 DEBUG:rgw_multi.tests:master meta markers={0: '1_1684962392.358207_90.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:06:32.567 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-24T21:06:32.567 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:06:32.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:06:32.568 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:06:32.617 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:06:32.617 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:06:32.658 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:06:32.652+0000 7fdc26111780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:06:32.659 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:06:32.653+0000 7fdc26111780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:06:32.675 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:06:32.675 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:06:32.675 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:06:32.675 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:06:32.676 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:06:32.676 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:06:32.676 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:06:32.676 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:06:32.676 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:06:32.676 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:06:32.677 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:06:32.677 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:06:32.677 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:06:32.677 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.358207_90.1", 2023-05-24T21:06:32.677 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:06:32.677 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:06:32.677 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:06:32.678 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.358207Z", 2023-05-24T21:06:32.678 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:06:32.678 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:06:32.678 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:06:32.678 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:06:32.678 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:06:32.679 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:06:32.679 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:06:32.679 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:06:32.679 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:06:32.679 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:06:32.679 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:06:32.679 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:06:32.680 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:06:32.680 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:06:32.680 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:06:32.680 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:06:32.680 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:06:32.680 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:06:32.680 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:06:32.681 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:06:32.681 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:06:32.681 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:06:32.681 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:06:32.681 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:06:32.681 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:06:32.681 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:06:32.682 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:06:32.682 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:06:32.682 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:06:32.682 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:06:32.682 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:06:32.682 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:06:32.682 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:06:32.683 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:06:32.683 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:06:32.683 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:06:32.683 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:06:32.683 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:06:32.683 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:06:32.683 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:06:32.684 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:06:32.684 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:06:32.684 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:06:32.684 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:06:32.684 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:06:32.684 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:06:32Z" 2023-05-24T21:06:32.684 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:06:32.685 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962392.358207_90.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:06:32.358207Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:06:32Z" } 2023-05-24T21:06:32.686 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962392.358207_90.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.358207Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:06:32.686 DEBUG:rgw_multi.tests:log_status={0: '1_1684962392.358207_90.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:06:32.687 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962392.358207_90.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:06:32.687 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-24T21:06:32.687 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:06:32.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:06:32.688 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-9 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:06:32.744 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:06:32.745 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:06:32.778 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:06:32.772+0000 7f7df0111780 1 waiting to reach incremental sync.. 2023-05-24T21:06:37.779 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:06:37.773+0000 7f7df0111780 1 waiting to reach incremental sync.. 2023-05-24T21:06:42.780 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:06:42.774+0000 7f7df0111780 1 waiting to reach incremental sync.. 2023-05-24T21:06:47.780 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:06:47.774+0000 7f7df0111780 1 waiting to reach incremental sync.. 2023-05-24T21:06:52.782 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:06:52.776+0000 7f7df0111780 1 bucket sync caught up with source: 2023-05-24T21:06:52.782 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000001.4.6, , , , , , , , , ] 2023-05-24T21:06:52.782 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000001.4.6, , , , , , , , , ] 2023-05-24T21:06:52.782 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:06:52.776+0000 7f7df0111780 0 bucket checkpoint complete 2023-05-24T21:06:52.797 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qskrps-9 zones={a2, a1} 2023-05-24T21:06:52.798 DEBUG:boto:path=/qskrps-9/ 2023-05-24T21:06:52.798 DEBUG:boto:auth_path=/qskrps-9/ 2023-05-24T21:06:52.799 DEBUG:boto:Method: HEAD 2023-05-24T21:06:52.799 DEBUG:boto:Path: /qskrps-9/ 2023-05-24T21:06:52.799 DEBUG:boto:Data: 2023-05-24T21:06:52.800 DEBUG:boto:Headers: {} 2023-05-24T21:06:52.800 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:52.800 DEBUG:boto:Port: 8001 2023-05-24T21:06:52.801 DEBUG:boto:Params: {} 2023-05-24T21:06:52.801 DEBUG:boto:Token: None 2023-05-24T21:06:52.802 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:06:52 GMT /qskrps-9/ 2023-05-24T21:06:52.802 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Jr4fUAGaDCLACjjwirYlTqEv7UE= 2023-05-24T21:06:52.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Jr4fUAGaDCLACjjwirYlTqEv7UE=', 'Content-Length': '0'} 2023-05-24T21:06:52.805 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000038edc3c81feaef26-00646e7c6c-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:06:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:52.805 DEBUG:boto:path=/qskrps-9/ 2023-05-24T21:06:52.806 DEBUG:boto:auth_path=/qskrps-9/ 2023-05-24T21:06:52.806 DEBUG:boto:Method: HEAD 2023-05-24T21:06:52.807 DEBUG:boto:Path: /qskrps-9/ 2023-05-24T21:06:52.807 DEBUG:boto:Data: 2023-05-24T21:06:52.807 DEBUG:boto:Headers: {} 2023-05-24T21:06:52.808 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:52.808 DEBUG:boto:Port: 8000 2023-05-24T21:06:52.808 DEBUG:boto:Params: {} 2023-05-24T21:06:52.809 DEBUG:boto:Token: None 2023-05-24T21:06:52.809 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:06:52 GMT /qskrps-9/ 2023-05-24T21:06:52.810 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Jr4fUAGaDCLACjjwirYlTqEv7UE= 2023-05-24T21:06:52.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Jr4fUAGaDCLACjjwirYlTqEv7UE=', 'Content-Length': '0'} 2023-05-24T21:06:52.812 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000047e236d3897363f7-00646e7c6c-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:06:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:52.813 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-24T21:06:52.813 DEBUG:boto:path=/qskrps-9/ 2023-05-24T21:06:52.814 DEBUG:boto:auth_path=/qskrps-9/ 2023-05-24T21:06:52.814 DEBUG:boto:path=/qskrps-9/?versions&max-keys=999 2023-05-24T21:06:52.814 DEBUG:boto:auth_path=/qskrps-9/?versions&max-keys=999 2023-05-24T21:06:52.815 DEBUG:boto:Method: GET 2023-05-24T21:06:52.815 DEBUG:boto:Path: /qskrps-9/?versions&max-keys=999 2023-05-24T21:06:52.815 DEBUG:boto:Data: 2023-05-24T21:06:52.816 DEBUG:boto:Headers: {} 2023-05-24T21:06:52.816 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:52.816 DEBUG:boto:Port: 8001 2023-05-24T21:06:52.817 DEBUG:boto:Params: {} 2023-05-24T21:06:52.817 DEBUG:boto:Token: None 2023-05-24T21:06:52.818 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:52 GMT /qskrps-9/?versions 2023-05-24T21:06:52.818 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:L9mp2lZLI2mTIRcw2rwv6KnAc+s= 2023-05-24T21:06:52.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:L9mp2lZLI2mTIRcw2rwv6KnAc+s=', 'Content-Length': '0'} 2023-05-24T21:06:52.820 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000111ce85dbd439860-00646e7c6c-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:06:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:52.864 DEBUG:boto:b'qskrps-9999falsemyobjnulltrue2023-05-24T21:06:32.383Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:06:52.865 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-05-24T21:06:52.866 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-24T21:06:52.866 DEBUG:boto:path=/qskrps-9/ 2023-05-24T21:06:52.866 DEBUG:boto:auth_path=/qskrps-9/ 2023-05-24T21:06:52.867 DEBUG:boto:path=/qskrps-9/?versions&max-keys=999 2023-05-24T21:06:52.867 DEBUG:boto:auth_path=/qskrps-9/?versions&max-keys=999 2023-05-24T21:06:52.868 DEBUG:boto:Method: GET 2023-05-24T21:06:52.868 DEBUG:boto:Path: /qskrps-9/?versions&max-keys=999 2023-05-24T21:06:52.868 DEBUG:boto:Data: 2023-05-24T21:06:52.869 DEBUG:boto:Headers: {} 2023-05-24T21:06:52.869 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:52.869 DEBUG:boto:Port: 8000 2023-05-24T21:06:52.870 DEBUG:boto:Params: {} 2023-05-24T21:06:52.870 DEBUG:boto:Token: None 2023-05-24T21:06:52.871 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:52 GMT /qskrps-9/?versions 2023-05-24T21:06:52.871 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:L9mp2lZLI2mTIRcw2rwv6KnAc+s= 2023-05-24T21:06:52.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:L9mp2lZLI2mTIRcw2rwv6KnAc+s=', 'Content-Length': '0'} 2023-05-24T21:06:52.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033de3f1bb88a4a98-00646e7c6c-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:06:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:52.875 DEBUG:boto:b'qskrps-9999falsemyobjnulltrue2023-05-24T21:06:32.383Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:06:52.875 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-05-24T21:06:52.876 DEBUG:boto:path=/qskrps-9/ 2023-05-24T21:06:52.876 DEBUG:boto:auth_path=/qskrps-9/ 2023-05-24T21:06:52.877 DEBUG:boto:path=/qskrps-9/?versions&max-keys=999 2023-05-24T21:06:52.877 DEBUG:boto:auth_path=/qskrps-9/?versions&max-keys=999 2023-05-24T21:06:52.877 DEBUG:boto:Method: GET 2023-05-24T21:06:52.878 DEBUG:boto:Path: /qskrps-9/?versions&max-keys=999 2023-05-24T21:06:52.878 DEBUG:boto:Data: 2023-05-24T21:06:52.878 DEBUG:boto:Headers: {} 2023-05-24T21:06:52.879 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:52.879 DEBUG:boto:Port: 8001 2023-05-24T21:06:52.879 DEBUG:boto:Params: {} 2023-05-24T21:06:52.880 DEBUG:boto:Token: None 2023-05-24T21:06:52.880 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:52 GMT /qskrps-9/?versions 2023-05-24T21:06:52.881 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:L9mp2lZLI2mTIRcw2rwv6KnAc+s= 2023-05-24T21:06:52.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:L9mp2lZLI2mTIRcw2rwv6KnAc+s=', 'Content-Length': '0'} 2023-05-24T21:06:52.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067cb0dc768c0c4e0-00646e7c6c-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:06:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:52.884 DEBUG:boto:b'qskrps-9999falsemyobjnulltrue2023-05-24T21:06:32.383Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:06:52.885 DEBUG:boto:path=/qskrps-9/ 2023-05-24T21:06:52.885 DEBUG:boto:auth_path=/qskrps-9/ 2023-05-24T21:06:52.885 DEBUG:boto:path=/qskrps-9/?versions&max-keys=999 2023-05-24T21:06:52.886 DEBUG:boto:auth_path=/qskrps-9/?versions&max-keys=999 2023-05-24T21:06:52.886 DEBUG:boto:Method: GET 2023-05-24T21:06:52.887 DEBUG:boto:Path: /qskrps-9/?versions&max-keys=999 2023-05-24T21:06:52.887 DEBUG:boto:Data: 2023-05-24T21:06:52.887 DEBUG:boto:Headers: {} 2023-05-24T21:06:52.888 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:52.888 DEBUG:boto:Port: 8000 2023-05-24T21:06:52.888 DEBUG:boto:Params: {} 2023-05-24T21:06:52.889 DEBUG:boto:Token: None 2023-05-24T21:06:52.889 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:52 GMT /qskrps-9/?versions 2023-05-24T21:06:52.890 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:L9mp2lZLI2mTIRcw2rwv6KnAc+s= 2023-05-24T21:06:52.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:L9mp2lZLI2mTIRcw2rwv6KnAc+s=', 'Content-Length': '0'} 2023-05-24T21:06:52.892 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c79091bd2a6134d2-00646e7c6c-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:06:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:52.893 DEBUG:boto:b'qskrps-9999falsemyobjnulltrue2023-05-24T21:06:32.383Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:06:52.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-05-24T21:06:52.894 DEBUG:boto:path=/qskrps-9/myobj 2023-05-24T21:06:52.894 DEBUG:boto:auth_path=/qskrps-9/myobj 2023-05-24T21:06:52.894 DEBUG:boto:path=/qskrps-9/myobj?versionId=null 2023-05-24T21:06:52.895 DEBUG:boto:auth_path=/qskrps-9/myobj?versionId=null 2023-05-24T21:06:52.895 DEBUG:boto:Method: GET 2023-05-24T21:06:52.896 DEBUG:boto:Path: /qskrps-9/myobj?versionId=null 2023-05-24T21:06:52.896 DEBUG:boto:Data: 2023-05-24T21:06:52.896 DEBUG:boto:Headers: {} 2023-05-24T21:06:52.897 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:52.897 DEBUG:boto:Port: 8001 2023-05-24T21:06:52.897 DEBUG:boto:Params: {} 2023-05-24T21:06:52.898 DEBUG:boto:Token: None 2023-05-24T21:06:52.898 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:52 GMT /qskrps-9/myobj?versionId=null 2023-05-24T21:06:52.899 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:IZVGGpacee4MUTd/u6WWbJfIRsg= 2023-05-24T21:06:52.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:IZVGGpacee4MUTd/u6WWbJfIRsg=', 'Content-Length': '0'} 2023-05-24T21:06:52.902 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:06:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c2278303d77a0126-00646e7c6c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:52.902 DEBUG:boto:path=/qskrps-9/myobj 2023-05-24T21:06:52.903 DEBUG:boto:auth_path=/qskrps-9/myobj 2023-05-24T21:06:52.903 DEBUG:boto:path=/qskrps-9/myobj?versionId=null 2023-05-24T21:06:52.904 DEBUG:boto:auth_path=/qskrps-9/myobj?versionId=null 2023-05-24T21:06:52.904 DEBUG:boto:Method: GET 2023-05-24T21:06:52.904 DEBUG:boto:Path: /qskrps-9/myobj?versionId=null 2023-05-24T21:06:52.905 DEBUG:boto:Data: 2023-05-24T21:06:52.905 DEBUG:boto:Headers: {} 2023-05-24T21:06:52.905 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:52.906 DEBUG:boto:Port: 8000 2023-05-24T21:06:52.906 DEBUG:boto:Params: {} 2023-05-24T21:06:52.906 DEBUG:boto:Token: None 2023-05-24T21:06:52.907 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:52 GMT /qskrps-9/myobj?versionId=null 2023-05-24T21:06:52.907 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:IZVGGpacee4MUTd/u6WWbJfIRsg= 2023-05-24T21:06:52.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:IZVGGpacee4MUTd/u6WWbJfIRsg=', 'Content-Length': '0'} 2023-05-24T21:06:52.909 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:06:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000492f8ffb7e6ea2dd-00646e7c6c-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:52.910 DEBUG:boto:path=/qskrps-9/myobj 2023-05-24T21:06:52.910 DEBUG:boto:auth_path=/qskrps-9/myobj 2023-05-24T21:06:52.911 DEBUG:boto:path=/qskrps-9/myobj?versionId=null 2023-05-24T21:06:52.911 DEBUG:boto:auth_path=/qskrps-9/myobj?versionId=null 2023-05-24T21:06:52.911 DEBUG:boto:Method: HEAD 2023-05-24T21:06:52.912 DEBUG:boto:Path: /qskrps-9/myobj?versionId=null 2023-05-24T21:06:52.912 DEBUG:boto:Data: 2023-05-24T21:06:52.913 DEBUG:boto:Headers: {} 2023-05-24T21:06:52.913 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:52.913 DEBUG:boto:Port: 8001 2023-05-24T21:06:52.914 DEBUG:boto:Params: {} 2023-05-24T21:06:52.914 DEBUG:boto:Token: None 2023-05-24T21:06:52.914 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:06:52 GMT /qskrps-9/myobj?versionId=null 2023-05-24T21:06:52.915 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:DwW7MmAZ9GqG8PhO13lPAaZ6sQQ= 2023-05-24T21:06:52.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:DwW7MmAZ9GqG8PhO13lPAaZ6sQQ=', 'Content-Length': '0'} 2023-05-24T21:06:52.917 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:06:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aa92770441402f4e-00646e7c6c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:52.917 DEBUG:boto:path=/qskrps-9/myobj 2023-05-24T21:06:52.918 DEBUG:boto:auth_path=/qskrps-9/myobj 2023-05-24T21:06:52.918 DEBUG:boto:path=/qskrps-9/myobj?versionId=null 2023-05-24T21:06:52.918 DEBUG:boto:auth_path=/qskrps-9/myobj?versionId=null 2023-05-24T21:06:52.919 DEBUG:boto:Method: HEAD 2023-05-24T21:06:52.919 DEBUG:boto:Path: /qskrps-9/myobj?versionId=null 2023-05-24T21:06:52.919 DEBUG:boto:Data: 2023-05-24T21:06:52.920 DEBUG:boto:Headers: {} 2023-05-24T21:06:52.920 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:52.920 DEBUG:boto:Port: 8000 2023-05-24T21:06:52.921 DEBUG:boto:Params: {} 2023-05-24T21:06:52.921 DEBUG:boto:Token: None 2023-05-24T21:06:52.921 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:06:52 GMT /qskrps-9/myobj?versionId=null 2023-05-24T21:06:52.922 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:DwW7MmAZ9GqG8PhO13lPAaZ6sQQ= 2023-05-24T21:06:52.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:DwW7MmAZ9GqG8PhO13lPAaZ6sQQ=', 'Content-Length': '0'} 2023-05-24T21:06:52.924 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:06:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c7229e8147213109-00646e7c6c-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:52.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-05-24T21:06:52.925 DEBUG:boto:path=/qskrps-9/myobj 2023-05-24T21:06:52.925 DEBUG:boto:auth_path=/qskrps-9/myobj 2023-05-24T21:06:52.926 DEBUG:boto:path=/qskrps-9/myobj?versionId=null 2023-05-24T21:06:52.926 DEBUG:boto:auth_path=/qskrps-9/myobj?versionId=null 2023-05-24T21:06:52.926 DEBUG:boto:Method: GET 2023-05-24T21:06:52.927 DEBUG:boto:Path: /qskrps-9/myobj?versionId=null 2023-05-24T21:06:52.927 DEBUG:boto:Data: 2023-05-24T21:06:52.927 DEBUG:boto:Headers: {} 2023-05-24T21:06:52.928 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:52.928 DEBUG:boto:Port: 8001 2023-05-24T21:06:52.928 DEBUG:boto:Params: {} 2023-05-24T21:06:52.929 DEBUG:boto:Token: None 2023-05-24T21:06:52.929 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:52 GMT /qskrps-9/myobj?versionId=null 2023-05-24T21:06:52.930 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:IZVGGpacee4MUTd/u6WWbJfIRsg= 2023-05-24T21:06:52.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:IZVGGpacee4MUTd/u6WWbJfIRsg=', 'Content-Length': '0'} 2023-05-24T21:06:52.932 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:06:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000624b3de26a6c3664-00646e7c6c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:52.972 DEBUG:boto:path=/qskrps-9/myobj 2023-05-24T21:06:52.973 DEBUG:boto:auth_path=/qskrps-9/myobj 2023-05-24T21:06:52.973 DEBUG:boto:path=/qskrps-9/myobj?versionId=null 2023-05-24T21:06:52.974 DEBUG:boto:auth_path=/qskrps-9/myobj?versionId=null 2023-05-24T21:06:52.974 DEBUG:boto:Method: GET 2023-05-24T21:06:52.974 DEBUG:boto:Path: /qskrps-9/myobj?versionId=null 2023-05-24T21:06:52.975 DEBUG:boto:Data: 2023-05-24T21:06:52.975 DEBUG:boto:Headers: {} 2023-05-24T21:06:52.975 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:52.976 DEBUG:boto:Port: 8000 2023-05-24T21:06:52.976 DEBUG:boto:Params: {} 2023-05-24T21:06:52.976 DEBUG:boto:Token: None 2023-05-24T21:06:52.977 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:52 GMT /qskrps-9/myobj?versionId=null 2023-05-24T21:06:52.977 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:IZVGGpacee4MUTd/u6WWbJfIRsg= 2023-05-24T21:06:52.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:IZVGGpacee4MUTd/u6WWbJfIRsg=', 'Content-Length': '0'} 2023-05-24T21:06:52.980 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:06:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002d51dca07d9099ad-00646e7c6c-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:52.980 DEBUG:boto:path=/qskrps-9/ 2023-05-24T21:06:52.981 DEBUG:boto:auth_path=/qskrps-9/ 2023-05-24T21:06:52.981 DEBUG:boto:path=/qskrps-9/?prefix=myobj 2023-05-24T21:06:52.981 DEBUG:boto:auth_path=/qskrps-9/?prefix=myobj 2023-05-24T21:06:52.982 DEBUG:boto:Method: GET 2023-05-24T21:06:52.982 DEBUG:boto:Path: /qskrps-9/?prefix=myobj 2023-05-24T21:06:52.983 DEBUG:boto:Data: 2023-05-24T21:06:52.983 DEBUG:boto:Headers: {} 2023-05-24T21:06:52.983 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:52.984 DEBUG:boto:Port: 8001 2023-05-24T21:06:52.984 DEBUG:boto:Params: {} 2023-05-24T21:06:52.984 DEBUG:boto:Token: None 2023-05-24T21:06:52.985 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:52 GMT /qskrps-9/ 2023-05-24T21:06:52.985 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:2SymTbrmE1FKNL4QahUHtrKDf0M= 2023-05-24T21:06:52.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:2SymTbrmE1FKNL4QahUHtrKDf0M=', 'Content-Length': '0'} 2023-05-24T21:06:52.988 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4193185b3a34cc8-00646e7c6c-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:06:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:52.988 DEBUG:boto:b'qskrps-9myobj1000falsemyobj2023-05-24T21:06:32.383Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:06:52.989 DEBUG:boto:path=/qskrps-9/ 2023-05-24T21:06:52.989 DEBUG:boto:auth_path=/qskrps-9/ 2023-05-24T21:06:52.990 DEBUG:boto:path=/qskrps-9/?prefix=myobj 2023-05-24T21:06:52.990 DEBUG:boto:auth_path=/qskrps-9/?prefix=myobj 2023-05-24T21:06:52.991 DEBUG:boto:Method: GET 2023-05-24T21:06:52.991 DEBUG:boto:Path: /qskrps-9/?prefix=myobj 2023-05-24T21:06:52.991 DEBUG:boto:Data: 2023-05-24T21:06:52.992 DEBUG:boto:Headers: {} 2023-05-24T21:06:52.992 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:52.992 DEBUG:boto:Port: 8000 2023-05-24T21:06:52.993 DEBUG:boto:Params: {} 2023-05-24T21:06:52.993 DEBUG:boto:Token: None 2023-05-24T21:06:52.994 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:52 GMT /qskrps-9/ 2023-05-24T21:06:52.994 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:2SymTbrmE1FKNL4QahUHtrKDf0M= 2023-05-24T21:06:52.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:2SymTbrmE1FKNL4QahUHtrKDf0M=', 'Content-Length': '0'} 2023-05-24T21:06:52.997 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002066e616857caa89-00646e7c6c-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:06:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:52.998 DEBUG:boto:b'qskrps-9myobj1000falsemyobj2023-05-24T21:06:32.383Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:06:52.998 DEBUG:boto:path=/qskrps-9/myobj 2023-05-24T21:06:52.999 DEBUG:boto:auth_path=/qskrps-9/myobj 2023-05-24T21:06:52.999 DEBUG:boto:Method: HEAD 2023-05-24T21:06:52.999 DEBUG:boto:Path: /qskrps-9/myobj 2023-05-24T21:06:53.000 DEBUG:boto:Data: 2023-05-24T21:06:53.000 DEBUG:boto:Headers: {} 2023-05-24T21:06:53.000 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:53.001 DEBUG:boto:Port: 8001 2023-05-24T21:06:53.001 DEBUG:boto:Params: {} 2023-05-24T21:06:53.001 DEBUG:boto:Token: None 2023-05-24T21:06:53.002 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:06:53 GMT /qskrps-9/myobj 2023-05-24T21:06:53.002 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:fjctJMRuw2g/HiQVL+YF7x+8o9U= 2023-05-24T21:06:53.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:fjctJMRuw2g/HiQVL+YF7x+8o9U=', 'Content-Length': '0'} 2023-05-24T21:06:53.005 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:06:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf7728b5724a0a42-00646e7c6c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:53.005 DEBUG:boto:path=/qskrps-9/myobj 2023-05-24T21:06:53.006 DEBUG:boto:auth_path=/qskrps-9/myobj 2023-05-24T21:06:53.006 DEBUG:boto:Method: HEAD 2023-05-24T21:06:53.006 DEBUG:boto:Path: /qskrps-9/myobj 2023-05-24T21:06:53.007 DEBUG:boto:Data: 2023-05-24T21:06:53.007 DEBUG:boto:Headers: {} 2023-05-24T21:06:53.007 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:53.008 DEBUG:boto:Port: 8000 2023-05-24T21:06:53.008 DEBUG:boto:Params: {} 2023-05-24T21:06:53.009 DEBUG:boto:Token: None 2023-05-24T21:06:53.009 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:06:53 GMT /qskrps-9/myobj 2023-05-24T21:06:53.009 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:fjctJMRuw2g/HiQVL+YF7x+8o9U= 2023-05-24T21:06:53.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:fjctJMRuw2g/HiQVL+YF7x+8o9U=', 'Content-Length': '0'} 2023-05-24T21:06:53.012 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:06:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e78c7e3bd34ebd1-00646e7c6d-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:53.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-05-24T21:06:53.013 DEBUG:boto:path=/qskrps-9/myobj 2023-05-24T21:06:53.013 DEBUG:boto:auth_path=/qskrps-9/myobj 2023-05-24T21:06:53.013 DEBUG:boto:Method: GET 2023-05-24T21:06:53.014 DEBUG:boto:Path: /qskrps-9/myobj 2023-05-24T21:06:53.014 DEBUG:boto:Data: 2023-05-24T21:06:53.014 DEBUG:boto:Headers: {} 2023-05-24T21:06:53.015 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:53.015 DEBUG:boto:Port: 8001 2023-05-24T21:06:53.016 DEBUG:boto:Params: {} 2023-05-24T21:06:53.016 DEBUG:boto:Token: None 2023-05-24T21:06:53.016 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:53 GMT /qskrps-9/myobj 2023-05-24T21:06:53.017 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Lw7wbdPR9QupqWeWclpUGAoG9fc= 2023-05-24T21:06:53.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Lw7wbdPR9QupqWeWclpUGAoG9fc=', 'Content-Length': '0'} 2023-05-24T21:06:53.019 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:06:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000006d0aebec1beca8-00646e7c6d-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:53.019 DEBUG:boto:path=/qskrps-9/myobj 2023-05-24T21:06:53.020 DEBUG:boto:auth_path=/qskrps-9/myobj 2023-05-24T21:06:53.020 DEBUG:boto:Method: GET 2023-05-24T21:06:53.021 DEBUG:boto:Path: /qskrps-9/myobj 2023-05-24T21:06:53.021 DEBUG:boto:Data: 2023-05-24T21:06:53.021 DEBUG:boto:Headers: {} 2023-05-24T21:06:53.022 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:53.022 DEBUG:boto:Port: 8000 2023-05-24T21:06:53.022 DEBUG:boto:Params: {} 2023-05-24T21:06:53.023 DEBUG:boto:Token: None 2023-05-24T21:06:53.023 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:53 GMT /qskrps-9/myobj 2023-05-24T21:06:53.024 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Lw7wbdPR9QupqWeWclpUGAoG9fc= 2023-05-24T21:06:53.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Lw7wbdPR9QupqWeWclpUGAoG9fc=', 'Content-Length': '0'} 2023-05-24T21:06:53.026 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:06:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000d1186b33efed7cb-00646e7c6d-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:53.027 DEBUG:boto:path=/qskrps-9/ 2023-05-24T21:06:53.027 DEBUG:boto:auth_path=/qskrps-9/ 2023-05-24T21:06:53.027 DEBUG:boto:path=/qskrps-9/?prefix=myobj 2023-05-24T21:06:53.028 DEBUG:boto:auth_path=/qskrps-9/?prefix=myobj 2023-05-24T21:06:53.028 DEBUG:boto:Method: GET 2023-05-24T21:06:53.029 DEBUG:boto:Path: /qskrps-9/?prefix=myobj 2023-05-24T21:06:53.029 DEBUG:boto:Data: 2023-05-24T21:06:53.029 DEBUG:boto:Headers: {} 2023-05-24T21:06:53.030 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:53.030 DEBUG:boto:Port: 8001 2023-05-24T21:06:53.030 DEBUG:boto:Params: {} 2023-05-24T21:06:53.031 DEBUG:boto:Token: None 2023-05-24T21:06:53.031 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:53 GMT /qskrps-9/ 2023-05-24T21:06:53.032 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:GxHk48b9YmdCQmjc3UsntUwYC8g= 2023-05-24T21:06:53.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:GxHk48b9YmdCQmjc3UsntUwYC8g=', 'Content-Length': '0'} 2023-05-24T21:06:53.034 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000259cb7cd915bf3c7-00646e7c6d-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:06:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:53.035 DEBUG:boto:b'qskrps-9myobj1000falsemyobj2023-05-24T21:06:32.383Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:06:53.036 DEBUG:boto:path=/qskrps-9/ 2023-05-24T21:06:53.036 DEBUG:boto:auth_path=/qskrps-9/ 2023-05-24T21:06:53.036 DEBUG:boto:path=/qskrps-9/?prefix=myobj 2023-05-24T21:06:53.037 DEBUG:boto:auth_path=/qskrps-9/?prefix=myobj 2023-05-24T21:06:53.037 DEBUG:boto:Method: GET 2023-05-24T21:06:53.037 DEBUG:boto:Path: /qskrps-9/?prefix=myobj 2023-05-24T21:06:53.038 DEBUG:boto:Data: 2023-05-24T21:06:53.038 DEBUG:boto:Headers: {} 2023-05-24T21:06:53.039 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:53.039 DEBUG:boto:Port: 8000 2023-05-24T21:06:53.039 DEBUG:boto:Params: {} 2023-05-24T21:06:53.040 DEBUG:boto:Token: None 2023-05-24T21:06:53.040 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:53 GMT /qskrps-9/ 2023-05-24T21:06:53.041 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:GxHk48b9YmdCQmjc3UsntUwYC8g= 2023-05-24T21:06:53.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:GxHk48b9YmdCQmjc3UsntUwYC8g=', 'Content-Length': '0'} 2023-05-24T21:06:53.043 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002416f9043011af78-00646e7c6d-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:06:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:53.044 DEBUG:boto:b'qskrps-9myobj1000falsemyobj2023-05-24T21:06:32.383Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:06:53.045 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qskrps-9 zones={a2, a1} 2023-05-24T21:06:53.045 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:06:53.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:06:53.046 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-10 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:06:53.109 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:06:53.110 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:06:53.147 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:06:53.141+0000 7f876313e780 1 bucket sync caught up with source: 2023-05-24T21:06:53.148 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000001.20.6, , , , , , , , , ] 2023-05-24T21:06:53.148 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000001.20.6, , , , , , , , , ] 2023-05-24T21:06:53.148 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:06:53.141+0000 7f876313e780 0 bucket checkpoint complete 2023-05-24T21:06:53.160 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qskrps-10 zones={a1, a2} 2023-05-24T21:06:53.161 DEBUG:boto:path=/qskrps-10/ 2023-05-24T21:06:53.161 DEBUG:boto:auth_path=/qskrps-10/ 2023-05-24T21:06:53.162 DEBUG:boto:Method: HEAD 2023-05-24T21:06:53.162 DEBUG:boto:Path: /qskrps-10/ 2023-05-24T21:06:53.163 DEBUG:boto:Data: 2023-05-24T21:06:53.163 DEBUG:boto:Headers: {} 2023-05-24T21:06:53.163 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:53.164 DEBUG:boto:Port: 8000 2023-05-24T21:06:53.164 DEBUG:boto:Params: {} 2023-05-24T21:06:53.165 DEBUG:boto:Token: None 2023-05-24T21:06:53.165 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:06:53 GMT /qskrps-10/ 2023-05-24T21:06:53.166 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:gdTM5egmciQfgzh1OuAVkAB/rbw= 2023-05-24T21:06:53.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:gdTM5egmciQfgzh1OuAVkAB/rbw=', 'Content-Length': '0'} 2023-05-24T21:06:53.169 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b13cf91bef1c1915-00646e7c6d-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:06:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:53.169 DEBUG:boto:path=/qskrps-10/ 2023-05-24T21:06:53.170 DEBUG:boto:auth_path=/qskrps-10/ 2023-05-24T21:06:53.170 DEBUG:boto:Method: HEAD 2023-05-24T21:06:53.171 DEBUG:boto:Path: /qskrps-10/ 2023-05-24T21:06:53.171 DEBUG:boto:Data: 2023-05-24T21:06:53.171 DEBUG:boto:Headers: {} 2023-05-24T21:06:53.172 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:53.172 DEBUG:boto:Port: 8001 2023-05-24T21:06:53.172 DEBUG:boto:Params: {} 2023-05-24T21:06:53.173 DEBUG:boto:Token: None 2023-05-24T21:06:53.173 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:06:53 GMT /qskrps-10/ 2023-05-24T21:06:53.174 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:gdTM5egmciQfgzh1OuAVkAB/rbw= 2023-05-24T21:06:53.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:gdTM5egmciQfgzh1OuAVkAB/rbw=', 'Content-Length': '0'} 2023-05-24T21:06:53.176 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d5091c0e3e00f2b3-00646e7c6d-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:06:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:53.177 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-24T21:06:53.177 DEBUG:boto:path=/qskrps-10/ 2023-05-24T21:06:53.177 DEBUG:boto:auth_path=/qskrps-10/ 2023-05-24T21:06:53.178 DEBUG:boto:path=/qskrps-10/?versions&max-keys=999 2023-05-24T21:06:53.178 DEBUG:boto:auth_path=/qskrps-10/?versions&max-keys=999 2023-05-24T21:06:53.178 DEBUG:boto:Method: GET 2023-05-24T21:06:53.179 DEBUG:boto:Path: /qskrps-10/?versions&max-keys=999 2023-05-24T21:06:53.179 DEBUG:boto:Data: 2023-05-24T21:06:53.179 DEBUG:boto:Headers: {} 2023-05-24T21:06:53.180 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:53.180 DEBUG:boto:Port: 8000 2023-05-24T21:06:53.180 DEBUG:boto:Params: {} 2023-05-24T21:06:53.181 DEBUG:boto:Token: None 2023-05-24T21:06:53.181 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:53 GMT /qskrps-10/?versions 2023-05-24T21:06:53.182 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:6he7QJxcQlyuKKrPdG0EygdxsRU= 2023-05-24T21:06:53.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:6he7QJxcQlyuKKrPdG0EygdxsRU=', 'Content-Length': '0'} 2023-05-24T21:06:53.184 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072b9acd6ad1f776c-00646e7c6d-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:06:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:53.228 DEBUG:boto:b'qskrps-10999falsemyobjnulltrue2023-05-24T21:06:32.438Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:06:53.229 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-05-24T21:06:53.229 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-24T21:06:53.230 DEBUG:boto:path=/qskrps-10/ 2023-05-24T21:06:53.230 DEBUG:boto:auth_path=/qskrps-10/ 2023-05-24T21:06:53.230 DEBUG:boto:path=/qskrps-10/?versions&max-keys=999 2023-05-24T21:06:53.231 DEBUG:boto:auth_path=/qskrps-10/?versions&max-keys=999 2023-05-24T21:06:53.231 DEBUG:boto:Method: GET 2023-05-24T21:06:53.232 DEBUG:boto:Path: /qskrps-10/?versions&max-keys=999 2023-05-24T21:06:53.232 DEBUG:boto:Data: 2023-05-24T21:06:53.232 DEBUG:boto:Headers: {} 2023-05-24T21:06:53.233 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:53.233 DEBUG:boto:Port: 8001 2023-05-24T21:06:53.233 DEBUG:boto:Params: {} 2023-05-24T21:06:53.234 DEBUG:boto:Token: None 2023-05-24T21:06:53.234 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:53 GMT /qskrps-10/?versions 2023-05-24T21:06:53.234 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:6he7QJxcQlyuKKrPdG0EygdxsRU= 2023-05-24T21:06:53.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:6he7QJxcQlyuKKrPdG0EygdxsRU=', 'Content-Length': '0'} 2023-05-24T21:06:53.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcca6c8c6812b913-00646e7c6d-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:06:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:53.238 DEBUG:boto:b'qskrps-10999falsemyobjnulltrue2023-05-24T21:06:32.438Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:06:53.238 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-05-24T21:06:53.239 DEBUG:boto:path=/qskrps-10/ 2023-05-24T21:06:53.239 DEBUG:boto:auth_path=/qskrps-10/ 2023-05-24T21:06:53.239 DEBUG:boto:path=/qskrps-10/?versions&max-keys=999 2023-05-24T21:06:53.240 DEBUG:boto:auth_path=/qskrps-10/?versions&max-keys=999 2023-05-24T21:06:53.240 DEBUG:boto:Method: GET 2023-05-24T21:06:53.240 DEBUG:boto:Path: /qskrps-10/?versions&max-keys=999 2023-05-24T21:06:53.241 DEBUG:boto:Data: 2023-05-24T21:06:53.241 DEBUG:boto:Headers: {} 2023-05-24T21:06:53.241 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:53.242 DEBUG:boto:Port: 8000 2023-05-24T21:06:53.242 DEBUG:boto:Params: {} 2023-05-24T21:06:53.242 DEBUG:boto:Token: None 2023-05-24T21:06:53.243 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:53 GMT /qskrps-10/?versions 2023-05-24T21:06:53.243 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:6he7QJxcQlyuKKrPdG0EygdxsRU= 2023-05-24T21:06:53.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:6he7QJxcQlyuKKrPdG0EygdxsRU=', 'Content-Length': '0'} 2023-05-24T21:06:53.246 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005140d05ca75cea2d-00646e7c6d-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:06:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:53.246 DEBUG:boto:b'qskrps-10999falsemyobjnulltrue2023-05-24T21:06:32.438Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:06:53.247 DEBUG:boto:path=/qskrps-10/ 2023-05-24T21:06:53.247 DEBUG:boto:auth_path=/qskrps-10/ 2023-05-24T21:06:53.248 DEBUG:boto:path=/qskrps-10/?versions&max-keys=999 2023-05-24T21:06:53.248 DEBUG:boto:auth_path=/qskrps-10/?versions&max-keys=999 2023-05-24T21:06:53.249 DEBUG:boto:Method: GET 2023-05-24T21:06:53.249 DEBUG:boto:Path: /qskrps-10/?versions&max-keys=999 2023-05-24T21:06:53.249 DEBUG:boto:Data: 2023-05-24T21:06:53.250 DEBUG:boto:Headers: {} 2023-05-24T21:06:53.250 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:53.250 DEBUG:boto:Port: 8001 2023-05-24T21:06:53.251 DEBUG:boto:Params: {} 2023-05-24T21:06:53.251 DEBUG:boto:Token: None 2023-05-24T21:06:53.251 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:53 GMT /qskrps-10/?versions 2023-05-24T21:06:53.252 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:6he7QJxcQlyuKKrPdG0EygdxsRU= 2023-05-24T21:06:53.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:6he7QJxcQlyuKKrPdG0EygdxsRU=', 'Content-Length': '0'} 2023-05-24T21:06:53.255 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020a6e8ee5971c395-00646e7c6d-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:06:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:53.296 DEBUG:boto:b'qskrps-10999falsemyobjnulltrue2023-05-24T21:06:32.438Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:06:53.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-05-24T21:06:53.298 DEBUG:boto:path=/qskrps-10/myobj 2023-05-24T21:06:53.299 DEBUG:boto:auth_path=/qskrps-10/myobj 2023-05-24T21:06:53.300 DEBUG:boto:path=/qskrps-10/myobj?versionId=null 2023-05-24T21:06:53.300 DEBUG:boto:auth_path=/qskrps-10/myobj?versionId=null 2023-05-24T21:06:53.301 DEBUG:boto:Method: GET 2023-05-24T21:06:53.302 DEBUG:boto:Path: /qskrps-10/myobj?versionId=null 2023-05-24T21:06:53.302 DEBUG:boto:Data: 2023-05-24T21:06:53.303 DEBUG:boto:Headers: {} 2023-05-24T21:06:53.304 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:53.304 DEBUG:boto:Port: 8000 2023-05-24T21:06:53.305 DEBUG:boto:Params: {} 2023-05-24T21:06:53.306 DEBUG:boto:Token: None 2023-05-24T21:06:53.307 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:53 GMT /qskrps-10/myobj?versionId=null 2023-05-24T21:06:53.307 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:szabMc43SWEq1O8HkWlbbFmKXjc= 2023-05-24T21:06:53.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:szabMc43SWEq1O8HkWlbbFmKXjc=', 'Content-Length': '0'} 2023-05-24T21:06:53.310 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:06:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008a1961ab68535745-00646e7c6d-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:53.311 DEBUG:boto:path=/qskrps-10/myobj 2023-05-24T21:06:53.312 DEBUG:boto:auth_path=/qskrps-10/myobj 2023-05-24T21:06:53.313 DEBUG:boto:path=/qskrps-10/myobj?versionId=null 2023-05-24T21:06:53.313 DEBUG:boto:auth_path=/qskrps-10/myobj?versionId=null 2023-05-24T21:06:53.314 DEBUG:boto:Method: GET 2023-05-24T21:06:53.315 DEBUG:boto:Path: /qskrps-10/myobj?versionId=null 2023-05-24T21:06:53.315 DEBUG:boto:Data: 2023-05-24T21:06:53.316 DEBUG:boto:Headers: {} 2023-05-24T21:06:53.317 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:53.318 DEBUG:boto:Port: 8001 2023-05-24T21:06:53.318 DEBUG:boto:Params: {} 2023-05-24T21:06:53.319 DEBUG:boto:Token: None 2023-05-24T21:06:53.320 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:53 GMT /qskrps-10/myobj?versionId=null 2023-05-24T21:06:53.321 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:szabMc43SWEq1O8HkWlbbFmKXjc= 2023-05-24T21:06:53.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:szabMc43SWEq1O8HkWlbbFmKXjc=', 'Content-Length': '0'} 2023-05-24T21:06:53.323 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:06:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000785c30b6f7957d99-00646e7c6d-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:53.324 DEBUG:boto:path=/qskrps-10/myobj 2023-05-24T21:06:53.325 DEBUG:boto:auth_path=/qskrps-10/myobj 2023-05-24T21:06:53.326 DEBUG:boto:path=/qskrps-10/myobj?versionId=null 2023-05-24T21:06:53.326 DEBUG:boto:auth_path=/qskrps-10/myobj?versionId=null 2023-05-24T21:06:53.327 DEBUG:boto:Method: HEAD 2023-05-24T21:06:53.328 DEBUG:boto:Path: /qskrps-10/myobj?versionId=null 2023-05-24T21:06:53.328 DEBUG:boto:Data: 2023-05-24T21:06:53.329 DEBUG:boto:Headers: {} 2023-05-24T21:06:53.329 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:53.330 DEBUG:boto:Port: 8000 2023-05-24T21:06:53.331 DEBUG:boto:Params: {} 2023-05-24T21:06:53.331 DEBUG:boto:Token: None 2023-05-24T21:06:53.332 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:06:53 GMT /qskrps-10/myobj?versionId=null 2023-05-24T21:06:53.333 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:X1QfVYSQ8nzckpXJXUT4cZCWRIc= 2023-05-24T21:06:53.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:X1QfVYSQ8nzckpXJXUT4cZCWRIc=', 'Content-Length': '0'} 2023-05-24T21:06:53.336 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:06:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000932006056a58f08b-00646e7c6d-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:53.337 DEBUG:boto:path=/qskrps-10/myobj 2023-05-24T21:06:53.337 DEBUG:boto:auth_path=/qskrps-10/myobj 2023-05-24T21:06:53.338 DEBUG:boto:path=/qskrps-10/myobj?versionId=null 2023-05-24T21:06:53.338 DEBUG:boto:auth_path=/qskrps-10/myobj?versionId=null 2023-05-24T21:06:53.339 DEBUG:boto:Method: HEAD 2023-05-24T21:06:53.340 DEBUG:boto:Path: /qskrps-10/myobj?versionId=null 2023-05-24T21:06:53.341 DEBUG:boto:Data: 2023-05-24T21:06:53.341 DEBUG:boto:Headers: {} 2023-05-24T21:06:53.342 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:53.342 DEBUG:boto:Port: 8001 2023-05-24T21:06:53.343 DEBUG:boto:Params: {} 2023-05-24T21:06:53.344 DEBUG:boto:Token: None 2023-05-24T21:06:53.344 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:06:53 GMT /qskrps-10/myobj?versionId=null 2023-05-24T21:06:53.345 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:X1QfVYSQ8nzckpXJXUT4cZCWRIc= 2023-05-24T21:06:53.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:X1QfVYSQ8nzckpXJXUT4cZCWRIc=', 'Content-Length': '0'} 2023-05-24T21:06:53.348 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:06:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000867985800e736c7a-00646e7c6d-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:53.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-05-24T21:06:53.350 DEBUG:boto:path=/qskrps-10/myobj 2023-05-24T21:06:53.350 DEBUG:boto:auth_path=/qskrps-10/myobj 2023-05-24T21:06:53.351 DEBUG:boto:path=/qskrps-10/myobj?versionId=null 2023-05-24T21:06:53.352 DEBUG:boto:auth_path=/qskrps-10/myobj?versionId=null 2023-05-24T21:06:53.352 DEBUG:boto:Method: GET 2023-05-24T21:06:53.353 DEBUG:boto:Path: /qskrps-10/myobj?versionId=null 2023-05-24T21:06:53.354 DEBUG:boto:Data: 2023-05-24T21:06:53.354 DEBUG:boto:Headers: {} 2023-05-24T21:06:53.355 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:53.356 DEBUG:boto:Port: 8000 2023-05-24T21:06:53.356 DEBUG:boto:Params: {} 2023-05-24T21:06:53.357 DEBUG:boto:Token: None 2023-05-24T21:06:53.358 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:53 GMT /qskrps-10/myobj?versionId=null 2023-05-24T21:06:53.359 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:szabMc43SWEq1O8HkWlbbFmKXjc= 2023-05-24T21:06:53.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:szabMc43SWEq1O8HkWlbbFmKXjc=', 'Content-Length': '0'} 2023-05-24T21:06:53.362 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:06:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000a34619dc1181c24-00646e7c6d-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:53.363 DEBUG:boto:path=/qskrps-10/myobj 2023-05-24T21:06:53.363 DEBUG:boto:auth_path=/qskrps-10/myobj 2023-05-24T21:06:53.364 DEBUG:boto:path=/qskrps-10/myobj?versionId=null 2023-05-24T21:06:53.365 DEBUG:boto:auth_path=/qskrps-10/myobj?versionId=null 2023-05-24T21:06:53.365 DEBUG:boto:Method: GET 2023-05-24T21:06:53.366 DEBUG:boto:Path: /qskrps-10/myobj?versionId=null 2023-05-24T21:06:53.367 DEBUG:boto:Data: 2023-05-24T21:06:53.367 DEBUG:boto:Headers: {} 2023-05-24T21:06:53.368 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:53.369 DEBUG:boto:Port: 8001 2023-05-24T21:06:53.369 DEBUG:boto:Params: {} 2023-05-24T21:06:53.370 DEBUG:boto:Token: None 2023-05-24T21:06:53.370 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:53 GMT /qskrps-10/myobj?versionId=null 2023-05-24T21:06:53.370 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:szabMc43SWEq1O8HkWlbbFmKXjc= 2023-05-24T21:06:53.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:szabMc43SWEq1O8HkWlbbFmKXjc=', 'Content-Length': '0'} 2023-05-24T21:06:53.373 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:06:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000932ce28b02e8ffaf-00646e7c6d-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:53.373 DEBUG:boto:path=/qskrps-10/ 2023-05-24T21:06:53.374 DEBUG:boto:auth_path=/qskrps-10/ 2023-05-24T21:06:53.374 DEBUG:boto:path=/qskrps-10/?prefix=myobj 2023-05-24T21:06:53.374 DEBUG:boto:auth_path=/qskrps-10/?prefix=myobj 2023-05-24T21:06:53.375 DEBUG:boto:Method: GET 2023-05-24T21:06:53.375 DEBUG:boto:Path: /qskrps-10/?prefix=myobj 2023-05-24T21:06:53.375 DEBUG:boto:Data: 2023-05-24T21:06:53.376 DEBUG:boto:Headers: {} 2023-05-24T21:06:53.376 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:53.377 DEBUG:boto:Port: 8000 2023-05-24T21:06:53.377 DEBUG:boto:Params: {} 2023-05-24T21:06:53.377 DEBUG:boto:Token: None 2023-05-24T21:06:53.378 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:53 GMT /qskrps-10/ 2023-05-24T21:06:53.378 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:JW4cBOdk7hGsapJYXlGnmm1YSWo= 2023-05-24T21:06:53.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:JW4cBOdk7hGsapJYXlGnmm1YSWo=', 'Content-Length': '0'} 2023-05-24T21:06:53.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e08aadde5f1294a0-00646e7c6d-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:06:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:53.381 DEBUG:boto:b'qskrps-10myobj1000falsemyobj2023-05-24T21:06:32.438Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:06:53.382 DEBUG:boto:path=/qskrps-10/ 2023-05-24T21:06:53.382 DEBUG:boto:auth_path=/qskrps-10/ 2023-05-24T21:06:53.382 DEBUG:boto:path=/qskrps-10/?prefix=myobj 2023-05-24T21:06:53.383 DEBUG:boto:auth_path=/qskrps-10/?prefix=myobj 2023-05-24T21:06:53.383 DEBUG:boto:Method: GET 2023-05-24T21:06:53.383 DEBUG:boto:Path: /qskrps-10/?prefix=myobj 2023-05-24T21:06:53.384 DEBUG:boto:Data: 2023-05-24T21:06:53.384 DEBUG:boto:Headers: {} 2023-05-24T21:06:53.385 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:53.385 DEBUG:boto:Port: 8001 2023-05-24T21:06:53.385 DEBUG:boto:Params: {} 2023-05-24T21:06:53.386 DEBUG:boto:Token: None 2023-05-24T21:06:53.386 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:53 GMT /qskrps-10/ 2023-05-24T21:06:53.387 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:JW4cBOdk7hGsapJYXlGnmm1YSWo= 2023-05-24T21:06:53.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:JW4cBOdk7hGsapJYXlGnmm1YSWo=', 'Content-Length': '0'} 2023-05-24T21:06:53.389 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bb28a7b1e79fabb-00646e7c6d-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:06:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:53.390 DEBUG:boto:b'qskrps-10myobj1000falsemyobj2023-05-24T21:06:32.438Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:06:53.391 DEBUG:boto:path=/qskrps-10/myobj 2023-05-24T21:06:53.391 DEBUG:boto:auth_path=/qskrps-10/myobj 2023-05-24T21:06:53.391 DEBUG:boto:Method: HEAD 2023-05-24T21:06:53.392 DEBUG:boto:Path: /qskrps-10/myobj 2023-05-24T21:06:53.392 DEBUG:boto:Data: 2023-05-24T21:06:53.392 DEBUG:boto:Headers: {} 2023-05-24T21:06:53.393 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:53.393 DEBUG:boto:Port: 8000 2023-05-24T21:06:53.393 DEBUG:boto:Params: {} 2023-05-24T21:06:53.394 DEBUG:boto:Token: None 2023-05-24T21:06:53.394 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:06:53 GMT /qskrps-10/myobj 2023-05-24T21:06:53.395 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:zEBjBrAAyGokJnm7DTVPi6La+7g= 2023-05-24T21:06:53.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:zEBjBrAAyGokJnm7DTVPi6La+7g=', 'Content-Length': '0'} 2023-05-24T21:06:53.397 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:06:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d18c05ebb984a000-00646e7c6d-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:53.397 DEBUG:boto:path=/qskrps-10/myobj 2023-05-24T21:06:53.397 DEBUG:boto:auth_path=/qskrps-10/myobj 2023-05-24T21:06:53.398 DEBUG:boto:Method: HEAD 2023-05-24T21:06:53.398 DEBUG:boto:Path: /qskrps-10/myobj 2023-05-24T21:06:53.398 DEBUG:boto:Data: 2023-05-24T21:06:53.399 DEBUG:boto:Headers: {} 2023-05-24T21:06:53.399 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:53.399 DEBUG:boto:Port: 8001 2023-05-24T21:06:53.400 DEBUG:boto:Params: {} 2023-05-24T21:06:53.400 DEBUG:boto:Token: None 2023-05-24T21:06:53.401 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:06:53 GMT /qskrps-10/myobj 2023-05-24T21:06:53.401 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:zEBjBrAAyGokJnm7DTVPi6La+7g= 2023-05-24T21:06:53.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:zEBjBrAAyGokJnm7DTVPi6La+7g=', 'Content-Length': '0'} 2023-05-24T21:06:53.403 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:06:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000846a7b78c3f8adb2-00646e7c6d-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:53.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-05-24T21:06:53.404 DEBUG:boto:path=/qskrps-10/myobj 2023-05-24T21:06:53.404 DEBUG:boto:auth_path=/qskrps-10/myobj 2023-05-24T21:06:53.405 DEBUG:boto:Method: GET 2023-05-24T21:06:53.405 DEBUG:boto:Path: /qskrps-10/myobj 2023-05-24T21:06:53.405 DEBUG:boto:Data: 2023-05-24T21:06:53.406 DEBUG:boto:Headers: {} 2023-05-24T21:06:53.406 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:53.406 DEBUG:boto:Port: 8000 2023-05-24T21:06:53.407 DEBUG:boto:Params: {} 2023-05-24T21:06:53.407 DEBUG:boto:Token: None 2023-05-24T21:06:53.408 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:53 GMT /qskrps-10/myobj 2023-05-24T21:06:53.408 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Zh/XZ2P4YwKpCkDOSACVLoiv45g= 2023-05-24T21:06:53.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Zh/XZ2P4YwKpCkDOSACVLoiv45g=', 'Content-Length': '0'} 2023-05-24T21:06:53.410 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:06:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c11fc66795b9a700-00646e7c6d-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:53.452 DEBUG:boto:path=/qskrps-10/myobj 2023-05-24T21:06:53.453 DEBUG:boto:auth_path=/qskrps-10/myobj 2023-05-24T21:06:53.454 DEBUG:boto:Method: GET 2023-05-24T21:06:53.455 DEBUG:boto:Path: /qskrps-10/myobj 2023-05-24T21:06:53.456 DEBUG:boto:Data: 2023-05-24T21:06:53.456 DEBUG:boto:Headers: {} 2023-05-24T21:06:53.456 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:53.457 DEBUG:boto:Port: 8001 2023-05-24T21:06:53.457 DEBUG:boto:Params: {} 2023-05-24T21:06:53.458 DEBUG:boto:Token: None 2023-05-24T21:06:53.458 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:53 GMT /qskrps-10/myobj 2023-05-24T21:06:53.459 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Zh/XZ2P4YwKpCkDOSACVLoiv45g= 2023-05-24T21:06:53.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Zh/XZ2P4YwKpCkDOSACVLoiv45g=', 'Content-Length': '0'} 2023-05-24T21:06:53.461 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:06:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ecb91a25a5e6844a-00646e7c6d-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:53.462 DEBUG:boto:path=/qskrps-10/ 2023-05-24T21:06:53.462 DEBUG:boto:auth_path=/qskrps-10/ 2023-05-24T21:06:53.463 DEBUG:boto:path=/qskrps-10/?prefix=myobj 2023-05-24T21:06:53.463 DEBUG:boto:auth_path=/qskrps-10/?prefix=myobj 2023-05-24T21:06:53.464 DEBUG:boto:Method: GET 2023-05-24T21:06:53.464 DEBUG:boto:Path: /qskrps-10/?prefix=myobj 2023-05-24T21:06:53.464 DEBUG:boto:Data: 2023-05-24T21:06:53.465 DEBUG:boto:Headers: {} 2023-05-24T21:06:53.465 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:53.466 DEBUG:boto:Port: 8000 2023-05-24T21:06:53.466 DEBUG:boto:Params: {} 2023-05-24T21:06:53.466 DEBUG:boto:Token: None 2023-05-24T21:06:53.467 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:53 GMT /qskrps-10/ 2023-05-24T21:06:53.467 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:JW4cBOdk7hGsapJYXlGnmm1YSWo= 2023-05-24T21:06:53.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:JW4cBOdk7hGsapJYXlGnmm1YSWo=', 'Content-Length': '0'} 2023-05-24T21:06:53.470 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2ebeae111666077-00646e7c6d-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:06:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:53.470 DEBUG:boto:b'qskrps-10myobj1000falsemyobj2023-05-24T21:06:32.438Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:06:53.471 DEBUG:boto:path=/qskrps-10/ 2023-05-24T21:06:53.472 DEBUG:boto:auth_path=/qskrps-10/ 2023-05-24T21:06:53.472 DEBUG:boto:path=/qskrps-10/?prefix=myobj 2023-05-24T21:06:53.472 DEBUG:boto:auth_path=/qskrps-10/?prefix=myobj 2023-05-24T21:06:53.473 DEBUG:boto:Method: GET 2023-05-24T21:06:53.473 DEBUG:boto:Path: /qskrps-10/?prefix=myobj 2023-05-24T21:06:53.473 DEBUG:boto:Data: 2023-05-24T21:06:53.474 DEBUG:boto:Headers: {} 2023-05-24T21:06:53.474 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:06:53.475 DEBUG:boto:Port: 8001 2023-05-24T21:06:53.475 DEBUG:boto:Params: {} 2023-05-24T21:06:53.475 DEBUG:boto:Token: None 2023-05-24T21:06:53.476 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:06:53 GMT /qskrps-10/ 2023-05-24T21:06:53.476 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:JW4cBOdk7hGsapJYXlGnmm1YSWo= 2023-05-24T21:06:53.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:JW4cBOdk7hGsapJYXlGnmm1YSWo=', 'Content-Length': '0'} 2023-05-24T21:06:53.479 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027a9b2de4d95fa26-00646e7c6d-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:06:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:53.479 DEBUG:boto:b'qskrps-10myobj1000falsemyobj2023-05-24T21:06:32.438Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:06:53.480 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qskrps-10 zones={a1, a2} 2023-05-24T21:06:53.480 DEBUG:boto:path=/qskrps-9/ 2023-05-24T21:06:53.481 DEBUG:boto:auth_path=/qskrps-9/ 2023-05-24T21:06:53.481 DEBUG:boto:Method: HEAD 2023-05-24T21:06:53.482 DEBUG:boto:Path: /qskrps-9/ 2023-05-24T21:06:53.482 DEBUG:boto:Data: 2023-05-24T21:06:53.482 DEBUG:boto:Headers: {} 2023-05-24T21:06:53.482 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:53.483 DEBUG:boto:Port: 8000 2023-05-24T21:06:53.483 DEBUG:boto:Params: {} 2023-05-24T21:06:53.483 DEBUG:boto:Token: None 2023-05-24T21:06:53.484 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:06:53 GMT /qskrps-9/ 2023-05-24T21:06:53.484 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Iqh60FkMyRBFQQvyivukqylcVvo= 2023-05-24T21:06:53.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Iqh60FkMyRBFQQvyivukqylcVvo=', 'Content-Length': '0'} 2023-05-24T21:06:53.487 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000400adebd9320a390-00646e7c6d-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:06:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:53.487 DEBUG:boto:path=/qskrps-9/myobj 2023-05-24T21:06:53.488 DEBUG:boto:auth_path=/qskrps-9/myobj 2023-05-24T21:06:53.488 DEBUG:boto:Method: HEAD 2023-05-24T21:06:53.488 DEBUG:boto:Path: /qskrps-9/myobj 2023-05-24T21:06:53.489 DEBUG:boto:Data: 2023-05-24T21:06:53.490 DEBUG:boto:Headers: {} 2023-05-24T21:06:53.490 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:53.491 DEBUG:boto:Port: 8000 2023-05-24T21:06:53.492 DEBUG:boto:Params: {} 2023-05-24T21:06:53.493 DEBUG:boto:Token: None 2023-05-24T21:06:53.494 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:06:53 GMT /qskrps-9/myobj 2023-05-24T21:06:53.494 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:fjctJMRuw2g/HiQVL+YF7x+8o9U= 2023-05-24T21:06:53.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:fjctJMRuw2g/HiQVL+YF7x+8o9U=', 'Content-Length': '0'} 2023-05-24T21:06:53.496 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:06:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001f0f0d532e2bf625-00646e7c6d-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:06:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:53.497 DEBUG:boto:path=/qskrps-9/myobj 2023-05-24T21:06:53.497 DEBUG:boto:auth_path=/qskrps-9/myobj 2023-05-24T21:06:53.498 DEBUG:boto:Method: DELETE 2023-05-24T21:06:53.498 DEBUG:boto:Path: /qskrps-9/myobj 2023-05-24T21:06:53.498 DEBUG:boto:Data: 2023-05-24T21:06:53.499 DEBUG:boto:Headers: {} 2023-05-24T21:06:53.499 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:06:53.499 DEBUG:boto:Port: 8000 2023-05-24T21:06:53.500 DEBUG:boto:Params: {} 2023-05-24T21:06:53.500 DEBUG:boto:Token: None 2023-05-24T21:06:53.500 DEBUG:boto:StringToSign: DELETE Wed, 24 May 2023 21:06:53 GMT /qskrps-9/myobj 2023-05-24T21:06:53.501 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:AnFw4e/Jr3GSgeKJeqsW+7Su7D8= 2023-05-24T21:06:53.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:06:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:AnFw4e/Jr3GSgeKJeqsW+7Su7D8=', 'Content-Length': '0'} 2023-05-24T21:06:53.505 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c14b1d4e4fb8b94d-00646e7c6d-110d-a1'), ('Date', 'Wed, 24 May 2023 21:06:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:06:53.505 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:06:53.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:06:53.506 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-9 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:06:53.564 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:06:53.564 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:06:53.615 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:06:53.609+0000 7f96ad993780 1 waiting for incremental sync to catch up: 2023-05-24T21:06:53.615 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000001.4.6, , , , , , , , , ] 2023-05-24T21:06:53.616 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000002.6.6, , , , , , , , , ] 2023-05-24T21:06:58.616 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:06:58.609+0000 7f96ad993780 1 waiting for incremental sync to catch up: 2023-05-24T21:06:58.616 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000001.4.6, , , , , , , , , ] 2023-05-24T21:06:58.616 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000002.6.6, , , , , , , , , ] 2023-05-24T21:07:03.617 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:07:03.610+0000 7f96ad993780 1 waiting for incremental sync to catch up: 2023-05-24T21:07:03.617 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000001.4.6, , , , , , , , , ] 2023-05-24T21:07:03.617 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000002.6.6, , , , , , , , , ] 2023-05-24T21:07:08.618 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:07:08.611+0000 7f96ad993780 1 waiting for incremental sync to catch up: 2023-05-24T21:07:08.618 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000001.4.6, , , , , , , , , ] 2023-05-24T21:07:08.618 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000002.6.6, , , , , , , , , ] 2023-05-24T21:07:13.619 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:07:13.612+0000 7f96ad993780 1 bucket sync caught up with source: 2023-05-24T21:07:13.620 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000002.6.6, , , , , , , , , ] 2023-05-24T21:07:13.620 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000002.6.6, , , , , , , , , ] 2023-05-24T21:07:13.620 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:07:13.612+0000 7f96ad993780 0 bucket checkpoint complete 2023-05-24T21:07:13.638 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qskrps-9 zones={a2, a1} 2023-05-24T21:07:13.639 DEBUG:boto:path=/qskrps-9/ 2023-05-24T21:07:13.639 DEBUG:boto:auth_path=/qskrps-9/ 2023-05-24T21:07:13.640 DEBUG:boto:Method: HEAD 2023-05-24T21:07:13.640 DEBUG:boto:Path: /qskrps-9/ 2023-05-24T21:07:13.640 DEBUG:boto:Data: 2023-05-24T21:07:13.641 DEBUG:boto:Headers: {} 2023-05-24T21:07:13.641 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:07:13.641 DEBUG:boto:Port: 8001 2023-05-24T21:07:13.642 DEBUG:boto:Params: {} 2023-05-24T21:07:13.642 DEBUG:boto:Token: None 2023-05-24T21:07:13.643 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:07:13 GMT /qskrps-9/ 2023-05-24T21:07:13.643 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:F61hTyp93MiZHzjRyeW4iuw8Glg= 2023-05-24T21:07:13.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:07:13 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:F61hTyp93MiZHzjRyeW4iuw8Glg=', 'Content-Length': '0'} 2023-05-24T21:07:13.646 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000b377e9123cf1611-00646e7c81-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:07:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:07:13.646 DEBUG:boto:path=/qskrps-9/ 2023-05-24T21:07:13.647 DEBUG:boto:auth_path=/qskrps-9/ 2023-05-24T21:07:13.647 DEBUG:boto:Method: HEAD 2023-05-24T21:07:13.648 DEBUG:boto:Path: /qskrps-9/ 2023-05-24T21:07:13.648 DEBUG:boto:Data: 2023-05-24T21:07:13.648 DEBUG:boto:Headers: {} 2023-05-24T21:07:13.649 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:07:13.649 DEBUG:boto:Port: 8000 2023-05-24T21:07:13.649 DEBUG:boto:Params: {} 2023-05-24T21:07:13.650 DEBUG:boto:Token: None 2023-05-24T21:07:13.650 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:07:13 GMT /qskrps-9/ 2023-05-24T21:07:13.650 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:F61hTyp93MiZHzjRyeW4iuw8Glg= 2023-05-24T21:07:13.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:07:13 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:F61hTyp93MiZHzjRyeW4iuw8Glg=', 'Content-Length': '0'} 2023-05-24T21:07:13.653 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c1f3716fa380653-00646e7c81-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:07:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:07:13.654 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-24T21:07:13.654 DEBUG:boto:path=/qskrps-9/ 2023-05-24T21:07:13.654 DEBUG:boto:auth_path=/qskrps-9/ 2023-05-24T21:07:13.655 DEBUG:boto:path=/qskrps-9/?versions&max-keys=999 2023-05-24T21:07:13.655 DEBUG:boto:auth_path=/qskrps-9/?versions&max-keys=999 2023-05-24T21:07:13.655 DEBUG:boto:Method: GET 2023-05-24T21:07:13.656 DEBUG:boto:Path: /qskrps-9/?versions&max-keys=999 2023-05-24T21:07:13.656 DEBUG:boto:Data: 2023-05-24T21:07:13.656 DEBUG:boto:Headers: {} 2023-05-24T21:07:13.657 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:07:13.657 DEBUG:boto:Port: 8001 2023-05-24T21:07:13.657 DEBUG:boto:Params: {} 2023-05-24T21:07:13.658 DEBUG:boto:Token: None 2023-05-24T21:07:13.658 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:07:13 GMT /qskrps-9/?versions 2023-05-24T21:07:13.659 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:MwcWSBuqKPsGbE92JygXlpiqkaQ= 2023-05-24T21:07:13.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:07:13 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:MwcWSBuqKPsGbE92JygXlpiqkaQ=', 'Content-Length': '0'} 2023-05-24T21:07:13.661 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea9a7f34dfc4a719-00646e7c81-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:07:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:07:13.662 DEBUG:boto:b'qskrps-9999false' 2023-05-24T21:07:13.663 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-24T21:07:13.663 DEBUG:boto:path=/qskrps-9/ 2023-05-24T21:07:13.663 DEBUG:boto:auth_path=/qskrps-9/ 2023-05-24T21:07:13.664 DEBUG:boto:path=/qskrps-9/?versions&max-keys=999 2023-05-24T21:07:13.664 DEBUG:boto:auth_path=/qskrps-9/?versions&max-keys=999 2023-05-24T21:07:13.665 DEBUG:boto:Method: GET 2023-05-24T21:07:13.665 DEBUG:boto:Path: /qskrps-9/?versions&max-keys=999 2023-05-24T21:07:13.665 DEBUG:boto:Data: 2023-05-24T21:07:13.666 DEBUG:boto:Headers: {} 2023-05-24T21:07:13.666 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:07:13.666 DEBUG:boto:Port: 8000 2023-05-24T21:07:13.667 DEBUG:boto:Params: {} 2023-05-24T21:07:13.667 DEBUG:boto:Token: None 2023-05-24T21:07:13.667 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:07:13 GMT /qskrps-9/?versions 2023-05-24T21:07:13.668 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:MwcWSBuqKPsGbE92JygXlpiqkaQ= 2023-05-24T21:07:13.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:07:13 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:MwcWSBuqKPsGbE92JygXlpiqkaQ=', 'Content-Length': '0'} 2023-05-24T21:07:13.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005ffc70a9eca3056-00646e7c81-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:07:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:07:13.712 DEBUG:boto:b'qskrps-9999false' 2023-05-24T21:07:13.713 DEBUG:boto:path=/qskrps-9/ 2023-05-24T21:07:13.714 DEBUG:boto:auth_path=/qskrps-9/ 2023-05-24T21:07:13.714 DEBUG:boto:path=/qskrps-9/?versions&max-keys=999 2023-05-24T21:07:13.714 DEBUG:boto:auth_path=/qskrps-9/?versions&max-keys=999 2023-05-24T21:07:13.715 DEBUG:boto:Method: GET 2023-05-24T21:07:13.715 DEBUG:boto:Path: /qskrps-9/?versions&max-keys=999 2023-05-24T21:07:13.715 DEBUG:boto:Data: 2023-05-24T21:07:13.716 DEBUG:boto:Headers: {} 2023-05-24T21:07:13.716 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:07:13.717 DEBUG:boto:Port: 8001 2023-05-24T21:07:13.717 DEBUG:boto:Params: {} 2023-05-24T21:07:13.717 DEBUG:boto:Token: None 2023-05-24T21:07:13.718 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:07:13 GMT /qskrps-9/?versions 2023-05-24T21:07:13.718 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:MwcWSBuqKPsGbE92JygXlpiqkaQ= 2023-05-24T21:07:13.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:07:13 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:MwcWSBuqKPsGbE92JygXlpiqkaQ=', 'Content-Length': '0'} 2023-05-24T21:07:13.721 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d58ba0a62d98649-00646e7c81-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:07:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:07:13.722 DEBUG:boto:b'qskrps-9999false' 2023-05-24T21:07:13.722 DEBUG:boto:path=/qskrps-9/ 2023-05-24T21:07:13.723 DEBUG:boto:auth_path=/qskrps-9/ 2023-05-24T21:07:13.723 DEBUG:boto:path=/qskrps-9/?versions&max-keys=999 2023-05-24T21:07:13.723 DEBUG:boto:auth_path=/qskrps-9/?versions&max-keys=999 2023-05-24T21:07:13.724 DEBUG:boto:Method: GET 2023-05-24T21:07:13.724 DEBUG:boto:Path: /qskrps-9/?versions&max-keys=999 2023-05-24T21:07:13.724 DEBUG:boto:Data: 2023-05-24T21:07:13.725 DEBUG:boto:Headers: {} 2023-05-24T21:07:13.725 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:07:13.725 DEBUG:boto:Port: 8000 2023-05-24T21:07:13.726 DEBUG:boto:Params: {} 2023-05-24T21:07:13.726 DEBUG:boto:Token: None 2023-05-24T21:07:13.727 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:07:13 GMT /qskrps-9/?versions 2023-05-24T21:07:13.727 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:MwcWSBuqKPsGbE92JygXlpiqkaQ= 2023-05-24T21:07:13.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:07:13 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:MwcWSBuqKPsGbE92JygXlpiqkaQ=', 'Content-Length': '0'} 2023-05-24T21:07:13.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa7621acb963bd66-00646e7c81-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:07:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:07:13.730 DEBUG:boto:b'qskrps-9999false' 2023-05-24T21:07:13.731 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qskrps-9 zones={a2, a1} 2023-05-24T21:07:13.731 DEBUG:boto:path=/qskrps-10/ 2023-05-24T21:07:13.732 DEBUG:boto:auth_path=/qskrps-10/ 2023-05-24T21:07:13.732 DEBUG:boto:Method: HEAD 2023-05-24T21:07:13.732 DEBUG:boto:Path: /qskrps-10/ 2023-05-24T21:07:13.733 DEBUG:boto:Data: 2023-05-24T21:07:13.733 DEBUG:boto:Headers: {} 2023-05-24T21:07:13.733 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:07:13.734 DEBUG:boto:Port: 8001 2023-05-24T21:07:13.734 DEBUG:boto:Params: {} 2023-05-24T21:07:13.735 DEBUG:boto:Token: None 2023-05-24T21:07:13.735 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:07:13 GMT /qskrps-10/ 2023-05-24T21:07:13.735 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ql1WR2a/xAqjySyFoYdPdAUujmw= 2023-05-24T21:07:13.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:07:13 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ql1WR2a/xAqjySyFoYdPdAUujmw=', 'Content-Length': '0'} 2023-05-24T21:07:13.738 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000152d5660d24ef269-00646e7c81-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:07:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:07:13.739 DEBUG:boto:path=/qskrps-10/myobj 2023-05-24T21:07:13.740 DEBUG:boto:auth_path=/qskrps-10/myobj 2023-05-24T21:07:13.741 DEBUG:boto:Method: HEAD 2023-05-24T21:07:13.742 DEBUG:boto:Path: /qskrps-10/myobj 2023-05-24T21:07:13.742 DEBUG:boto:Data: 2023-05-24T21:07:13.743 DEBUG:boto:Headers: {} 2023-05-24T21:07:13.744 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:07:13.745 DEBUG:boto:Port: 8001 2023-05-24T21:07:13.746 DEBUG:boto:Params: {} 2023-05-24T21:07:13.746 DEBUG:boto:Token: None 2023-05-24T21:07:13.747 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:07:13 GMT /qskrps-10/myobj 2023-05-24T21:07:13.748 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:WjiwuKMaYO0qA4lNZGD/KzGeN4Y= 2023-05-24T21:07:13.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:07:13 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WjiwuKMaYO0qA4lNZGD/KzGeN4Y=', 'Content-Length': '0'} 2023-05-24T21:07:13.750 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:06:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c763567077f5eb07-00646e7c81-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:07:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:07:13.751 DEBUG:boto:path=/qskrps-10/myobj 2023-05-24T21:07:13.751 DEBUG:boto:auth_path=/qskrps-10/myobj 2023-05-24T21:07:13.752 DEBUG:boto:Method: DELETE 2023-05-24T21:07:13.752 DEBUG:boto:Path: /qskrps-10/myobj 2023-05-24T21:07:13.753 DEBUG:boto:Data: 2023-05-24T21:07:13.753 DEBUG:boto:Headers: {} 2023-05-24T21:07:13.753 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:07:13.754 DEBUG:boto:Port: 8001 2023-05-24T21:07:13.754 DEBUG:boto:Params: {} 2023-05-24T21:07:13.754 DEBUG:boto:Token: None 2023-05-24T21:07:13.755 DEBUG:boto:StringToSign: DELETE Wed, 24 May 2023 21:07:13 GMT /qskrps-10/myobj 2023-05-24T21:07:13.755 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:HcEQVcqAmZdueIM1y9U6ptA5AfU= 2023-05-24T21:07:13.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:07:13 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:HcEQVcqAmZdueIM1y9U6ptA5AfU=', 'Content-Length': '0'} 2023-05-24T21:07:13.762 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b003e96d619743e3-00646e7c81-1111-a2'), ('Date', 'Wed, 24 May 2023 21:07:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:07:13.763 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:07:13.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:07:13.764 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-10 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:07:13.824 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:07:13.825 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:07:13.862 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:07:13.855+0000 7f7a8031c780 1 waiting for incremental sync to catch up: 2023-05-24T21:07:13.862 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000001.20.6, , , , , , , , , ] 2023-05-24T21:07:13.862 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000002.23.6, , , , , , , , , ] 2023-05-24T21:07:18.863 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:07:18.857+0000 7f7a8031c780 1 waiting for incremental sync to catch up: 2023-05-24T21:07:18.864 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000001.20.6, , , , , , , , , ] 2023-05-24T21:07:18.864 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000002.23.6, , , , , , , , , ] 2023-05-24T21:07:23.864 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:07:23.858+0000 7f7a8031c780 1 waiting for incremental sync to catch up: 2023-05-24T21:07:23.865 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000001.20.6, , , , , , , , , ] 2023-05-24T21:07:23.865 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000002.23.6, , , , , , , , , ] 2023-05-24T21:07:28.865 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:07:28.859+0000 7f7a8031c780 1 waiting for incremental sync to catch up: 2023-05-24T21:07:28.866 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000001.20.6, , , , , , , , , ] 2023-05-24T21:07:28.866 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000002.23.6, , , , , , , , , ] 2023-05-24T21:07:33.866 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:07:33.861+0000 7f7a8031c780 1 bucket sync caught up with source: 2023-05-24T21:07:33.867 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000002.23.6, , , , , , , , , ] 2023-05-24T21:07:33.867 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000002.23.6, , , , , , , , , ] 2023-05-24T21:07:33.867 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:07:33.861+0000 7f7a8031c780 0 bucket checkpoint complete 2023-05-24T21:07:33.891 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qskrps-10 zones={a1, a2} 2023-05-24T21:07:33.891 DEBUG:boto:path=/qskrps-10/ 2023-05-24T21:07:33.892 DEBUG:boto:auth_path=/qskrps-10/ 2023-05-24T21:07:33.892 DEBUG:boto:Method: HEAD 2023-05-24T21:07:33.892 DEBUG:boto:Path: /qskrps-10/ 2023-05-24T21:07:33.893 DEBUG:boto:Data: 2023-05-24T21:07:33.893 DEBUG:boto:Headers: {} 2023-05-24T21:07:33.893 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:07:33.894 DEBUG:boto:Port: 8000 2023-05-24T21:07:33.894 DEBUG:boto:Params: {} 2023-05-24T21:07:33.895 DEBUG:boto:Token: None 2023-05-24T21:07:33.895 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:07:33 GMT /qskrps-10/ 2023-05-24T21:07:33.895 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:UXaJuzhdmXXSRD2ncsHslzMnXEo= 2023-05-24T21:07:33.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:07:33 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:UXaJuzhdmXXSRD2ncsHslzMnXEo=', 'Content-Length': '0'} 2023-05-24T21:07:33.898 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000100c2384d9afa58e-00646e7c95-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:07:33.899 DEBUG:boto:path=/qskrps-10/ 2023-05-24T21:07:33.899 DEBUG:boto:auth_path=/qskrps-10/ 2023-05-24T21:07:33.899 DEBUG:boto:Method: HEAD 2023-05-24T21:07:33.900 DEBUG:boto:Path: /qskrps-10/ 2023-05-24T21:07:33.900 DEBUG:boto:Data: 2023-05-24T21:07:33.900 DEBUG:boto:Headers: {} 2023-05-24T21:07:33.901 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:07:33.901 DEBUG:boto:Port: 8001 2023-05-24T21:07:33.901 DEBUG:boto:Params: {} 2023-05-24T21:07:33.902 DEBUG:boto:Token: None 2023-05-24T21:07:33.902 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:07:33 GMT /qskrps-10/ 2023-05-24T21:07:33.903 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:UXaJuzhdmXXSRD2ncsHslzMnXEo= 2023-05-24T21:07:33.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:07:33 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:UXaJuzhdmXXSRD2ncsHslzMnXEo=', 'Content-Length': '0'} 2023-05-24T21:07:33.905 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000022eb2e31709f1446-00646e7c95-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:07:33.906 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-24T21:07:33.906 DEBUG:boto:path=/qskrps-10/ 2023-05-24T21:07:33.907 DEBUG:boto:auth_path=/qskrps-10/ 2023-05-24T21:07:33.907 DEBUG:boto:path=/qskrps-10/?versions&max-keys=999 2023-05-24T21:07:33.907 DEBUG:boto:auth_path=/qskrps-10/?versions&max-keys=999 2023-05-24T21:07:33.908 DEBUG:boto:Method: GET 2023-05-24T21:07:33.908 DEBUG:boto:Path: /qskrps-10/?versions&max-keys=999 2023-05-24T21:07:33.908 DEBUG:boto:Data: 2023-05-24T21:07:33.909 DEBUG:boto:Headers: {} 2023-05-24T21:07:33.909 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:07:33.909 DEBUG:boto:Port: 8000 2023-05-24T21:07:33.910 DEBUG:boto:Params: {} 2023-05-24T21:07:33.910 DEBUG:boto:Token: None 2023-05-24T21:07:33.910 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:07:33 GMT /qskrps-10/?versions 2023-05-24T21:07:33.911 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ZgFDRNWeZutBOeBsVF1yV+0sDrI= 2023-05-24T21:07:33.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:07:33 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ZgFDRNWeZutBOeBsVF1yV+0sDrI=', 'Content-Length': '0'} 2023-05-24T21:07:33.914 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b345de1e92f8ff2-00646e7c95-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:07:33.914 DEBUG:boto:b'qskrps-10999false' 2023-05-24T21:07:33.915 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-24T21:07:33.915 DEBUG:boto:path=/qskrps-10/ 2023-05-24T21:07:33.915 DEBUG:boto:auth_path=/qskrps-10/ 2023-05-24T21:07:33.916 DEBUG:boto:path=/qskrps-10/?versions&max-keys=999 2023-05-24T21:07:33.916 DEBUG:boto:auth_path=/qskrps-10/?versions&max-keys=999 2023-05-24T21:07:33.917 DEBUG:boto:Method: GET 2023-05-24T21:07:33.917 DEBUG:boto:Path: /qskrps-10/?versions&max-keys=999 2023-05-24T21:07:33.917 DEBUG:boto:Data: 2023-05-24T21:07:33.918 DEBUG:boto:Headers: {} 2023-05-24T21:07:33.918 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:07:33.918 DEBUG:boto:Port: 8001 2023-05-24T21:07:33.919 DEBUG:boto:Params: {} 2023-05-24T21:07:33.919 DEBUG:boto:Token: None 2023-05-24T21:07:33.920 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:07:33 GMT /qskrps-10/?versions 2023-05-24T21:07:33.920 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ZgFDRNWeZutBOeBsVF1yV+0sDrI= 2023-05-24T21:07:33.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:07:33 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ZgFDRNWeZutBOeBsVF1yV+0sDrI=', 'Content-Length': '0'} 2023-05-24T21:07:33.923 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075f2845731d4c905-00646e7c95-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:07:33.964 DEBUG:boto:b'qskrps-10999false' 2023-05-24T21:07:33.965 DEBUG:boto:path=/qskrps-10/ 2023-05-24T21:07:33.965 DEBUG:boto:auth_path=/qskrps-10/ 2023-05-24T21:07:33.966 DEBUG:boto:path=/qskrps-10/?versions&max-keys=999 2023-05-24T21:07:33.966 DEBUG:boto:auth_path=/qskrps-10/?versions&max-keys=999 2023-05-24T21:07:33.966 DEBUG:boto:Method: GET 2023-05-24T21:07:33.967 DEBUG:boto:Path: /qskrps-10/?versions&max-keys=999 2023-05-24T21:07:33.967 DEBUG:boto:Data: 2023-05-24T21:07:33.967 DEBUG:boto:Headers: {} 2023-05-24T21:07:33.968 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:07:33.968 DEBUG:boto:Port: 8000 2023-05-24T21:07:33.968 DEBUG:boto:Params: {} 2023-05-24T21:07:33.969 DEBUG:boto:Token: None 2023-05-24T21:07:33.969 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:07:33 GMT /qskrps-10/?versions 2023-05-24T21:07:33.970 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ZgFDRNWeZutBOeBsVF1yV+0sDrI= 2023-05-24T21:07:33.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:07:33 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ZgFDRNWeZutBOeBsVF1yV+0sDrI=', 'Content-Length': '0'} 2023-05-24T21:07:33.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082d154dc6ef16d51-00646e7c95-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:07:33.973 DEBUG:boto:b'qskrps-10999false' 2023-05-24T21:07:33.973 DEBUG:boto:path=/qskrps-10/ 2023-05-24T21:07:33.974 DEBUG:boto:auth_path=/qskrps-10/ 2023-05-24T21:07:33.974 DEBUG:boto:path=/qskrps-10/?versions&max-keys=999 2023-05-24T21:07:33.975 DEBUG:boto:auth_path=/qskrps-10/?versions&max-keys=999 2023-05-24T21:07:33.975 DEBUG:boto:Method: GET 2023-05-24T21:07:33.975 DEBUG:boto:Path: /qskrps-10/?versions&max-keys=999 2023-05-24T21:07:33.976 DEBUG:boto:Data: 2023-05-24T21:07:33.976 DEBUG:boto:Headers: {} 2023-05-24T21:07:33.976 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:07:33.977 DEBUG:boto:Port: 8001 2023-05-24T21:07:33.977 DEBUG:boto:Params: {} 2023-05-24T21:07:33.977 DEBUG:boto:Token: None 2023-05-24T21:07:33.978 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:07:33 GMT /qskrps-10/?versions 2023-05-24T21:07:33.978 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ZgFDRNWeZutBOeBsVF1yV+0sDrI= 2023-05-24T21:07:33.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:07:33 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ZgFDRNWeZutBOeBsVF1yV+0sDrI=', 'Content-Length': '0'} 2023-05-24T21:07:33.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068053836bd47d631-00646e7c95-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:07:33.981 DEBUG:boto:b'qskrps-10999false' 2023-05-24T21:07:33.982 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qskrps-10 zones={a1, a2} 2023-05-24T21:07:33.982 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_delete ... ok 2023-05-24T21:07:33.983 INFO:rgw_multi.tests:create bucket zone=a1 name=qskrps-11 2023-05-24T21:07:33.984 DEBUG:boto:path=/qskrps-11/ 2023-05-24T21:07:33.984 DEBUG:boto:auth_path=/qskrps-11/ 2023-05-24T21:07:33.984 DEBUG:boto:Method: PUT 2023-05-24T21:07:33.985 DEBUG:boto:Path: /qskrps-11/ 2023-05-24T21:07:33.985 DEBUG:boto:Data: 2023-05-24T21:07:33.985 DEBUG:boto:Headers: {} 2023-05-24T21:07:33.986 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:07:33.986 DEBUG:boto:Port: 8000 2023-05-24T21:07:33.986 DEBUG:boto:Params: {} 2023-05-24T21:07:33.987 DEBUG:boto:Token: None 2023-05-24T21:07:33.987 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:07:33 GMT /qskrps-11/ 2023-05-24T21:07:33.987 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:1bk2azBOosf2P8degbzoixBKpqw= 2023-05-24T21:07:33.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:07:33 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:1bk2azBOosf2P8degbzoixBKpqw=', 'Content-Length': '0'} 2023-05-24T21:07:34.005 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000859c45149687ca04-00646e7c95-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:07:34.005 INFO:rgw_multi.tests:create bucket zone=a2 name=qskrps-12 2023-05-24T21:07:34.006 DEBUG:boto:path=/qskrps-12/ 2023-05-24T21:07:34.006 DEBUG:boto:auth_path=/qskrps-12/ 2023-05-24T21:07:34.006 DEBUG:boto:Method: PUT 2023-05-24T21:07:34.007 DEBUG:boto:Path: /qskrps-12/ 2023-05-24T21:07:34.007 DEBUG:boto:Data: 2023-05-24T21:07:34.007 DEBUG:boto:Headers: {} 2023-05-24T21:07:34.008 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:07:34.008 DEBUG:boto:Port: 8001 2023-05-24T21:07:34.008 DEBUG:boto:Params: {} 2023-05-24T21:07:34.009 DEBUG:boto:Token: None 2023-05-24T21:07:34.009 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:07:34 GMT /qskrps-12/ 2023-05-24T21:07:34.010 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:fyWsI1/Gw6IRsaRzaQGJyIZt2gw= 2023-05-24T21:07:34.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:07:34 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:fyWsI1/Gw6IRsaRzaQGJyIZt2gw=', 'Content-Length': '0'} 2023-05-24T21:07:34.034 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d2a7124662a8c471-00646e7c96-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:07:34.034 DEBUG:boto:path=/qskrps-11/ 2023-05-24T21:07:34.035 DEBUG:boto:auth_path=/qskrps-11/ 2023-05-24T21:07:34.035 DEBUG:boto:path=/qskrps-11/?versioning 2023-05-24T21:07:34.035 DEBUG:boto:auth_path=/qskrps-11/?versioning 2023-05-24T21:07:34.035 DEBUG:boto:Method: PUT 2023-05-24T21:07:34.036 DEBUG:boto:Path: /qskrps-11/?versioning 2023-05-24T21:07:34.036 DEBUG:boto:Data: Enabled Disabled 2023-05-24T21:07:34.037 DEBUG:boto:Headers: {} 2023-05-24T21:07:34.037 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:07:34.037 DEBUG:boto:Port: 8000 2023-05-24T21:07:34.038 DEBUG:boto:Params: {} 2023-05-24T21:07:34.038 DEBUG:boto:Token: None 2023-05-24T21:07:34.039 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:07:34 GMT /qskrps-11/?versioning 2023-05-24T21:07:34.039 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:RsSUBJHeZ0hBKAuepYq4dWT/O2s= 2023-05-24T21:07:34.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:07:34 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:RsSUBJHeZ0hBKAuepYq4dWT/O2s=', 'Content-Length': '228'} 2023-05-24T21:07:34.044 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000027a8b22949ef26d3-00646e7c96-110d-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:07:34.045 DEBUG:boto:path=/qskrps-12/ 2023-05-24T21:07:34.045 DEBUG:boto:auth_path=/qskrps-12/ 2023-05-24T21:07:34.046 DEBUG:boto:path=/qskrps-12/?versioning 2023-05-24T21:07:34.046 DEBUG:boto:auth_path=/qskrps-12/?versioning 2023-05-24T21:07:34.046 DEBUG:boto:Method: PUT 2023-05-24T21:07:34.047 DEBUG:boto:Path: /qskrps-12/?versioning 2023-05-24T21:07:34.047 DEBUG:boto:Data: Enabled Disabled 2023-05-24T21:07:34.047 DEBUG:boto:Headers: {} 2023-05-24T21:07:34.047 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:07:34.048 DEBUG:boto:Port: 8001 2023-05-24T21:07:34.048 DEBUG:boto:Params: {} 2023-05-24T21:07:34.048 DEBUG:boto:Token: None 2023-05-24T21:07:34.049 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:07:34 GMT /qskrps-12/?versioning 2023-05-24T21:07:34.049 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:XsM1z1GEbRhPmZCpCxa+xhZtbX8= 2023-05-24T21:07:34.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:07:34 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:XsM1z1GEbRhPmZCpCxa+xhZtbX8=', 'Content-Length': '228'} 2023-05-24T21:07:34.057 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fac128c353b5a65b-00646e7c96-1111-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:07:34.058 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:07:34.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:07:34.059 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:07:34.109 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:07:34.109 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:07:34.139 INFO:teuthology.orchestra.run.smithi071.stderr:No --period given, using current period=5c04936b-70ea-4415-b4eb-3aad3dc612e5 2023-05-24T21:07:34.167 INFO:teuthology.orchestra.run.smithi071.stdout:[ 2023-05-24T21:07:34.168 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:07:34.168 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962454.049579_106.1", 2023-05-24T21:07:34.168 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:07:34.049579Z" 2023-05-24T21:07:34.168 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:07:34.169 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:07:34.169 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:07:34.169 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:06:32.333844Z" 2023-05-24T21:07:34.169 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:07:34.169 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:07:34.169 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:07:34.169 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:07:34.170 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:07:34.170 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:07:34.170 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:07:34.170 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:07:34.170 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:07:34.170 INFO:teuthology.orchestra.run.smithi071.stdout:] 2023-05-24T21:07:34.171 DEBUG:rgw_multi.tests:master meta markers={0: '1_1684962454.049579_106.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:07:34.172 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-24T21:07:34.172 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:07:34.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:07:34.173 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:07:34.222 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:07:34.222 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:07:34.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:07:34.270+0000 7fe25cb62780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:07:34.277 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:07:34.271+0000 7fe25cb62780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:07:34.297 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:07:34.297 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:07:34.297 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:07:34.297 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:07:34.297 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:07:34.298 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:07:34.298 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:07:34.298 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:07:34.298 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:07:34.298 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:07:34.298 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:07:34.298 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:07:34.299 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:07:34.299 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962453.990744_95.1", 2023-05-24T21:07:34.299 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:07:34.299 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:07:34.299 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:07:34.299 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:07:33.990744Z", 2023-05-24T21:07:34.299 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:07:34.300 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:07:34.300 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:07:34.300 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:07:34.300 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:07:34.300 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:07:34.300 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:07:34.300 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:07:34.301 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:07:34.301 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:07:34.301 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:07:34.301 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:07:34.301 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:07:34.301 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:07:34.302 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:07:34.302 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:07:34.302 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:07:34.302 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:07:34.302 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:07:34.302 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:07:34.302 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:07:34.303 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:07:34.303 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:07:34.303 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:07:34.303 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:07:34.303 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:07:34.303 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:07:34.303 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:07:34.304 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:07:34.304 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:07:34.305 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:07:34.305 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:07:34.305 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:07:34.305 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:07:34.305 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:07:34.306 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:07:34.306 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:07:34.306 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:07:34.306 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:07:34.306 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:07:34.306 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:07:34.306 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:07:34.307 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:07:34.307 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:07:34.307 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:07:34.307 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:07:34Z" 2023-05-24T21:07:34.307 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:07:34.308 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962453.990744_95.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:07:33.990744Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:07:34Z" } 2023-05-24T21:07:34.309 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962453.990744_95.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:07:33.990744Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:07:34.309 DEBUG:rgw_multi.tests:log_status={0: '1_1684962454.049579_106.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:07:34.310 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962453.990744_95.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:07:34.310 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684962454.049579_106.1 target=1_1684962453.990744_95.1 2023-05-24T21:07:39.311 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:07:39.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:07:39.312 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:07:39.369 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:07:39.370 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:07:39.428 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:07:39.422+0000 7fc1b78f2780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:07:39.429 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:07:39.423+0000 7fc1b78f2780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:07:39.449 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:07:39.450 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:07:39.450 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:07:39.450 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:07:39.450 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:07:39.450 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:07:39.450 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:07:39.451 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:07:39.451 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:07:39.451 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:07:39.451 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:07:39.451 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:07:39.451 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:07:39.451 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962454.049579_106.1", 2023-05-24T21:07:39.452 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:07:39.452 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:07:39.452 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:07:39.452 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:07:34.049579Z", 2023-05-24T21:07:39.452 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:07:39.452 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:07:39.452 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:07:39.453 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:07:39.453 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:07:39.453 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:07:39.453 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:07:39.453 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:07:39.453 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:07:39.453 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:07:39.454 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:07:39.454 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:07:39.454 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:07:39.454 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:07:39.454 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:07:39.454 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:07:39.454 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:07:39.455 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:07:39.455 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:07:39.455 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:07:39.455 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:07:39.455 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:07:39.455 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:07:39.455 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:07:39.456 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:07:39.456 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:07:39.456 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:07:39.456 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:07:39.456 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:07:39.456 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:07:39.456 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:07:39.457 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:07:39.457 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:07:39.457 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:07:39.457 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:07:39.457 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:07:39.457 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:07:39.457 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:07:39.458 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:07:39.458 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:07:39.458 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:07:39.458 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:07:39.458 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:07:39.458 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:07:39.458 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:07:39.459 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:07:39Z" 2023-05-24T21:07:39.459 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:07:39.460 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962454.049579_106.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:07:34.049579Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:07:39Z" } 2023-05-24T21:07:39.460 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962454.049579_106.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:07:34.049579Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:07:39.461 DEBUG:rgw_multi.tests:log_status={0: '1_1684962454.049579_106.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:07:39.461 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962454.049579_106.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:07:39.461 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-24T21:07:39.462 DEBUG:boto:path=/qskrps-11/ 2023-05-24T21:07:39.462 DEBUG:boto:auth_path=/qskrps-11/ 2023-05-24T21:07:39.462 DEBUG:boto:Method: HEAD 2023-05-24T21:07:39.463 DEBUG:boto:Path: /qskrps-11/ 2023-05-24T21:07:39.463 DEBUG:boto:Data: 2023-05-24T21:07:39.463 DEBUG:boto:Headers: {} 2023-05-24T21:07:39.464 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:07:39.464 DEBUG:boto:Port: 8000 2023-05-24T21:07:39.464 DEBUG:boto:Params: {} 2023-05-24T21:07:39.465 DEBUG:boto:Token: None 2023-05-24T21:07:39.465 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:07:39 GMT /qskrps-11/ 2023-05-24T21:07:39.465 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:XYjr7Ved2z96kV+b7Bx6aNqMo2c= 2023-05-24T21:07:39.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:07:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:XYjr7Ved2z96kV+b7Bx6aNqMo2c=', 'Content-Length': '0'} 2023-05-24T21:07:39.468 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006d4778f3298f0793-00646e7c9b-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:07:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:07:39.469 DEBUG:boto:path=/qskrps-11/dummy 2023-05-24T21:07:39.469 DEBUG:boto:auth_path=/qskrps-11/dummy 2023-05-24T21:07:39.469 DEBUG:boto:Method: PUT 2023-05-24T21:07:39.470 DEBUG:boto:Path: /qskrps-11/dummy 2023-05-24T21:07:39.470 DEBUG:boto:Data: 2023-05-24T21:07:39.470 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-05-24T21:07:39.471 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:07:39.471 DEBUG:boto:Port: 8000 2023-05-24T21:07:39.471 DEBUG:boto:Params: {} 2023-05-24T21:07:39.472 DEBUG:boto:Token: None 2023-05-24T21:07:39.472 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Wed, 24 May 2023 21:07:39 GMT /qskrps-11/dummy 2023-05-24T21:07:39.473 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:WbOmXbWYhwqdbg2hEBHjDEaOfok= 2023-05-24T21:07:39.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:07:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WbOmXbWYhwqdbg2hEBHjDEaOfok='} 2023-05-24T21:07:39.489 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'K0LgvK2trkm-uHhxLikH3sxyZSZDazN'), ('x-amz-request-id', 'tx000008a1216e1aaf25ad9-00646e7c9b-110d-a1'), ('Date', 'Wed, 24 May 2023 21:07:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:07:39.490 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:07:39.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:07:39.490 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-11 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:07:39.546 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:07:39.547 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:07:39.584 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:07:39.578+0000 7f2eccd11780 1 waiting to reach incremental sync.. 2023-05-24T21:07:44.584 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:07:44.578+0000 7f2eccd11780 1 waiting to reach incremental sync.. 2023-05-24T21:07:49.585 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:07:49.579+0000 7f2eccd11780 1 waiting to reach incremental sync.. 2023-05-24T21:07:54.587 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:07:54.580+0000 7f2eccd11780 1 bucket sync caught up with source: 2023-05-24T21:07:54.587 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000002.6.11, , , , , ] 2023-05-24T21:07:54.587 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000002.6.11, , , , , ] 2023-05-24T21:07:54.588 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:07:54.580+0000 7f2eccd11780 0 bucket checkpoint complete 2023-05-24T21:07:54.603 DEBUG:boto:path=/qskrps-12/ 2023-05-24T21:07:54.604 DEBUG:boto:auth_path=/qskrps-12/ 2023-05-24T21:07:54.604 DEBUG:boto:Method: HEAD 2023-05-24T21:07:54.605 DEBUG:boto:Path: /qskrps-12/ 2023-05-24T21:07:54.606 DEBUG:boto:Data: 2023-05-24T21:07:54.607 DEBUG:boto:Headers: {} 2023-05-24T21:07:54.607 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:07:54.608 DEBUG:boto:Port: 8001 2023-05-24T21:07:54.609 DEBUG:boto:Params: {} 2023-05-24T21:07:54.611 DEBUG:boto:Token: None 2023-05-24T21:07:54.612 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:07:54 GMT /qskrps-12/ 2023-05-24T21:07:54.613 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5pgG8lsmWJg/HJAvePlR25CTVxk= 2023-05-24T21:07:54.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:07:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5pgG8lsmWJg/HJAvePlR25CTVxk=', 'Content-Length': '0'} 2023-05-24T21:07:54.617 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b19457fa3de96140-00646e7caa-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:07:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:07:54.618 DEBUG:boto:path=/qskrps-12/dummy 2023-05-24T21:07:54.619 DEBUG:boto:auth_path=/qskrps-12/dummy 2023-05-24T21:07:54.620 DEBUG:boto:Method: PUT 2023-05-24T21:07:54.620 DEBUG:boto:Path: /qskrps-12/dummy 2023-05-24T21:07:54.621 DEBUG:boto:Data: 2023-05-24T21:07:54.622 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-05-24T21:07:54.623 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:07:54.623 DEBUG:boto:Port: 8001 2023-05-24T21:07:54.624 DEBUG:boto:Params: {} 2023-05-24T21:07:54.624 DEBUG:boto:Token: None 2023-05-24T21:07:54.625 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Wed, 24 May 2023 21:07:54 GMT /qskrps-12/dummy 2023-05-24T21:07:54.626 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:GNG3TdrhNTB4OZpDGK20MshOg44= 2023-05-24T21:07:54.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:07:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:GNG3TdrhNTB4OZpDGK20MshOg44='} 2023-05-24T21:07:54.639 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'km8zCAetA1TzFWDKQlSimNmDeLQdejH'), ('x-amz-request-id', 'tx000007d8a51e991e834a8-00646e7caa-1111-a2'), ('Date', 'Wed, 24 May 2023 21:07:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:07:54.640 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:07:54.641 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:07:54.641 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-12 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:07:54.703 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:07:54.703 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:07:54.751 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:07:54.744+0000 7fb063a63780 1 waiting to reach incremental sync.. 2023-05-24T21:07:59.751 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:07:59.744+0000 7fb063a63780 1 waiting to reach incremental sync.. 2023-05-24T21:08:04.752 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:08:04.745+0000 7fb063a63780 1 waiting to reach incremental sync.. 2023-05-24T21:08:09.752 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:08:09.745+0000 7fb063a63780 1 waiting to reach incremental sync.. 2023-05-24T21:08:14.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:08:14.748+0000 7fb063a63780 1 bucket sync caught up with source: 2023-05-24T21:08:14.754 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000002.34.11, , , , , ] 2023-05-24T21:08:14.754 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000002.34.11, , , , , ] 2023-05-24T21:08:14.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:08:14.748+0000 7fb063a63780 0 bucket checkpoint complete 2023-05-24T21:08:14.776 DEBUG:boto:path=/qskrps-11/ 2023-05-24T21:08:14.777 DEBUG:boto:auth_path=/qskrps-11/ 2023-05-24T21:08:14.777 DEBUG:boto:Method: HEAD 2023-05-24T21:08:14.777 DEBUG:boto:Path: /qskrps-11/ 2023-05-24T21:08:14.778 DEBUG:boto:Data: 2023-05-24T21:08:14.778 DEBUG:boto:Headers: {} 2023-05-24T21:08:14.778 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:08:14.779 DEBUG:boto:Port: 8000 2023-05-24T21:08:14.779 DEBUG:boto:Params: {} 2023-05-24T21:08:14.780 DEBUG:boto:Token: None 2023-05-24T21:08:14.780 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:08:14 GMT /qskrps-11/ 2023-05-24T21:08:14.781 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:t0iHg6ge4tffRKNWQ+8hSpKDH00= 2023-05-24T21:08:14.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:14 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:t0iHg6ge4tffRKNWQ+8hSpKDH00=', 'Content-Length': '0'} 2023-05-24T21:08:14.784 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000032b20f2b453c5c5f-00646e7cbe-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:08:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:14.785 DEBUG:boto:path=/qskrps-11/obj-a1 2023-05-24T21:08:14.785 DEBUG:boto:auth_path=/qskrps-11/obj-a1 2023-05-24T21:08:14.785 DEBUG:boto:Method: PUT 2023-05-24T21:08:14.786 DEBUG:boto:Path: /qskrps-11/obj-a1 2023-05-24T21:08:14.786 DEBUG:boto:Data: 2023-05-24T21:08:14.786 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-24T21:08:14.787 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:08:14.787 DEBUG:boto:Port: 8000 2023-05-24T21:08:14.787 DEBUG:boto:Params: {} 2023-05-24T21:08:14.788 DEBUG:boto:Token: None 2023-05-24T21:08:14.788 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 24 May 2023 21:08:14 GMT /qskrps-11/obj-a1 2023-05-24T21:08:14.789 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Oad5pNyCs0wIqTT5Vw2cWKr3Rmc= 2023-05-24T21:08:14.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:08:14 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Oad5pNyCs0wIqTT5Vw2cWKr3Rmc='} 2023-05-24T21:08:14.809 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '54a54vsjHFa46aVpWzOVMoAJ0PQA-.v'), ('x-amz-request-id', 'tx000002a2bdcb3229e134b-00646e7cbe-110d-a1'), ('Date', 'Wed, 24 May 2023 21:08:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:14.810 DEBUG:rgw_multi.tests:version1 id=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:08:14.811 DEBUG:boto:path=/qskrps-11/obj-a1 2023-05-24T21:08:14.811 DEBUG:boto:auth_path=/qskrps-11/obj-a1 2023-05-24T21:08:14.812 DEBUG:boto:Method: PUT 2023-05-24T21:08:14.812 DEBUG:boto:Path: /qskrps-11/obj-a1 2023-05-24T21:08:14.813 DEBUG:boto:Data: 2023-05-24T21:08:14.813 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-24T21:08:14.813 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:08:14.814 DEBUG:boto:Port: 8000 2023-05-24T21:08:14.814 DEBUG:boto:Params: {} 2023-05-24T21:08:14.815 DEBUG:boto:Token: None 2023-05-24T21:08:14.815 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Wed, 24 May 2023 21:08:14 GMT /qskrps-11/obj-a1 2023-05-24T21:08:14.815 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:fLmopCo3mI8AlVThh+uol9Hv8Mo= 2023-05-24T21:08:14.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:08:14 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:fLmopCo3mI8AlVThh+uol9Hv8Mo='} 2023-05-24T21:08:14.834 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'zTbik.BUrGZSZdMx5C8gAgpYGOkQYLY'), ('x-amz-request-id', 'tx0000002ce52fd641bb8a8-00646e7cbe-110d-a1'), ('Date', 'Wed, 24 May 2023 21:08:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:14.835 DEBUG:rgw_multi.tests:version2 id=zTbik.BUrGZSZdMx5C8gAgpYGOkQYLY 2023-05-24T21:08:14.835 DEBUG:boto:path=/qskrps-11/obj-a1 2023-05-24T21:08:14.835 DEBUG:boto:auth_path=/qskrps-11/obj-a1 2023-05-24T21:08:14.836 DEBUG:boto:path=/qskrps-11/obj-a1?versionId=zTbik.BUrGZSZdMx5C8gAgpYGOkQYLY 2023-05-24T21:08:14.836 DEBUG:boto:auth_path=/qskrps-11/obj-a1?versionId=zTbik.BUrGZSZdMx5C8gAgpYGOkQYLY 2023-05-24T21:08:14.836 DEBUG:boto:Method: DELETE 2023-05-24T21:08:14.837 DEBUG:boto:Path: /qskrps-11/obj-a1?versionId=zTbik.BUrGZSZdMx5C8gAgpYGOkQYLY 2023-05-24T21:08:14.837 DEBUG:boto:Data: 2023-05-24T21:08:14.837 DEBUG:boto:Headers: {} 2023-05-24T21:08:14.838 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:08:14.838 DEBUG:boto:Port: 8000 2023-05-24T21:08:14.838 DEBUG:boto:Params: {} 2023-05-24T21:08:14.839 DEBUG:boto:Token: None 2023-05-24T21:08:14.839 DEBUG:boto:StringToSign: DELETE Wed, 24 May 2023 21:08:14 GMT /qskrps-11/obj-a1?versionId=zTbik.BUrGZSZdMx5C8gAgpYGOkQYLY 2023-05-24T21:08:14.840 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4PhV+FOV39eO8pWvNhuUMchp94o= 2023-05-24T21:08:14.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:14 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4PhV+FOV39eO8pWvNhuUMchp94o=', 'Content-Length': '0'} 2023-05-24T21:08:14.859 DEBUG:boto:Response headers: [('x-amz-version-id', 'zTbik.BUrGZSZdMx5C8gAgpYGOkQYLY'), ('x-amz-request-id', 'tx00000e372b936f3f46a56-00646e7cbe-110d-a1'), ('Date', 'Wed, 24 May 2023 21:08:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:14.860 DEBUG:boto:path=/qskrps-11/obj-a1 2023-05-24T21:08:14.861 DEBUG:boto:auth_path=/qskrps-11/obj-a1 2023-05-24T21:08:14.861 DEBUG:boto:Method: PUT 2023-05-24T21:08:14.861 DEBUG:boto:Path: /qskrps-11/obj-a1 2023-05-24T21:08:14.862 DEBUG:boto:Data: 2023-05-24T21:08:14.862 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-24T21:08:14.862 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:08:14.863 DEBUG:boto:Port: 8000 2023-05-24T21:08:14.863 DEBUG:boto:Params: {} 2023-05-24T21:08:14.863 DEBUG:boto:Token: None 2023-05-24T21:08:14.864 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Wed, 24 May 2023 21:08:14 GMT /qskrps-11/obj-a1 2023-05-24T21:08:14.864 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:YrYaHklga0120mAK5MA8T+rYU54= 2023-05-24T21:08:14.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:08:14 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:YrYaHklga0120mAK5MA8T+rYU54='} 2023-05-24T21:08:14.908 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'UFuXOKN-VjQy1xp0vAr.m.OAm9xODjJ'), ('x-amz-request-id', 'tx0000057123e7d8141a8af-00646e7cbe-110d-a1'), ('Date', 'Wed, 24 May 2023 21:08:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:14.909 DEBUG:rgw_multi.tests:version3 id=UFuXOKN-VjQy1xp0vAr.m.OAm9xODjJ 2023-05-24T21:08:14.909 DEBUG:boto:path=/qskrps-11/obj-a1 2023-05-24T21:08:14.910 DEBUG:boto:auth_path=/qskrps-11/obj-a1 2023-05-24T21:08:14.910 DEBUG:boto:path=/qskrps-11/obj-a1?versionId=UFuXOKN-VjQy1xp0vAr.m.OAm9xODjJ 2023-05-24T21:08:14.910 DEBUG:boto:auth_path=/qskrps-11/obj-a1?versionId=UFuXOKN-VjQy1xp0vAr.m.OAm9xODjJ 2023-05-24T21:08:14.911 DEBUG:boto:Method: DELETE 2023-05-24T21:08:14.911 DEBUG:boto:Path: /qskrps-11/obj-a1?versionId=UFuXOKN-VjQy1xp0vAr.m.OAm9xODjJ 2023-05-24T21:08:14.911 DEBUG:boto:Data: 2023-05-24T21:08:14.912 DEBUG:boto:Headers: {} 2023-05-24T21:08:14.912 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:08:14.912 DEBUG:boto:Port: 8000 2023-05-24T21:08:14.913 DEBUG:boto:Params: {} 2023-05-24T21:08:14.913 DEBUG:boto:Token: None 2023-05-24T21:08:14.914 DEBUG:boto:StringToSign: DELETE Wed, 24 May 2023 21:08:14 GMT /qskrps-11/obj-a1?versionId=UFuXOKN-VjQy1xp0vAr.m.OAm9xODjJ 2023-05-24T21:08:14.914 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:cqdIaCvX2/1rg8MvYFUKt8O6smA= 2023-05-24T21:08:14.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:14 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:cqdIaCvX2/1rg8MvYFUKt8O6smA=', 'Content-Length': '0'} 2023-05-24T21:08:14.930 DEBUG:boto:Response headers: [('x-amz-version-id', 'UFuXOKN-VjQy1xp0vAr.m.OAm9xODjJ'), ('x-amz-request-id', 'tx0000021684e12081a6cc4-00646e7cbe-110d-a1'), ('Date', 'Wed, 24 May 2023 21:08:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:14.931 DEBUG:boto:path=/qskrps-11/ 2023-05-24T21:08:14.931 DEBUG:boto:auth_path=/qskrps-11/ 2023-05-24T21:08:14.932 DEBUG:boto:Method: HEAD 2023-05-24T21:08:14.932 DEBUG:boto:Path: /qskrps-11/ 2023-05-24T21:08:14.932 DEBUG:boto:Data: 2023-05-24T21:08:14.933 DEBUG:boto:Headers: {} 2023-05-24T21:08:14.933 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:08:14.933 DEBUG:boto:Port: 8001 2023-05-24T21:08:14.934 DEBUG:boto:Params: {} 2023-05-24T21:08:14.934 DEBUG:boto:Token: None 2023-05-24T21:08:14.934 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:08:14 GMT /qskrps-11/ 2023-05-24T21:08:14.935 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:t0iHg6ge4tffRKNWQ+8hSpKDH00= 2023-05-24T21:08:14.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:14 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:t0iHg6ge4tffRKNWQ+8hSpKDH00=', 'Content-Length': '0'} 2023-05-24T21:08:14.937 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000050ece9049fba0267-00646e7cbe-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:08:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:14.938 DEBUG:boto:path=/qskrps-11/obj-a2 2023-05-24T21:08:14.938 DEBUG:boto:auth_path=/qskrps-11/obj-a2 2023-05-24T21:08:14.939 DEBUG:boto:Method: PUT 2023-05-24T21:08:14.939 DEBUG:boto:Path: /qskrps-11/obj-a2 2023-05-24T21:08:14.939 DEBUG:boto:Data: 2023-05-24T21:08:14.940 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-24T21:08:14.940 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:08:14.940 DEBUG:boto:Port: 8001 2023-05-24T21:08:14.941 DEBUG:boto:Params: {} 2023-05-24T21:08:14.941 DEBUG:boto:Token: None 2023-05-24T21:08:14.941 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 24 May 2023 21:08:14 GMT /qskrps-11/obj-a2 2023-05-24T21:08:14.942 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:nalaA+brh8AJfBWl2gwoaAQAg6s= 2023-05-24T21:08:14.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:08:14 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:nalaA+brh8AJfBWl2gwoaAQAg6s='} 2023-05-24T21:08:14.964 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'YniDC1out7TFV1sI.TLfuqiPcVFd.5S'), ('x-amz-request-id', 'tx00000cd8102ab6babeb5b-00646e7cbe-1111-a2'), ('Date', 'Wed, 24 May 2023 21:08:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:14.965 DEBUG:rgw_multi.tests:version1 id=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:08:14.965 DEBUG:boto:path=/qskrps-11/obj-a2 2023-05-24T21:08:14.966 DEBUG:boto:auth_path=/qskrps-11/obj-a2 2023-05-24T21:08:14.966 DEBUG:boto:Method: PUT 2023-05-24T21:08:14.967 DEBUG:boto:Path: /qskrps-11/obj-a2 2023-05-24T21:08:14.967 DEBUG:boto:Data: 2023-05-24T21:08:14.967 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-24T21:08:14.968 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:08:14.968 DEBUG:boto:Port: 8001 2023-05-24T21:08:14.968 DEBUG:boto:Params: {} 2023-05-24T21:08:14.969 DEBUG:boto:Token: None 2023-05-24T21:08:14.969 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Wed, 24 May 2023 21:08:14 GMT /qskrps-11/obj-a2 2023-05-24T21:08:14.969 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:rFAIvUR60X1o8O6M2+d2kltHDi4= 2023-05-24T21:08:14.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:08:14 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:rFAIvUR60X1o8O6M2+d2kltHDi4='} 2023-05-24T21:08:15.012 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'MSPJ3AedFg8U60FWLuYWRAlk3CK1eJc'), ('x-amz-request-id', 'tx00000799cea0a7320cb85-00646e7cbe-1111-a2'), ('Date', 'Wed, 24 May 2023 21:08:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:15.013 DEBUG:rgw_multi.tests:version2 id=MSPJ3AedFg8U60FWLuYWRAlk3CK1eJc 2023-05-24T21:08:15.013 DEBUG:boto:path=/qskrps-11/obj-a2 2023-05-24T21:08:15.014 DEBUG:boto:auth_path=/qskrps-11/obj-a2 2023-05-24T21:08:15.014 DEBUG:boto:path=/qskrps-11/obj-a2?versionId=MSPJ3AedFg8U60FWLuYWRAlk3CK1eJc 2023-05-24T21:08:15.014 DEBUG:boto:auth_path=/qskrps-11/obj-a2?versionId=MSPJ3AedFg8U60FWLuYWRAlk3CK1eJc 2023-05-24T21:08:15.015 DEBUG:boto:Method: DELETE 2023-05-24T21:08:15.015 DEBUG:boto:Path: /qskrps-11/obj-a2?versionId=MSPJ3AedFg8U60FWLuYWRAlk3CK1eJc 2023-05-24T21:08:15.015 DEBUG:boto:Data: 2023-05-24T21:08:15.016 DEBUG:boto:Headers: {} 2023-05-24T21:08:15.016 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:08:15.016 DEBUG:boto:Port: 8001 2023-05-24T21:08:15.017 DEBUG:boto:Params: {} 2023-05-24T21:08:15.017 DEBUG:boto:Token: None 2023-05-24T21:08:15.017 DEBUG:boto:StringToSign: DELETE Wed, 24 May 2023 21:08:15 GMT /qskrps-11/obj-a2?versionId=MSPJ3AedFg8U60FWLuYWRAlk3CK1eJc 2023-05-24T21:08:15.018 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:nuCJr0xNzz00NRMQy6DLQAwjdA8= 2023-05-24T21:08:15.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:nuCJr0xNzz00NRMQy6DLQAwjdA8=', 'Content-Length': '0'} 2023-05-24T21:08:15.029 DEBUG:boto:Response headers: [('x-amz-version-id', 'MSPJ3AedFg8U60FWLuYWRAlk3CK1eJc'), ('x-amz-request-id', 'tx000009743ab2ada28e221-00646e7cbf-1111-a2'), ('Date', 'Wed, 24 May 2023 21:08:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:15.029 DEBUG:boto:path=/qskrps-11/obj-a2 2023-05-24T21:08:15.030 DEBUG:boto:auth_path=/qskrps-11/obj-a2 2023-05-24T21:08:15.030 DEBUG:boto:Method: PUT 2023-05-24T21:08:15.031 DEBUG:boto:Path: /qskrps-11/obj-a2 2023-05-24T21:08:15.031 DEBUG:boto:Data: 2023-05-24T21:08:15.031 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-24T21:08:15.032 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:08:15.032 DEBUG:boto:Port: 8001 2023-05-24T21:08:15.032 DEBUG:boto:Params: {} 2023-05-24T21:08:15.033 DEBUG:boto:Token: None 2023-05-24T21:08:15.033 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Wed, 24 May 2023 21:08:15 GMT /qskrps-11/obj-a2 2023-05-24T21:08:15.033 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:f4AtZn9N56tQy0/dE2Rc/mqE9WI= 2023-05-24T21:08:15.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:08:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:f4AtZn9N56tQy0/dE2Rc/mqE9WI='} 2023-05-24T21:08:15.043 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'WPbxkBZScAOUyFomi.DqdKY5E0fYc8f'), ('x-amz-request-id', 'tx000006e97a2f1903867fd-00646e7cbf-1111-a2'), ('Date', 'Wed, 24 May 2023 21:08:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:15.043 DEBUG:rgw_multi.tests:version3 id=WPbxkBZScAOUyFomi.DqdKY5E0fYc8f 2023-05-24T21:08:15.044 DEBUG:boto:path=/qskrps-11/obj-a2 2023-05-24T21:08:15.044 DEBUG:boto:auth_path=/qskrps-11/obj-a2 2023-05-24T21:08:15.045 DEBUG:boto:path=/qskrps-11/obj-a2?versionId=WPbxkBZScAOUyFomi.DqdKY5E0fYc8f 2023-05-24T21:08:15.045 DEBUG:boto:auth_path=/qskrps-11/obj-a2?versionId=WPbxkBZScAOUyFomi.DqdKY5E0fYc8f 2023-05-24T21:08:15.045 DEBUG:boto:Method: DELETE 2023-05-24T21:08:15.046 DEBUG:boto:Path: /qskrps-11/obj-a2?versionId=WPbxkBZScAOUyFomi.DqdKY5E0fYc8f 2023-05-24T21:08:15.046 DEBUG:boto:Data: 2023-05-24T21:08:15.046 DEBUG:boto:Headers: {} 2023-05-24T21:08:15.046 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:08:15.047 DEBUG:boto:Port: 8001 2023-05-24T21:08:15.047 DEBUG:boto:Params: {} 2023-05-24T21:08:15.047 DEBUG:boto:Token: None 2023-05-24T21:08:15.048 DEBUG:boto:StringToSign: DELETE Wed, 24 May 2023 21:08:15 GMT /qskrps-11/obj-a2?versionId=WPbxkBZScAOUyFomi.DqdKY5E0fYc8f 2023-05-24T21:08:15.048 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:08jdLaKgAq0aAZuTgmmLDaemhTQ= 2023-05-24T21:08:15.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:08jdLaKgAq0aAZuTgmmLDaemhTQ=', 'Content-Length': '0'} 2023-05-24T21:08:15.058 DEBUG:boto:Response headers: [('x-amz-version-id', 'WPbxkBZScAOUyFomi.DqdKY5E0fYc8f'), ('x-amz-request-id', 'tx00000627000064464c0f2-00646e7cbf-1111-a2'), ('Date', 'Wed, 24 May 2023 21:08:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:15.059 DEBUG:boto:path=/qskrps-12/ 2023-05-24T21:08:15.059 DEBUG:boto:auth_path=/qskrps-12/ 2023-05-24T21:08:15.060 DEBUG:boto:Method: HEAD 2023-05-24T21:08:15.060 DEBUG:boto:Path: /qskrps-12/ 2023-05-24T21:08:15.060 DEBUG:boto:Data: 2023-05-24T21:08:15.061 DEBUG:boto:Headers: {} 2023-05-24T21:08:15.061 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:08:15.061 DEBUG:boto:Port: 8000 2023-05-24T21:08:15.062 DEBUG:boto:Params: {} 2023-05-24T21:08:15.062 DEBUG:boto:Token: None 2023-05-24T21:08:15.062 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:08:15 GMT /qskrps-12/ 2023-05-24T21:08:15.063 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:6zXbh1b3OpLNxjUlfCXSoV+cJt0= 2023-05-24T21:08:15.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:6zXbh1b3OpLNxjUlfCXSoV+cJt0=', 'Content-Length': '0'} 2023-05-24T21:08:15.065 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002398d1d57d8d9006-00646e7cbf-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:08:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:15.066 DEBUG:boto:path=/qskrps-12/obj-a1 2023-05-24T21:08:15.066 DEBUG:boto:auth_path=/qskrps-12/obj-a1 2023-05-24T21:08:15.067 DEBUG:boto:Method: PUT 2023-05-24T21:08:15.067 DEBUG:boto:Path: /qskrps-12/obj-a1 2023-05-24T21:08:15.067 DEBUG:boto:Data: 2023-05-24T21:08:15.068 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-24T21:08:15.068 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:08:15.068 DEBUG:boto:Port: 8000 2023-05-24T21:08:15.069 DEBUG:boto:Params: {} 2023-05-24T21:08:15.069 DEBUG:boto:Token: None 2023-05-24T21:08:15.070 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 24 May 2023 21:08:15 GMT /qskrps-12/obj-a1 2023-05-24T21:08:15.070 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ZyE7BnUd5S0Y2Krwdg9x7TYLxI0= 2023-05-24T21:08:15.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:08:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ZyE7BnUd5S0Y2Krwdg9x7TYLxI0='} 2023-05-24T21:08:15.081 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV'), ('x-amz-request-id', 'tx000002fcad2969a649752-00646e7cbf-110d-a1'), ('Date', 'Wed, 24 May 2023 21:08:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:15.082 DEBUG:rgw_multi.tests:version1 id=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:08:15.082 DEBUG:boto:path=/qskrps-12/obj-a1 2023-05-24T21:08:15.083 DEBUG:boto:auth_path=/qskrps-12/obj-a1 2023-05-24T21:08:15.083 DEBUG:boto:Method: PUT 2023-05-24T21:08:15.083 DEBUG:boto:Path: /qskrps-12/obj-a1 2023-05-24T21:08:15.084 DEBUG:boto:Data: 2023-05-24T21:08:15.084 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-24T21:08:15.084 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:08:15.085 DEBUG:boto:Port: 8000 2023-05-24T21:08:15.085 DEBUG:boto:Params: {} 2023-05-24T21:08:15.085 DEBUG:boto:Token: None 2023-05-24T21:08:15.086 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Wed, 24 May 2023 21:08:15 GMT /qskrps-12/obj-a1 2023-05-24T21:08:15.086 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:xFmnI7ESlN1Bl2gaL6wlBqdDJXo= 2023-05-24T21:08:15.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:08:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:xFmnI7ESlN1Bl2gaL6wlBqdDJXo='} 2023-05-24T21:08:15.128 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'u0vC7fwIKAfOAm.KhunLJ7iVMux6Tui'), ('x-amz-request-id', 'tx000006f7c345048ae0b5e-00646e7cbf-110d-a1'), ('Date', 'Wed, 24 May 2023 21:08:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:15.129 DEBUG:rgw_multi.tests:version2 id=u0vC7fwIKAfOAm.KhunLJ7iVMux6Tui 2023-05-24T21:08:15.129 DEBUG:boto:path=/qskrps-12/obj-a1 2023-05-24T21:08:15.130 DEBUG:boto:auth_path=/qskrps-12/obj-a1 2023-05-24T21:08:15.130 DEBUG:boto:path=/qskrps-12/obj-a1?versionId=u0vC7fwIKAfOAm.KhunLJ7iVMux6Tui 2023-05-24T21:08:15.130 DEBUG:boto:auth_path=/qskrps-12/obj-a1?versionId=u0vC7fwIKAfOAm.KhunLJ7iVMux6Tui 2023-05-24T21:08:15.131 DEBUG:boto:Method: DELETE 2023-05-24T21:08:15.131 DEBUG:boto:Path: /qskrps-12/obj-a1?versionId=u0vC7fwIKAfOAm.KhunLJ7iVMux6Tui 2023-05-24T21:08:15.131 DEBUG:boto:Data: 2023-05-24T21:08:15.132 DEBUG:boto:Headers: {} 2023-05-24T21:08:15.132 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:08:15.132 DEBUG:boto:Port: 8000 2023-05-24T21:08:15.133 DEBUG:boto:Params: {} 2023-05-24T21:08:15.133 DEBUG:boto:Token: None 2023-05-24T21:08:15.133 DEBUG:boto:StringToSign: DELETE Wed, 24 May 2023 21:08:15 GMT /qskrps-12/obj-a1?versionId=u0vC7fwIKAfOAm.KhunLJ7iVMux6Tui 2023-05-24T21:08:15.134 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:n7MokQYrUC8RZ4YBXAIViekntt0= 2023-05-24T21:08:15.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:n7MokQYrUC8RZ4YBXAIViekntt0=', 'Content-Length': '0'} 2023-05-24T21:08:15.149 DEBUG:boto:Response headers: [('x-amz-version-id', 'u0vC7fwIKAfOAm.KhunLJ7iVMux6Tui'), ('x-amz-request-id', 'tx0000035f59bfe47d30db6-00646e7cbf-110d-a1'), ('Date', 'Wed, 24 May 2023 21:08:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:15.150 DEBUG:boto:path=/qskrps-12/obj-a1 2023-05-24T21:08:15.150 DEBUG:boto:auth_path=/qskrps-12/obj-a1 2023-05-24T21:08:15.150 DEBUG:boto:Method: PUT 2023-05-24T21:08:15.151 DEBUG:boto:Path: /qskrps-12/obj-a1 2023-05-24T21:08:15.151 DEBUG:boto:Data: 2023-05-24T21:08:15.151 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-24T21:08:15.152 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:08:15.152 DEBUG:boto:Port: 8000 2023-05-24T21:08:15.152 DEBUG:boto:Params: {} 2023-05-24T21:08:15.153 DEBUG:boto:Token: None 2023-05-24T21:08:15.153 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Wed, 24 May 2023 21:08:15 GMT /qskrps-12/obj-a1 2023-05-24T21:08:15.154 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:HIdh8EB5g8sgPdXGVcakxWogFKs= 2023-05-24T21:08:15.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:08:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:HIdh8EB5g8sgPdXGVcakxWogFKs='} 2023-05-24T21:08:15.166 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'WpGquVMEXPcrulBX0HMEV6hQmHD6lMQ'), ('x-amz-request-id', 'tx000007e6ca8f2a5def92a-00646e7cbf-110d-a1'), ('Date', 'Wed, 24 May 2023 21:08:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:15.167 DEBUG:rgw_multi.tests:version3 id=WpGquVMEXPcrulBX0HMEV6hQmHD6lMQ 2023-05-24T21:08:15.167 DEBUG:boto:path=/qskrps-12/obj-a1 2023-05-24T21:08:15.167 DEBUG:boto:auth_path=/qskrps-12/obj-a1 2023-05-24T21:08:15.168 DEBUG:boto:path=/qskrps-12/obj-a1?versionId=WpGquVMEXPcrulBX0HMEV6hQmHD6lMQ 2023-05-24T21:08:15.168 DEBUG:boto:auth_path=/qskrps-12/obj-a1?versionId=WpGquVMEXPcrulBX0HMEV6hQmHD6lMQ 2023-05-24T21:08:15.168 DEBUG:boto:Method: DELETE 2023-05-24T21:08:15.169 DEBUG:boto:Path: /qskrps-12/obj-a1?versionId=WpGquVMEXPcrulBX0HMEV6hQmHD6lMQ 2023-05-24T21:08:15.169 DEBUG:boto:Data: 2023-05-24T21:08:15.169 DEBUG:boto:Headers: {} 2023-05-24T21:08:15.169 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:08:15.170 DEBUG:boto:Port: 8000 2023-05-24T21:08:15.170 DEBUG:boto:Params: {} 2023-05-24T21:08:15.171 DEBUG:boto:Token: None 2023-05-24T21:08:15.171 DEBUG:boto:StringToSign: DELETE Wed, 24 May 2023 21:08:15 GMT /qskrps-12/obj-a1?versionId=WpGquVMEXPcrulBX0HMEV6hQmHD6lMQ 2023-05-24T21:08:15.171 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:B8zUqiOkvIs8D9Q1c8x1eCSbJCo= 2023-05-24T21:08:15.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:B8zUqiOkvIs8D9Q1c8x1eCSbJCo=', 'Content-Length': '0'} 2023-05-24T21:08:15.184 DEBUG:boto:Response headers: [('x-amz-version-id', 'WpGquVMEXPcrulBX0HMEV6hQmHD6lMQ'), ('x-amz-request-id', 'tx00000bfd70c727f8b0290-00646e7cbf-110d-a1'), ('Date', 'Wed, 24 May 2023 21:08:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:15.185 DEBUG:boto:path=/qskrps-12/ 2023-05-24T21:08:15.185 DEBUG:boto:auth_path=/qskrps-12/ 2023-05-24T21:08:15.186 DEBUG:boto:Method: HEAD 2023-05-24T21:08:15.186 DEBUG:boto:Path: /qskrps-12/ 2023-05-24T21:08:15.186 DEBUG:boto:Data: 2023-05-24T21:08:15.187 DEBUG:boto:Headers: {} 2023-05-24T21:08:15.187 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:08:15.187 DEBUG:boto:Port: 8001 2023-05-24T21:08:15.188 DEBUG:boto:Params: {} 2023-05-24T21:08:15.188 DEBUG:boto:Token: None 2023-05-24T21:08:15.189 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:08:15 GMT /qskrps-12/ 2023-05-24T21:08:15.189 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:6zXbh1b3OpLNxjUlfCXSoV+cJt0= 2023-05-24T21:08:15.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:6zXbh1b3OpLNxjUlfCXSoV+cJt0=', 'Content-Length': '0'} 2023-05-24T21:08:15.191 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e2163f093cb38da4-00646e7cbf-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:08:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:15.192 DEBUG:boto:path=/qskrps-12/obj-a2 2023-05-24T21:08:15.192 DEBUG:boto:auth_path=/qskrps-12/obj-a2 2023-05-24T21:08:15.193 DEBUG:boto:Method: PUT 2023-05-24T21:08:15.193 DEBUG:boto:Path: /qskrps-12/obj-a2 2023-05-24T21:08:15.193 DEBUG:boto:Data: 2023-05-24T21:08:15.194 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-24T21:08:15.194 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:08:15.194 DEBUG:boto:Port: 8001 2023-05-24T21:08:15.195 DEBUG:boto:Params: {} 2023-05-24T21:08:15.195 DEBUG:boto:Token: None 2023-05-24T21:08:15.196 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 24 May 2023 21:08:15 GMT /qskrps-12/obj-a2 2023-05-24T21:08:15.196 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:j5qSWogIZmWJeTOhIraHRAoX0M0= 2023-05-24T21:08:15.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:08:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:j5qSWogIZmWJeTOhIraHRAoX0M0='} 2023-05-24T21:08:15.215 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'PcWX5rYNtc410b9LwKxno3LHHZeAmhm'), ('x-amz-request-id', 'tx00000e3875d4989e14f44-00646e7cbf-1111-a2'), ('Date', 'Wed, 24 May 2023 21:08:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:15.216 DEBUG:rgw_multi.tests:version1 id=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:08:15.216 DEBUG:boto:path=/qskrps-12/obj-a2 2023-05-24T21:08:15.217 DEBUG:boto:auth_path=/qskrps-12/obj-a2 2023-05-24T21:08:15.217 DEBUG:boto:Method: PUT 2023-05-24T21:08:15.217 DEBUG:boto:Path: /qskrps-12/obj-a2 2023-05-24T21:08:15.218 DEBUG:boto:Data: 2023-05-24T21:08:15.218 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-24T21:08:15.218 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:08:15.219 DEBUG:boto:Port: 8001 2023-05-24T21:08:15.219 DEBUG:boto:Params: {} 2023-05-24T21:08:15.219 DEBUG:boto:Token: None 2023-05-24T21:08:15.220 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Wed, 24 May 2023 21:08:15 GMT /qskrps-12/obj-a2 2023-05-24T21:08:15.220 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ESHbG7zH3onFwXI8V69bVsiYAIc= 2023-05-24T21:08:15.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:08:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ESHbG7zH3onFwXI8V69bVsiYAIc='} 2023-05-24T21:08:15.241 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'uJRtaJiJcklx96y3hNC9Nf5BqHIP0to'), ('x-amz-request-id', 'tx0000081acc7dbe6396880-00646e7cbf-1111-a2'), ('Date', 'Wed, 24 May 2023 21:08:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:15.241 DEBUG:rgw_multi.tests:version2 id=uJRtaJiJcklx96y3hNC9Nf5BqHIP0to 2023-05-24T21:08:15.242 DEBUG:boto:path=/qskrps-12/obj-a2 2023-05-24T21:08:15.242 DEBUG:boto:auth_path=/qskrps-12/obj-a2 2023-05-24T21:08:15.242 DEBUG:boto:path=/qskrps-12/obj-a2?versionId=uJRtaJiJcklx96y3hNC9Nf5BqHIP0to 2023-05-24T21:08:15.243 DEBUG:boto:auth_path=/qskrps-12/obj-a2?versionId=uJRtaJiJcklx96y3hNC9Nf5BqHIP0to 2023-05-24T21:08:15.243 DEBUG:boto:Method: DELETE 2023-05-24T21:08:15.243 DEBUG:boto:Path: /qskrps-12/obj-a2?versionId=uJRtaJiJcklx96y3hNC9Nf5BqHIP0to 2023-05-24T21:08:15.244 DEBUG:boto:Data: 2023-05-24T21:08:15.244 DEBUG:boto:Headers: {} 2023-05-24T21:08:15.244 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:08:15.245 DEBUG:boto:Port: 8001 2023-05-24T21:08:15.245 DEBUG:boto:Params: {} 2023-05-24T21:08:15.245 DEBUG:boto:Token: None 2023-05-24T21:08:15.246 DEBUG:boto:StringToSign: DELETE Wed, 24 May 2023 21:08:15 GMT /qskrps-12/obj-a2?versionId=uJRtaJiJcklx96y3hNC9Nf5BqHIP0to 2023-05-24T21:08:15.246 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:DoshKLxqBjSPH1zkFrYEsDoXVys= 2023-05-24T21:08:15.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:DoshKLxqBjSPH1zkFrYEsDoXVys=', 'Content-Length': '0'} 2023-05-24T21:08:15.260 DEBUG:boto:Response headers: [('x-amz-version-id', 'uJRtaJiJcklx96y3hNC9Nf5BqHIP0to'), ('x-amz-request-id', 'tx00000f8698297cc5758e1-00646e7cbf-1111-a2'), ('Date', 'Wed, 24 May 2023 21:08:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:15.261 DEBUG:boto:path=/qskrps-12/obj-a2 2023-05-24T21:08:15.261 DEBUG:boto:auth_path=/qskrps-12/obj-a2 2023-05-24T21:08:15.262 DEBUG:boto:Method: PUT 2023-05-24T21:08:15.262 DEBUG:boto:Path: /qskrps-12/obj-a2 2023-05-24T21:08:15.262 DEBUG:boto:Data: 2023-05-24T21:08:15.263 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-24T21:08:15.263 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:08:15.263 DEBUG:boto:Port: 8001 2023-05-24T21:08:15.263 DEBUG:boto:Params: {} 2023-05-24T21:08:15.264 DEBUG:boto:Token: None 2023-05-24T21:08:15.264 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Wed, 24 May 2023 21:08:15 GMT /qskrps-12/obj-a2 2023-05-24T21:08:15.265 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:X/Fi0nsaYzURow/l6RaC0ncd6jM= 2023-05-24T21:08:15.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:08:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:X/Fi0nsaYzURow/l6RaC0ncd6jM='} 2023-05-24T21:08:15.309 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'b0mTn3VCFAHd1Bh3XHoAIBRpGlPP0rt'), ('x-amz-request-id', 'tx00000c77c89da648e05c5-00646e7cbf-1111-a2'), ('Date', 'Wed, 24 May 2023 21:08:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:15.309 DEBUG:rgw_multi.tests:version3 id=b0mTn3VCFAHd1Bh3XHoAIBRpGlPP0rt 2023-05-24T21:08:15.310 DEBUG:boto:path=/qskrps-12/obj-a2 2023-05-24T21:08:15.310 DEBUG:boto:auth_path=/qskrps-12/obj-a2 2023-05-24T21:08:15.310 DEBUG:boto:path=/qskrps-12/obj-a2?versionId=b0mTn3VCFAHd1Bh3XHoAIBRpGlPP0rt 2023-05-24T21:08:15.311 DEBUG:boto:auth_path=/qskrps-12/obj-a2?versionId=b0mTn3VCFAHd1Bh3XHoAIBRpGlPP0rt 2023-05-24T21:08:15.311 DEBUG:boto:Method: DELETE 2023-05-24T21:08:15.311 DEBUG:boto:Path: /qskrps-12/obj-a2?versionId=b0mTn3VCFAHd1Bh3XHoAIBRpGlPP0rt 2023-05-24T21:08:15.312 DEBUG:boto:Data: 2023-05-24T21:08:15.312 DEBUG:boto:Headers: {} 2023-05-24T21:08:15.312 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:08:15.313 DEBUG:boto:Port: 8001 2023-05-24T21:08:15.313 DEBUG:boto:Params: {} 2023-05-24T21:08:15.313 DEBUG:boto:Token: None 2023-05-24T21:08:15.314 DEBUG:boto:StringToSign: DELETE Wed, 24 May 2023 21:08:15 GMT /qskrps-12/obj-a2?versionId=b0mTn3VCFAHd1Bh3XHoAIBRpGlPP0rt 2023-05-24T21:08:15.314 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:xsBQfcRwaQynDZwf066E/DelcCA= 2023-05-24T21:08:15.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:xsBQfcRwaQynDZwf066E/DelcCA=', 'Content-Length': '0'} 2023-05-24T21:08:15.324 DEBUG:boto:Response headers: [('x-amz-version-id', 'b0mTn3VCFAHd1Bh3XHoAIBRpGlPP0rt'), ('x-amz-request-id', 'tx00000bea1fd80ff8f1b04-00646e7cbf-1111-a2'), ('Date', 'Wed, 24 May 2023 21:08:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:15.324 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=qskrps-11 2023-05-24T21:08:15.325 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:08:15.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:08:15.325 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-11 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:08:15.381 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:08:15.381 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:08:15.431 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:08:15.425+0000 7f2185271780 1 waiting for incremental sync to catch up: 2023-05-24T21:08:15.431 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000002.6.11, , , , , ] 2023-05-24T21:08:15.431 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000003.8.10, , , , 00000000010.25.6, ] 2023-05-24T21:08:20.432 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:08:20.426+0000 7f2185271780 1 waiting for incremental sync to catch up: 2023-05-24T21:08:20.433 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000002.6.11, , , , , ] 2023-05-24T21:08:20.433 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000003.8.10, , , , 00000000010.25.6, ] 2023-05-24T21:08:25.434 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:08:25.428+0000 7f2185271780 1 waiting for incremental sync to catch up: 2023-05-24T21:08:25.434 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000002.6.11, , , , , ] 2023-05-24T21:08:25.434 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000003.8.10, , , , 00000000010.25.6, ] 2023-05-24T21:08:30.435 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:08:30.429+0000 7f2185271780 1 waiting for incremental sync to catch up: 2023-05-24T21:08:30.435 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000002.6.11, , , , , ] 2023-05-24T21:08:30.435 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000003.8.10, , , , 00000000010.25.6, ] 2023-05-24T21:08:35.436 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:08:35.430+0000 7f2185271780 1 bucket sync caught up with source: 2023-05-24T21:08:35.437 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000003.8.10, , , , 00000000010.25.6, ] 2023-05-24T21:08:35.437 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000003.8.10, , , , 00000000010.25.6, ] 2023-05-24T21:08:35.437 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:08:35.430+0000 7f2185271780 0 bucket checkpoint complete 2023-05-24T21:08:35.452 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=qskrps-11 2023-05-24T21:08:35.453 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-11', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:08:35.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-11', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:08:35.454 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-11 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:08:35.515 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:08:35.516 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:08:35.579 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:08:35.573+0000 7f85c6bd1780 1 waiting for incremental sync to catch up: 2023-05-24T21:08:35.579 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000002.19.11, , , 00000000010.41.6, , ] 2023-05-24T21:08:35.579 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000002.19.11, , , 00000000010.41.6, 00000000002.26.11, ] 2023-05-24T21:08:40.580 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:08:40.575+0000 7f85c6bd1780 1 waiting for incremental sync to catch up: 2023-05-24T21:08:40.581 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000002.19.11, , , 00000000010.41.6, , ] 2023-05-24T21:08:40.581 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000002.19.11, , , 00000000010.41.6, 00000000002.26.11, ] 2023-05-24T21:08:45.582 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:08:45.576+0000 7f85c6bd1780 1 waiting for incremental sync to catch up: 2023-05-24T21:08:45.582 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000002.19.11, , , 00000000010.41.6, , ] 2023-05-24T21:08:45.583 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000002.19.11, , , 00000000010.41.6, 00000000002.26.11, ] 2023-05-24T21:08:50.583 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:08:50.576+0000 7f85c6bd1780 1 waiting for incremental sync to catch up: 2023-05-24T21:08:50.584 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000002.19.11, , , 00000000010.41.6, , ] 2023-05-24T21:08:50.584 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000002.19.11, , , 00000000010.41.6, 00000000002.26.11, ] 2023-05-24T21:08:55.585 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:08:55.578+0000 7f85c6bd1780 1 bucket sync caught up with source: 2023-05-24T21:08:55.585 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000002.19.11, , , 00000000010.41.6, 00000000002.26.11, ] 2023-05-24T21:08:55.585 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000002.19.11, , , 00000000010.41.6, 00000000002.26.11, ] 2023-05-24T21:08:55.585 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:08:55.578+0000 7f85c6bd1780 0 bucket checkpoint complete 2023-05-24T21:08:55.600 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qskrps-11 zones={a2, a1} 2023-05-24T21:08:55.601 DEBUG:boto:path=/qskrps-11/ 2023-05-24T21:08:55.602 DEBUG:boto:auth_path=/qskrps-11/ 2023-05-24T21:08:55.602 DEBUG:boto:Method: HEAD 2023-05-24T21:08:55.602 DEBUG:boto:Path: /qskrps-11/ 2023-05-24T21:08:55.603 DEBUG:boto:Data: 2023-05-24T21:08:55.603 DEBUG:boto:Headers: {} 2023-05-24T21:08:55.603 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:08:55.604 DEBUG:boto:Port: 8001 2023-05-24T21:08:55.604 DEBUG:boto:Params: {} 2023-05-24T21:08:55.605 DEBUG:boto:Token: None 2023-05-24T21:08:55.605 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:08:55 GMT /qskrps-11/ 2023-05-24T21:08:55.606 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:bAlXsmPjV7p6bVdnCfETRT6omV0= 2023-05-24T21:08:55.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:bAlXsmPjV7p6bVdnCfETRT6omV0=', 'Content-Length': '0'} 2023-05-24T21:08:55.609 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a02a864a0f0560de-00646e7ce7-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:08:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:55.609 DEBUG:boto:path=/qskrps-11/ 2023-05-24T21:08:55.610 DEBUG:boto:auth_path=/qskrps-11/ 2023-05-24T21:08:55.610 DEBUG:boto:Method: HEAD 2023-05-24T21:08:55.610 DEBUG:boto:Path: /qskrps-11/ 2023-05-24T21:08:55.611 DEBUG:boto:Data: 2023-05-24T21:08:55.611 DEBUG:boto:Headers: {} 2023-05-24T21:08:55.611 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:08:55.612 DEBUG:boto:Port: 8000 2023-05-24T21:08:55.612 DEBUG:boto:Params: {} 2023-05-24T21:08:55.612 DEBUG:boto:Token: None 2023-05-24T21:08:55.613 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:08:55 GMT /qskrps-11/ 2023-05-24T21:08:55.613 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:bAlXsmPjV7p6bVdnCfETRT6omV0= 2023-05-24T21:08:55.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:bAlXsmPjV7p6bVdnCfETRT6omV0=', 'Content-Length': '0'} 2023-05-24T21:08:55.616 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009f115df9de1e86f8-00646e7ce7-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:08:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:55.617 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-24T21:08:55.617 DEBUG:boto:path=/qskrps-11/ 2023-05-24T21:08:55.617 DEBUG:boto:auth_path=/qskrps-11/ 2023-05-24T21:08:55.618 DEBUG:boto:path=/qskrps-11/?versions&max-keys=999 2023-05-24T21:08:55.618 DEBUG:boto:auth_path=/qskrps-11/?versions&max-keys=999 2023-05-24T21:08:55.618 DEBUG:boto:Method: GET 2023-05-24T21:08:55.619 DEBUG:boto:Path: /qskrps-11/?versions&max-keys=999 2023-05-24T21:08:55.619 DEBUG:boto:Data: 2023-05-24T21:08:55.619 DEBUG:boto:Headers: {} 2023-05-24T21:08:55.620 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:08:55.620 DEBUG:boto:Port: 8001 2023-05-24T21:08:55.620 DEBUG:boto:Params: {} 2023-05-24T21:08:55.621 DEBUG:boto:Token: None 2023-05-24T21:08:55.621 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:08:55 GMT /qskrps-11/?versions 2023-05-24T21:08:55.622 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Demehhjf3xUB/nz/xBos6AFygFM= 2023-05-24T21:08:55.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Demehhjf3xUB/nz/xBos6AFygFM=', 'Content-Length': '0'} 2023-05-24T21:08:55.624 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a4aef76123348c9-00646e7ce7-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:08:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:55.625 DEBUG:boto:b'qskrps-11999falsedummyK0LgvK2trkm-uHhxLikH3sxyZSZDazNtrue2023-05-24T21:07:39.470Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a154a54vsjHFa46aVpWzOVMoAJ0PQA-.vtrue2023-05-24T21:08:14.786Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2YniDC1out7TFV1sI.TLfuqiPcVFd.5Strue2023-05-24T21:08:14.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:08:55.626 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-05-24T21:08:55.626 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-05-24T21:08:55.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-05-24T21:08:55.627 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-24T21:08:55.627 DEBUG:boto:path=/qskrps-11/ 2023-05-24T21:08:55.628 DEBUG:boto:auth_path=/qskrps-11/ 2023-05-24T21:08:55.628 DEBUG:boto:path=/qskrps-11/?versions&max-keys=999 2023-05-24T21:08:55.628 DEBUG:boto:auth_path=/qskrps-11/?versions&max-keys=999 2023-05-24T21:08:55.629 DEBUG:boto:Method: GET 2023-05-24T21:08:55.629 DEBUG:boto:Path: /qskrps-11/?versions&max-keys=999 2023-05-24T21:08:55.629 DEBUG:boto:Data: 2023-05-24T21:08:55.630 DEBUG:boto:Headers: {} 2023-05-24T21:08:55.630 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:08:55.630 DEBUG:boto:Port: 8000 2023-05-24T21:08:55.631 DEBUG:boto:Params: {} 2023-05-24T21:08:55.631 DEBUG:boto:Token: None 2023-05-24T21:08:55.631 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:08:55 GMT /qskrps-11/?versions 2023-05-24T21:08:55.632 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Demehhjf3xUB/nz/xBos6AFygFM= 2023-05-24T21:08:55.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Demehhjf3xUB/nz/xBos6AFygFM=', 'Content-Length': '0'} 2023-05-24T21:08:55.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d4c2993114aef40-00646e7ce7-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:08:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:55.635 DEBUG:boto:b'qskrps-11999falsedummyK0LgvK2trkm-uHhxLikH3sxyZSZDazNtrue2023-05-24T21:07:39.470Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a154a54vsjHFa46aVpWzOVMoAJ0PQA-.vtrue2023-05-24T21:08:14.786Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2YniDC1out7TFV1sI.TLfuqiPcVFd.5Strue2023-05-24T21:08:14.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:08:55.636 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-05-24T21:08:55.637 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-05-24T21:08:55.637 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-05-24T21:08:55.637 DEBUG:boto:path=/qskrps-11/ 2023-05-24T21:08:55.638 DEBUG:boto:auth_path=/qskrps-11/ 2023-05-24T21:08:55.638 DEBUG:boto:path=/qskrps-11/?versions&max-keys=999 2023-05-24T21:08:55.638 DEBUG:boto:auth_path=/qskrps-11/?versions&max-keys=999 2023-05-24T21:08:55.639 DEBUG:boto:Method: GET 2023-05-24T21:08:55.639 DEBUG:boto:Path: /qskrps-11/?versions&max-keys=999 2023-05-24T21:08:55.639 DEBUG:boto:Data: 2023-05-24T21:08:55.640 DEBUG:boto:Headers: {} 2023-05-24T21:08:55.640 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:08:55.640 DEBUG:boto:Port: 8001 2023-05-24T21:08:55.641 DEBUG:boto:Params: {} 2023-05-24T21:08:55.641 DEBUG:boto:Token: None 2023-05-24T21:08:55.642 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:08:55 GMT /qskrps-11/?versions 2023-05-24T21:08:55.642 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Demehhjf3xUB/nz/xBos6AFygFM= 2023-05-24T21:08:55.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Demehhjf3xUB/nz/xBos6AFygFM=', 'Content-Length': '0'} 2023-05-24T21:08:55.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009befaffbba593e56-00646e7ce7-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:08:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:55.645 DEBUG:boto:b'qskrps-11999falsedummyK0LgvK2trkm-uHhxLikH3sxyZSZDazNtrue2023-05-24T21:07:39.470Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a154a54vsjHFa46aVpWzOVMoAJ0PQA-.vtrue2023-05-24T21:08:14.786Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2YniDC1out7TFV1sI.TLfuqiPcVFd.5Strue2023-05-24T21:08:14.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:08:55.646 DEBUG:boto:path=/qskrps-11/ 2023-05-24T21:08:55.647 DEBUG:boto:auth_path=/qskrps-11/ 2023-05-24T21:08:55.647 DEBUG:boto:path=/qskrps-11/?versions&max-keys=999 2023-05-24T21:08:55.647 DEBUG:boto:auth_path=/qskrps-11/?versions&max-keys=999 2023-05-24T21:08:55.648 DEBUG:boto:Method: GET 2023-05-24T21:08:55.648 DEBUG:boto:Path: /qskrps-11/?versions&max-keys=999 2023-05-24T21:08:55.648 DEBUG:boto:Data: 2023-05-24T21:08:55.649 DEBUG:boto:Headers: {} 2023-05-24T21:08:55.649 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:08:55.649 DEBUG:boto:Port: 8000 2023-05-24T21:08:55.650 DEBUG:boto:Params: {} 2023-05-24T21:08:55.650 DEBUG:boto:Token: None 2023-05-24T21:08:55.651 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:08:55 GMT /qskrps-11/?versions 2023-05-24T21:08:55.651 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Demehhjf3xUB/nz/xBos6AFygFM= 2023-05-24T21:08:55.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Demehhjf3xUB/nz/xBos6AFygFM=', 'Content-Length': '0'} 2023-05-24T21:08:55.654 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd79458893a6a844-00646e7ce7-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:08:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:55.654 DEBUG:boto:b'qskrps-11999falsedummyK0LgvK2trkm-uHhxLikH3sxyZSZDazNtrue2023-05-24T21:07:39.470Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a154a54vsjHFa46aVpWzOVMoAJ0PQA-.vtrue2023-05-24T21:08:14.786Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2YniDC1out7TFV1sI.TLfuqiPcVFd.5Strue2023-05-24T21:08:14.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:08:55.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-05-24T21:08:55.655 DEBUG:boto:path=/qskrps-11/dummy 2023-05-24T21:08:55.656 DEBUG:boto:auth_path=/qskrps-11/dummy 2023-05-24T21:08:55.656 DEBUG:boto:path=/qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:08:55.656 DEBUG:boto:auth_path=/qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:08:55.657 DEBUG:boto:Method: GET 2023-05-24T21:08:55.657 DEBUG:boto:Path: /qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:08:55.657 DEBUG:boto:Data: 2023-05-24T21:08:55.658 DEBUG:boto:Headers: {} 2023-05-24T21:08:55.658 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:08:55.659 DEBUG:boto:Port: 8001 2023-05-24T21:08:55.659 DEBUG:boto:Params: {} 2023-05-24T21:08:55.659 DEBUG:boto:Token: None 2023-05-24T21:08:55.660 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:08:55 GMT /qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:08:55.660 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:rA/du2Pp6FKbOr1RhmtWoHZbYgo= 2023-05-24T21:08:55.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:rA/du2Pp6FKbOr1RhmtWoHZbYgo=', 'Content-Length': '0'} 2023-05-24T21:08:55.662 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:07:39 GMT'), ('x-amz-version-id', 'K0LgvK2trkm-uHhxLikH3sxyZSZDazN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009d487f5c9023d4c2-00646e7ce7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:08:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:55.663 DEBUG:boto:path=/qskrps-11/dummy 2023-05-24T21:08:55.663 DEBUG:boto:auth_path=/qskrps-11/dummy 2023-05-24T21:08:55.664 DEBUG:boto:path=/qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:08:55.664 DEBUG:boto:auth_path=/qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:08:55.664 DEBUG:boto:Method: GET 2023-05-24T21:08:55.665 DEBUG:boto:Path: /qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:08:55.665 DEBUG:boto:Data: 2023-05-24T21:08:55.666 DEBUG:boto:Headers: {} 2023-05-24T21:08:55.666 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:08:55.666 DEBUG:boto:Port: 8000 2023-05-24T21:08:55.667 DEBUG:boto:Params: {} 2023-05-24T21:08:55.667 DEBUG:boto:Token: None 2023-05-24T21:08:55.668 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:08:55 GMT /qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:08:55.668 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:rA/du2Pp6FKbOr1RhmtWoHZbYgo= 2023-05-24T21:08:55.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:rA/du2Pp6FKbOr1RhmtWoHZbYgo=', 'Content-Length': '0'} 2023-05-24T21:08:55.670 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:07:39 GMT'), ('x-amz-version-id', 'K0LgvK2trkm-uHhxLikH3sxyZSZDazN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000d9e282e5e47b88f-00646e7ce7-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:08:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:55.671 DEBUG:boto:path=/qskrps-11/dummy 2023-05-24T21:08:55.671 DEBUG:boto:auth_path=/qskrps-11/dummy 2023-05-24T21:08:55.671 DEBUG:boto:path=/qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:08:55.672 DEBUG:boto:auth_path=/qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:08:55.672 DEBUG:boto:Method: HEAD 2023-05-24T21:08:55.672 DEBUG:boto:Path: /qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:08:55.673 DEBUG:boto:Data: 2023-05-24T21:08:55.673 DEBUG:boto:Headers: {} 2023-05-24T21:08:55.673 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:08:55.674 DEBUG:boto:Port: 8001 2023-05-24T21:08:55.674 DEBUG:boto:Params: {} 2023-05-24T21:08:55.675 DEBUG:boto:Token: None 2023-05-24T21:08:55.675 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:08:55 GMT /qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:08:55.675 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:sf0K1fFEt+p34pSlWiar4e1bSWw= 2023-05-24T21:08:55.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:sf0K1fFEt+p34pSlWiar4e1bSWw=', 'Content-Length': '0'} 2023-05-24T21:08:55.678 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:07:39 GMT'), ('x-amz-version-id', 'K0LgvK2trkm-uHhxLikH3sxyZSZDazN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fb5eee49bd64935e-00646e7ce7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:08:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:55.678 DEBUG:boto:path=/qskrps-11/dummy 2023-05-24T21:08:55.678 DEBUG:boto:auth_path=/qskrps-11/dummy 2023-05-24T21:08:55.679 DEBUG:boto:path=/qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:08:55.679 DEBUG:boto:auth_path=/qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:08:55.680 DEBUG:boto:Method: HEAD 2023-05-24T21:08:55.680 DEBUG:boto:Path: /qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:08:55.680 DEBUG:boto:Data: 2023-05-24T21:08:55.681 DEBUG:boto:Headers: {} 2023-05-24T21:08:55.681 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:08:55.681 DEBUG:boto:Port: 8000 2023-05-24T21:08:55.682 DEBUG:boto:Params: {} 2023-05-24T21:08:55.682 DEBUG:boto:Token: None 2023-05-24T21:08:55.682 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:08:55 GMT /qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:08:55.683 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:sf0K1fFEt+p34pSlWiar4e1bSWw= 2023-05-24T21:08:55.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:sf0K1fFEt+p34pSlWiar4e1bSWw=', 'Content-Length': '0'} 2023-05-24T21:08:55.685 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:07:39 GMT'), ('x-amz-version-id', 'K0LgvK2trkm-uHhxLikH3sxyZSZDazN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c6e8e74c2db21c7c-00646e7ce7-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:08:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:55.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-05-24T21:08:55.686 DEBUG:boto:path=/qskrps-11/dummy 2023-05-24T21:08:55.686 DEBUG:boto:auth_path=/qskrps-11/dummy 2023-05-24T21:08:55.687 DEBUG:boto:path=/qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:08:55.687 DEBUG:boto:auth_path=/qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:08:55.687 DEBUG:boto:Method: GET 2023-05-24T21:08:55.688 DEBUG:boto:Path: /qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:08:55.688 DEBUG:boto:Data: 2023-05-24T21:08:55.688 DEBUG:boto:Headers: {} 2023-05-24T21:08:55.689 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:08:55.689 DEBUG:boto:Port: 8001 2023-05-24T21:08:55.689 DEBUG:boto:Params: {} 2023-05-24T21:08:55.690 DEBUG:boto:Token: None 2023-05-24T21:08:55.690 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:08:55 GMT /qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:08:55.691 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:rA/du2Pp6FKbOr1RhmtWoHZbYgo= 2023-05-24T21:08:55.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:rA/du2Pp6FKbOr1RhmtWoHZbYgo=', 'Content-Length': '0'} 2023-05-24T21:08:55.693 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:07:39 GMT'), ('x-amz-version-id', 'K0LgvK2trkm-uHhxLikH3sxyZSZDazN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ecf1dd21c215d55c-00646e7ce7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:08:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:55.693 DEBUG:boto:path=/qskrps-11/dummy 2023-05-24T21:08:55.694 DEBUG:boto:auth_path=/qskrps-11/dummy 2023-05-24T21:08:55.694 DEBUG:boto:path=/qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:08:55.694 DEBUG:boto:auth_path=/qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:08:55.695 DEBUG:boto:Method: GET 2023-05-24T21:08:55.695 DEBUG:boto:Path: /qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:08:55.696 DEBUG:boto:Data: 2023-05-24T21:08:55.696 DEBUG:boto:Headers: {} 2023-05-24T21:08:55.696 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:08:55.697 DEBUG:boto:Port: 8000 2023-05-24T21:08:55.697 DEBUG:boto:Params: {} 2023-05-24T21:08:55.697 DEBUG:boto:Token: None 2023-05-24T21:08:55.698 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:08:55 GMT /qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:08:55.698 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:rA/du2Pp6FKbOr1RhmtWoHZbYgo= 2023-05-24T21:08:55.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:rA/du2Pp6FKbOr1RhmtWoHZbYgo=', 'Content-Length': '0'} 2023-05-24T21:08:55.700 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:07:39 GMT'), ('x-amz-version-id', 'K0LgvK2trkm-uHhxLikH3sxyZSZDazN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005dcdf86c83aabbad-00646e7ce7-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:08:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:55.701 DEBUG:boto:path=/qskrps-11/ 2023-05-24T21:08:55.701 DEBUG:boto:auth_path=/qskrps-11/ 2023-05-24T21:08:55.702 DEBUG:boto:path=/qskrps-11/?prefix=dummy 2023-05-24T21:08:55.702 DEBUG:boto:auth_path=/qskrps-11/?prefix=dummy 2023-05-24T21:08:55.702 DEBUG:boto:Method: GET 2023-05-24T21:08:55.703 DEBUG:boto:Path: /qskrps-11/?prefix=dummy 2023-05-24T21:08:55.703 DEBUG:boto:Data: 2023-05-24T21:08:55.703 DEBUG:boto:Headers: {} 2023-05-24T21:08:55.704 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:08:55.704 DEBUG:boto:Port: 8001 2023-05-24T21:08:55.704 DEBUG:boto:Params: {} 2023-05-24T21:08:55.705 DEBUG:boto:Token: None 2023-05-24T21:08:55.705 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:08:55 GMT /qskrps-11/ 2023-05-24T21:08:55.706 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:RygqzBG1aZdHAgXXdSEaxZLoXd4= 2023-05-24T21:08:55.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:RygqzBG1aZdHAgXXdSEaxZLoXd4=', 'Content-Length': '0'} 2023-05-24T21:08:55.708 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7bd22a545692bf0-00646e7ce7-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:08:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:55.752 DEBUG:boto:b'qskrps-11dummy1000falsedummy2023-05-24T21:07:39.470Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:08:55.753 DEBUG:boto:path=/qskrps-11/ 2023-05-24T21:08:55.754 DEBUG:boto:auth_path=/qskrps-11/ 2023-05-24T21:08:55.754 DEBUG:boto:path=/qskrps-11/?prefix=dummy 2023-05-24T21:08:55.754 DEBUG:boto:auth_path=/qskrps-11/?prefix=dummy 2023-05-24T21:08:55.755 DEBUG:boto:Method: GET 2023-05-24T21:08:55.755 DEBUG:boto:Path: /qskrps-11/?prefix=dummy 2023-05-24T21:08:55.755 DEBUG:boto:Data: 2023-05-24T21:08:55.756 DEBUG:boto:Headers: {} 2023-05-24T21:08:55.756 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:08:55.756 DEBUG:boto:Port: 8000 2023-05-24T21:08:55.757 DEBUG:boto:Params: {} 2023-05-24T21:08:55.757 DEBUG:boto:Token: None 2023-05-24T21:08:55.758 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:08:55 GMT /qskrps-11/ 2023-05-24T21:08:55.758 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:RygqzBG1aZdHAgXXdSEaxZLoXd4= 2023-05-24T21:08:55.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:RygqzBG1aZdHAgXXdSEaxZLoXd4=', 'Content-Length': '0'} 2023-05-24T21:08:55.761 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe3146f8b193c5d9-00646e7ce7-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:08:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:55.762 DEBUG:boto:b'qskrps-11dummy1000falsedummy2023-05-24T21:07:39.470Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:08:55.762 DEBUG:boto:path=/qskrps-11/dummy 2023-05-24T21:08:55.763 DEBUG:boto:auth_path=/qskrps-11/dummy 2023-05-24T21:08:55.763 DEBUG:boto:Method: HEAD 2023-05-24T21:08:55.764 DEBUG:boto:Path: /qskrps-11/dummy 2023-05-24T21:08:55.764 DEBUG:boto:Data: 2023-05-24T21:08:55.764 DEBUG:boto:Headers: {} 2023-05-24T21:08:55.765 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:08:55.765 DEBUG:boto:Port: 8001 2023-05-24T21:08:55.765 DEBUG:boto:Params: {} 2023-05-24T21:08:55.766 DEBUG:boto:Token: None 2023-05-24T21:08:55.766 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:08:55 GMT /qskrps-11/dummy 2023-05-24T21:08:55.766 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:3rSYnQeD21HaSOq2Z3K1M930qbw= 2023-05-24T21:08:55.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:3rSYnQeD21HaSOq2Z3K1M930qbw=', 'Content-Length': '0'} 2023-05-24T21:08:55.769 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:07:39 GMT'), ('x-amz-version-id', 'K0LgvK2trkm-uHhxLikH3sxyZSZDazN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d09f7b97ea5be86b-00646e7ce7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:08:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:55.770 DEBUG:boto:path=/qskrps-11/dummy 2023-05-24T21:08:55.770 DEBUG:boto:auth_path=/qskrps-11/dummy 2023-05-24T21:08:55.771 DEBUG:boto:Method: HEAD 2023-05-24T21:08:55.771 DEBUG:boto:Path: /qskrps-11/dummy 2023-05-24T21:08:55.771 DEBUG:boto:Data: 2023-05-24T21:08:55.772 DEBUG:boto:Headers: {} 2023-05-24T21:08:55.772 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:08:55.772 DEBUG:boto:Port: 8000 2023-05-24T21:08:55.773 DEBUG:boto:Params: {} 2023-05-24T21:08:55.773 DEBUG:boto:Token: None 2023-05-24T21:08:55.773 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:08:55 GMT /qskrps-11/dummy 2023-05-24T21:08:55.774 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:3rSYnQeD21HaSOq2Z3K1M930qbw= 2023-05-24T21:08:55.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:3rSYnQeD21HaSOq2Z3K1M930qbw=', 'Content-Length': '0'} 2023-05-24T21:08:55.776 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:07:39 GMT'), ('x-amz-version-id', 'K0LgvK2trkm-uHhxLikH3sxyZSZDazN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000016e806535a4ee7c8-00646e7ce7-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:08:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:55.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-05-24T21:08:55.778 DEBUG:boto:path=/qskrps-11/dummy 2023-05-24T21:08:55.778 DEBUG:boto:auth_path=/qskrps-11/dummy 2023-05-24T21:08:55.778 DEBUG:boto:path=/qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:08:55.779 DEBUG:boto:auth_path=/qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:08:55.779 DEBUG:boto:Method: GET 2023-05-24T21:08:55.779 DEBUG:boto:Path: /qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:08:55.780 DEBUG:boto:Data: 2023-05-24T21:08:55.780 DEBUG:boto:Headers: {} 2023-05-24T21:08:55.780 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:08:55.781 DEBUG:boto:Port: 8001 2023-05-24T21:08:55.781 DEBUG:boto:Params: {} 2023-05-24T21:08:55.782 DEBUG:boto:Token: None 2023-05-24T21:08:55.782 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:08:55 GMT /qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:08:55.782 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:rA/du2Pp6FKbOr1RhmtWoHZbYgo= 2023-05-24T21:08:55.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:rA/du2Pp6FKbOr1RhmtWoHZbYgo=', 'Content-Length': '0'} 2023-05-24T21:08:55.785 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:07:39 GMT'), ('x-amz-version-id', 'K0LgvK2trkm-uHhxLikH3sxyZSZDazN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009992ddd68bfb4c92-00646e7ce7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:08:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:55.785 DEBUG:boto:path=/qskrps-11/dummy 2023-05-24T21:08:55.786 DEBUG:boto:auth_path=/qskrps-11/dummy 2023-05-24T21:08:55.786 DEBUG:boto:path=/qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:08:55.787 DEBUG:boto:auth_path=/qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:08:55.787 DEBUG:boto:Method: GET 2023-05-24T21:08:55.787 DEBUG:boto:Path: /qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:08:55.788 DEBUG:boto:Data: 2023-05-24T21:08:55.788 DEBUG:boto:Headers: {} 2023-05-24T21:08:55.788 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:08:55.789 DEBUG:boto:Port: 8000 2023-05-24T21:08:55.789 DEBUG:boto:Params: {} 2023-05-24T21:08:55.789 DEBUG:boto:Token: None 2023-05-24T21:08:55.790 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:08:55 GMT /qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:08:55.790 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:rA/du2Pp6FKbOr1RhmtWoHZbYgo= 2023-05-24T21:08:55.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:rA/du2Pp6FKbOr1RhmtWoHZbYgo=', 'Content-Length': '0'} 2023-05-24T21:08:55.793 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:07:39 GMT'), ('x-amz-version-id', 'K0LgvK2trkm-uHhxLikH3sxyZSZDazN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d156fdaf06f948cd-00646e7ce7-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:08:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:55.793 DEBUG:boto:path=/qskrps-11/ 2023-05-24T21:08:55.794 DEBUG:boto:auth_path=/qskrps-11/ 2023-05-24T21:08:55.794 DEBUG:boto:path=/qskrps-11/?prefix=dummy 2023-05-24T21:08:55.794 DEBUG:boto:auth_path=/qskrps-11/?prefix=dummy 2023-05-24T21:08:55.795 DEBUG:boto:Method: GET 2023-05-24T21:08:55.795 DEBUG:boto:Path: /qskrps-11/?prefix=dummy 2023-05-24T21:08:55.795 DEBUG:boto:Data: 2023-05-24T21:08:55.796 DEBUG:boto:Headers: {} 2023-05-24T21:08:55.796 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:08:55.796 DEBUG:boto:Port: 8001 2023-05-24T21:08:55.797 DEBUG:boto:Params: {} 2023-05-24T21:08:55.797 DEBUG:boto:Token: None 2023-05-24T21:08:55.798 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:08:55 GMT /qskrps-11/ 2023-05-24T21:08:55.798 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:RygqzBG1aZdHAgXXdSEaxZLoXd4= 2023-05-24T21:08:55.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:RygqzBG1aZdHAgXXdSEaxZLoXd4=', 'Content-Length': '0'} 2023-05-24T21:08:55.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ec63d1896933e82-00646e7ce7-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:08:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:55.844 DEBUG:boto:b'qskrps-11dummy1000falsedummy2023-05-24T21:07:39.470Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:08:55.845 DEBUG:boto:path=/qskrps-11/ 2023-05-24T21:08:55.846 DEBUG:boto:auth_path=/qskrps-11/ 2023-05-24T21:08:55.846 DEBUG:boto:path=/qskrps-11/?prefix=dummy 2023-05-24T21:08:55.846 DEBUG:boto:auth_path=/qskrps-11/?prefix=dummy 2023-05-24T21:08:55.847 DEBUG:boto:Method: GET 2023-05-24T21:08:55.847 DEBUG:boto:Path: /qskrps-11/?prefix=dummy 2023-05-24T21:08:55.847 DEBUG:boto:Data: 2023-05-24T21:08:55.848 DEBUG:boto:Headers: {} 2023-05-24T21:08:55.848 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:08:55.848 DEBUG:boto:Port: 8000 2023-05-24T21:08:55.849 DEBUG:boto:Params: {} 2023-05-24T21:08:55.849 DEBUG:boto:Token: None 2023-05-24T21:08:55.850 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:08:55 GMT /qskrps-11/ 2023-05-24T21:08:55.850 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:RygqzBG1aZdHAgXXdSEaxZLoXd4= 2023-05-24T21:08:55.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:RygqzBG1aZdHAgXXdSEaxZLoXd4=', 'Content-Length': '0'} 2023-05-24T21:08:55.853 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002484c6257fc72018-00646e7ce7-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:08:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:55.853 DEBUG:boto:b'qskrps-11dummy1000falsedummy2023-05-24T21:07:39.470Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:08:55.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-05-24T21:08:55.854 DEBUG:boto:path=/qskrps-11/obj-a1 2023-05-24T21:08:55.855 DEBUG:boto:auth_path=/qskrps-11/obj-a1 2023-05-24T21:08:55.855 DEBUG:boto:path=/qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:08:55.856 DEBUG:boto:auth_path=/qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:08:55.856 DEBUG:boto:Method: GET 2023-05-24T21:08:55.856 DEBUG:boto:Path: /qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:08:55.857 DEBUG:boto:Data: 2023-05-24T21:08:55.857 DEBUG:boto:Headers: {} 2023-05-24T21:08:55.857 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:08:55.858 DEBUG:boto:Port: 8001 2023-05-24T21:08:55.858 DEBUG:boto:Params: {} 2023-05-24T21:08:55.858 DEBUG:boto:Token: None 2023-05-24T21:08:55.859 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:08:55 GMT /qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:08:55.859 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:yKH495PB+ZgOLT8S1HSPpNbUp6E= 2023-05-24T21:08:55.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:yKH495PB+ZgOLT8S1HSPpNbUp6E=', 'Content-Length': '0'} 2023-05-24T21:08:55.862 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:08:14 GMT'), ('x-amz-version-id', '54a54vsjHFa46aVpWzOVMoAJ0PQA-.v'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000037f4e1ec7c66108e-00646e7ce7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:08:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:55.863 DEBUG:boto:path=/qskrps-11/obj-a1 2023-05-24T21:08:55.863 DEBUG:boto:auth_path=/qskrps-11/obj-a1 2023-05-24T21:08:55.863 DEBUG:boto:path=/qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:08:55.864 DEBUG:boto:auth_path=/qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:08:55.864 DEBUG:boto:Method: GET 2023-05-24T21:08:55.865 DEBUG:boto:Path: /qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:08:55.865 DEBUG:boto:Data: 2023-05-24T21:08:55.865 DEBUG:boto:Headers: {} 2023-05-24T21:08:55.866 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:08:55.866 DEBUG:boto:Port: 8000 2023-05-24T21:08:55.866 DEBUG:boto:Params: {} 2023-05-24T21:08:55.867 DEBUG:boto:Token: None 2023-05-24T21:08:55.867 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:08:55 GMT /qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:08:55.868 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:yKH495PB+ZgOLT8S1HSPpNbUp6E= 2023-05-24T21:08:55.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:yKH495PB+ZgOLT8S1HSPpNbUp6E=', 'Content-Length': '0'} 2023-05-24T21:08:55.870 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:08:14 GMT'), ('x-amz-version-id', '54a54vsjHFa46aVpWzOVMoAJ0PQA-.v'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000059944f6a4983d6c4-00646e7ce7-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:08:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:55.870 DEBUG:boto:path=/qskrps-11/obj-a1 2023-05-24T21:08:55.871 DEBUG:boto:auth_path=/qskrps-11/obj-a1 2023-05-24T21:08:55.871 DEBUG:boto:path=/qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:08:55.872 DEBUG:boto:auth_path=/qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:08:55.872 DEBUG:boto:Method: HEAD 2023-05-24T21:08:55.872 DEBUG:boto:Path: /qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:08:55.873 DEBUG:boto:Data: 2023-05-24T21:08:55.873 DEBUG:boto:Headers: {} 2023-05-24T21:08:55.873 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:08:55.874 DEBUG:boto:Port: 8001 2023-05-24T21:08:55.874 DEBUG:boto:Params: {} 2023-05-24T21:08:55.874 DEBUG:boto:Token: None 2023-05-24T21:08:55.875 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:08:55 GMT /qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:08:55.875 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Y/BcOfDxe5UoheVQuDpJPuPt1U8= 2023-05-24T21:08:55.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Y/BcOfDxe5UoheVQuDpJPuPt1U8=', 'Content-Length': '0'} 2023-05-24T21:08:55.877 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:08:14 GMT'), ('x-amz-version-id', '54a54vsjHFa46aVpWzOVMoAJ0PQA-.v'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003f2e3607690238c5-00646e7ce7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:08:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:55.878 DEBUG:boto:path=/qskrps-11/obj-a1 2023-05-24T21:08:55.878 DEBUG:boto:auth_path=/qskrps-11/obj-a1 2023-05-24T21:08:55.879 DEBUG:boto:path=/qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:08:55.879 DEBUG:boto:auth_path=/qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:08:55.879 DEBUG:boto:Method: HEAD 2023-05-24T21:08:55.880 DEBUG:boto:Path: /qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:08:55.880 DEBUG:boto:Data: 2023-05-24T21:08:55.881 DEBUG:boto:Headers: {} 2023-05-24T21:08:55.881 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:08:55.881 DEBUG:boto:Port: 8000 2023-05-24T21:08:55.882 DEBUG:boto:Params: {} 2023-05-24T21:08:55.882 DEBUG:boto:Token: None 2023-05-24T21:08:55.882 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:08:55 GMT /qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:08:55.883 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Y/BcOfDxe5UoheVQuDpJPuPt1U8= 2023-05-24T21:08:55.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Y/BcOfDxe5UoheVQuDpJPuPt1U8=', 'Content-Length': '0'} 2023-05-24T21:08:55.885 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:08:14 GMT'), ('x-amz-version-id', '54a54vsjHFa46aVpWzOVMoAJ0PQA-.v'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004c0f69985f7ee432-00646e7ce7-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:08:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:55.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-05-24T21:08:55.886 DEBUG:boto:path=/qskrps-11/obj-a1 2023-05-24T21:08:55.886 DEBUG:boto:auth_path=/qskrps-11/obj-a1 2023-05-24T21:08:55.887 DEBUG:boto:path=/qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:08:55.887 DEBUG:boto:auth_path=/qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:08:55.887 DEBUG:boto:Method: GET 2023-05-24T21:08:55.888 DEBUG:boto:Path: /qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:08:55.888 DEBUG:boto:Data: 2023-05-24T21:08:55.888 DEBUG:boto:Headers: {} 2023-05-24T21:08:55.889 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:08:55.889 DEBUG:boto:Port: 8001 2023-05-24T21:08:55.889 DEBUG:boto:Params: {} 2023-05-24T21:08:55.890 DEBUG:boto:Token: None 2023-05-24T21:08:55.890 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:08:55 GMT /qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:08:55.891 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:yKH495PB+ZgOLT8S1HSPpNbUp6E= 2023-05-24T21:08:55.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:yKH495PB+ZgOLT8S1HSPpNbUp6E=', 'Content-Length': '0'} 2023-05-24T21:08:55.893 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:08:14 GMT'), ('x-amz-version-id', '54a54vsjHFa46aVpWzOVMoAJ0PQA-.v'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c6c0a9908a7951a0-00646e7ce7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:08:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:55.894 DEBUG:boto:path=/qskrps-11/obj-a1 2023-05-24T21:08:55.894 DEBUG:boto:auth_path=/qskrps-11/obj-a1 2023-05-24T21:08:55.894 DEBUG:boto:path=/qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:08:55.895 DEBUG:boto:auth_path=/qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:08:55.895 DEBUG:boto:Method: GET 2023-05-24T21:08:55.896 DEBUG:boto:Path: /qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:08:55.896 DEBUG:boto:Data: 2023-05-24T21:08:55.896 DEBUG:boto:Headers: {} 2023-05-24T21:08:55.897 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:08:55.897 DEBUG:boto:Port: 8000 2023-05-24T21:08:55.897 DEBUG:boto:Params: {} 2023-05-24T21:08:55.898 DEBUG:boto:Token: None 2023-05-24T21:08:55.898 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:08:55 GMT /qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:08:55.899 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:yKH495PB+ZgOLT8S1HSPpNbUp6E= 2023-05-24T21:08:55.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:yKH495PB+ZgOLT8S1HSPpNbUp6E=', 'Content-Length': '0'} 2023-05-24T21:08:55.901 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:08:14 GMT'), ('x-amz-version-id', '54a54vsjHFa46aVpWzOVMoAJ0PQA-.v'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000134b2bcf0b25ec46-00646e7ce7-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:08:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:55.902 DEBUG:boto:path=/qskrps-11/ 2023-05-24T21:08:55.902 DEBUG:boto:auth_path=/qskrps-11/ 2023-05-24T21:08:55.903 DEBUG:boto:path=/qskrps-11/?prefix=obj-a1 2023-05-24T21:08:55.903 DEBUG:boto:auth_path=/qskrps-11/?prefix=obj-a1 2023-05-24T21:08:55.903 DEBUG:boto:Method: GET 2023-05-24T21:08:55.904 DEBUG:boto:Path: /qskrps-11/?prefix=obj-a1 2023-05-24T21:08:55.904 DEBUG:boto:Data: 2023-05-24T21:08:55.904 DEBUG:boto:Headers: {} 2023-05-24T21:08:55.905 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:08:55.905 DEBUG:boto:Port: 8001 2023-05-24T21:08:55.906 DEBUG:boto:Params: {} 2023-05-24T21:08:55.906 DEBUG:boto:Token: None 2023-05-24T21:08:55.906 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:08:55 GMT /qskrps-11/ 2023-05-24T21:08:55.907 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:RygqzBG1aZdHAgXXdSEaxZLoXd4= 2023-05-24T21:08:55.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:RygqzBG1aZdHAgXXdSEaxZLoXd4=', 'Content-Length': '0'} 2023-05-24T21:08:55.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a4d4ed309b11b65-00646e7ce7-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:08:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:55.910 DEBUG:boto:b'qskrps-11obj-a11000falseobj-a12023-05-24T21:08:14.786Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:08:55.911 DEBUG:boto:path=/qskrps-11/ 2023-05-24T21:08:55.911 DEBUG:boto:auth_path=/qskrps-11/ 2023-05-24T21:08:55.912 DEBUG:boto:path=/qskrps-11/?prefix=obj-a1 2023-05-24T21:08:55.912 DEBUG:boto:auth_path=/qskrps-11/?prefix=obj-a1 2023-05-24T21:08:55.912 DEBUG:boto:Method: GET 2023-05-24T21:08:55.913 DEBUG:boto:Path: /qskrps-11/?prefix=obj-a1 2023-05-24T21:08:55.913 DEBUG:boto:Data: 2023-05-24T21:08:55.914 DEBUG:boto:Headers: {} 2023-05-24T21:08:55.914 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:08:55.914 DEBUG:boto:Port: 8000 2023-05-24T21:08:55.915 DEBUG:boto:Params: {} 2023-05-24T21:08:55.915 DEBUG:boto:Token: None 2023-05-24T21:08:55.915 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:08:55 GMT /qskrps-11/ 2023-05-24T21:08:55.916 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:RygqzBG1aZdHAgXXdSEaxZLoXd4= 2023-05-24T21:08:55.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:RygqzBG1aZdHAgXXdSEaxZLoXd4=', 'Content-Length': '0'} 2023-05-24T21:08:55.918 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000717662b160bb186f-00646e7ce7-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:08:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:55.960 DEBUG:boto:b'qskrps-11obj-a11000falseobj-a12023-05-24T21:08:14.786Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:08:55.961 DEBUG:boto:path=/qskrps-11/obj-a1 2023-05-24T21:08:55.962 DEBUG:boto:auth_path=/qskrps-11/obj-a1 2023-05-24T21:08:55.962 DEBUG:boto:Method: HEAD 2023-05-24T21:08:55.962 DEBUG:boto:Path: /qskrps-11/obj-a1 2023-05-24T21:08:55.963 DEBUG:boto:Data: 2023-05-24T21:08:55.963 DEBUG:boto:Headers: {} 2023-05-24T21:08:55.963 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:08:55.964 DEBUG:boto:Port: 8001 2023-05-24T21:08:55.964 DEBUG:boto:Params: {} 2023-05-24T21:08:55.964 DEBUG:boto:Token: None 2023-05-24T21:08:55.965 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:08:55 GMT /qskrps-11/obj-a1 2023-05-24T21:08:55.965 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:IKy+RDQ89Svkh3qpmrIpB7NShsM= 2023-05-24T21:08:55.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:IKy+RDQ89Svkh3qpmrIpB7NShsM=', 'Content-Length': '0'} 2023-05-24T21:08:55.968 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:08:14 GMT'), ('x-amz-version-id', '54a54vsjHFa46aVpWzOVMoAJ0PQA-.v'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f94b90253a9ead30-00646e7ce7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:08:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:55.968 DEBUG:boto:path=/qskrps-11/obj-a1 2023-05-24T21:08:55.969 DEBUG:boto:auth_path=/qskrps-11/obj-a1 2023-05-24T21:08:55.969 DEBUG:boto:Method: HEAD 2023-05-24T21:08:55.970 DEBUG:boto:Path: /qskrps-11/obj-a1 2023-05-24T21:08:55.970 DEBUG:boto:Data: 2023-05-24T21:08:55.970 DEBUG:boto:Headers: {} 2023-05-24T21:08:55.971 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:08:55.971 DEBUG:boto:Port: 8000 2023-05-24T21:08:55.971 DEBUG:boto:Params: {} 2023-05-24T21:08:55.972 DEBUG:boto:Token: None 2023-05-24T21:08:55.972 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:08:55 GMT /qskrps-11/obj-a1 2023-05-24T21:08:55.972 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:IKy+RDQ89Svkh3qpmrIpB7NShsM= 2023-05-24T21:08:55.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:IKy+RDQ89Svkh3qpmrIpB7NShsM=', 'Content-Length': '0'} 2023-05-24T21:08:55.975 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:08:14 GMT'), ('x-amz-version-id', '54a54vsjHFa46aVpWzOVMoAJ0PQA-.v'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008540346f5f66494e-00646e7ce7-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:08:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:55.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-05-24T21:08:55.976 DEBUG:boto:path=/qskrps-11/obj-a1 2023-05-24T21:08:55.977 DEBUG:boto:auth_path=/qskrps-11/obj-a1 2023-05-24T21:08:55.977 DEBUG:boto:path=/qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:08:55.977 DEBUG:boto:auth_path=/qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:08:55.978 DEBUG:boto:Method: GET 2023-05-24T21:08:55.978 DEBUG:boto:Path: /qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:08:55.978 DEBUG:boto:Data: 2023-05-24T21:08:55.979 DEBUG:boto:Headers: {} 2023-05-24T21:08:55.979 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:08:55.979 DEBUG:boto:Port: 8001 2023-05-24T21:08:55.980 DEBUG:boto:Params: {} 2023-05-24T21:08:55.980 DEBUG:boto:Token: None 2023-05-24T21:08:55.981 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:08:55 GMT /qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:08:55.981 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:yKH495PB+ZgOLT8S1HSPpNbUp6E= 2023-05-24T21:08:55.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:yKH495PB+ZgOLT8S1HSPpNbUp6E=', 'Content-Length': '0'} 2023-05-24T21:08:55.983 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:08:14 GMT'), ('x-amz-version-id', '54a54vsjHFa46aVpWzOVMoAJ0PQA-.v'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009b4df1df474fb3f3-00646e7ce7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:08:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:55.984 DEBUG:boto:path=/qskrps-11/obj-a1 2023-05-24T21:08:55.984 DEBUG:boto:auth_path=/qskrps-11/obj-a1 2023-05-24T21:08:55.985 DEBUG:boto:path=/qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:08:55.985 DEBUG:boto:auth_path=/qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:08:55.985 DEBUG:boto:Method: GET 2023-05-24T21:08:55.986 DEBUG:boto:Path: /qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:08:55.986 DEBUG:boto:Data: 2023-05-24T21:08:55.986 DEBUG:boto:Headers: {} 2023-05-24T21:08:55.987 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:08:55.987 DEBUG:boto:Port: 8000 2023-05-24T21:08:55.988 DEBUG:boto:Params: {} 2023-05-24T21:08:55.988 DEBUG:boto:Token: None 2023-05-24T21:08:55.988 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:08:55 GMT /qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:08:55.989 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:yKH495PB+ZgOLT8S1HSPpNbUp6E= 2023-05-24T21:08:55.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:yKH495PB+ZgOLT8S1HSPpNbUp6E=', 'Content-Length': '0'} 2023-05-24T21:08:55.991 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:08:14 GMT'), ('x-amz-version-id', '54a54vsjHFa46aVpWzOVMoAJ0PQA-.v'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008725693635a961c6-00646e7ce7-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:08:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:55.992 DEBUG:boto:path=/qskrps-11/ 2023-05-24T21:08:55.992 DEBUG:boto:auth_path=/qskrps-11/ 2023-05-24T21:08:55.992 DEBUG:boto:path=/qskrps-11/?prefix=obj-a1 2023-05-24T21:08:55.993 DEBUG:boto:auth_path=/qskrps-11/?prefix=obj-a1 2023-05-24T21:08:55.993 DEBUG:boto:Method: GET 2023-05-24T21:08:55.994 DEBUG:boto:Path: /qskrps-11/?prefix=obj-a1 2023-05-24T21:08:55.994 DEBUG:boto:Data: 2023-05-24T21:08:55.994 DEBUG:boto:Headers: {} 2023-05-24T21:08:55.995 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:08:55.995 DEBUG:boto:Port: 8001 2023-05-24T21:08:55.995 DEBUG:boto:Params: {} 2023-05-24T21:08:55.996 DEBUG:boto:Token: None 2023-05-24T21:08:55.996 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:08:55 GMT /qskrps-11/ 2023-05-24T21:08:55.997 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:RygqzBG1aZdHAgXXdSEaxZLoXd4= 2023-05-24T21:08:55.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:RygqzBG1aZdHAgXXdSEaxZLoXd4=', 'Content-Length': '0'} 2023-05-24T21:08:55.999 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044a5262b5a46566f-00646e7ce7-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:08:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:56.000 DEBUG:boto:b'qskrps-11obj-a11000falseobj-a12023-05-24T21:08:14.786Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:08:56.000 DEBUG:boto:path=/qskrps-11/ 2023-05-24T21:08:56.001 DEBUG:boto:auth_path=/qskrps-11/ 2023-05-24T21:08:56.001 DEBUG:boto:path=/qskrps-11/?prefix=obj-a1 2023-05-24T21:08:56.002 DEBUG:boto:auth_path=/qskrps-11/?prefix=obj-a1 2023-05-24T21:08:56.002 DEBUG:boto:Method: GET 2023-05-24T21:08:56.002 DEBUG:boto:Path: /qskrps-11/?prefix=obj-a1 2023-05-24T21:08:56.003 DEBUG:boto:Data: 2023-05-24T21:08:56.003 DEBUG:boto:Headers: {} 2023-05-24T21:08:56.003 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:08:56.004 DEBUG:boto:Port: 8000 2023-05-24T21:08:56.004 DEBUG:boto:Params: {} 2023-05-24T21:08:56.004 DEBUG:boto:Token: None 2023-05-24T21:08:56.005 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:08:56 GMT /qskrps-11/ 2023-05-24T21:08:56.005 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:YmBxvgrOAVNEWvd5fnFyuuhgdLc= 2023-05-24T21:08:56.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:56 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:YmBxvgrOAVNEWvd5fnFyuuhgdLc=', 'Content-Length': '0'} 2023-05-24T21:08:56.008 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1c2714762301f49-00646e7ce8-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:08:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:56.009 DEBUG:boto:b'qskrps-11obj-a11000falseobj-a12023-05-24T21:08:14.786Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:08:56.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-05-24T21:08:56.010 DEBUG:boto:path=/qskrps-11/obj-a2 2023-05-24T21:08:56.010 DEBUG:boto:auth_path=/qskrps-11/obj-a2 2023-05-24T21:08:56.010 DEBUG:boto:path=/qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:08:56.011 DEBUG:boto:auth_path=/qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:08:56.011 DEBUG:boto:Method: GET 2023-05-24T21:08:56.011 DEBUG:boto:Path: /qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:08:56.012 DEBUG:boto:Data: 2023-05-24T21:08:56.012 DEBUG:boto:Headers: {} 2023-05-24T21:08:56.012 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:08:56.013 DEBUG:boto:Port: 8001 2023-05-24T21:08:56.013 DEBUG:boto:Params: {} 2023-05-24T21:08:56.014 DEBUG:boto:Token: None 2023-05-24T21:08:56.014 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:08:56 GMT /qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:08:56.014 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:YIAKQqclANRvfR1X0omE+Lfnp6A= 2023-05-24T21:08:56.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:56 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:YIAKQqclANRvfR1X0omE+Lfnp6A=', 'Content-Length': '0'} 2023-05-24T21:08:56.017 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:08:14 GMT'), ('x-amz-version-id', 'YniDC1out7TFV1sI.TLfuqiPcVFd.5S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b7b0ae4d024692dd-00646e7ce8-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:08:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:56.017 DEBUG:boto:path=/qskrps-11/obj-a2 2023-05-24T21:08:56.018 DEBUG:boto:auth_path=/qskrps-11/obj-a2 2023-05-24T21:08:56.018 DEBUG:boto:path=/qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:08:56.018 DEBUG:boto:auth_path=/qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:08:56.019 DEBUG:boto:Method: GET 2023-05-24T21:08:56.019 DEBUG:boto:Path: /qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:08:56.019 DEBUG:boto:Data: 2023-05-24T21:08:56.020 DEBUG:boto:Headers: {} 2023-05-24T21:08:56.020 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:08:56.021 DEBUG:boto:Port: 8000 2023-05-24T21:08:56.021 DEBUG:boto:Params: {} 2023-05-24T21:08:56.021 DEBUG:boto:Token: None 2023-05-24T21:08:56.022 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:08:56 GMT /qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:08:56.022 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:YIAKQqclANRvfR1X0omE+Lfnp6A= 2023-05-24T21:08:56.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:56 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:YIAKQqclANRvfR1X0omE+Lfnp6A=', 'Content-Length': '0'} 2023-05-24T21:08:56.024 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:08:14 GMT'), ('x-amz-version-id', 'YniDC1out7TFV1sI.TLfuqiPcVFd.5S'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000656e28ff30d406fe-00646e7ce8-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:08:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:56.025 DEBUG:boto:path=/qskrps-11/obj-a2 2023-05-24T21:08:56.025 DEBUG:boto:auth_path=/qskrps-11/obj-a2 2023-05-24T21:08:56.026 DEBUG:boto:path=/qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:08:56.026 DEBUG:boto:auth_path=/qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:08:56.026 DEBUG:boto:Method: HEAD 2023-05-24T21:08:56.027 DEBUG:boto:Path: /qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:08:56.027 DEBUG:boto:Data: 2023-05-24T21:08:56.027 DEBUG:boto:Headers: {} 2023-05-24T21:08:56.028 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:08:56.028 DEBUG:boto:Port: 8001 2023-05-24T21:08:56.028 DEBUG:boto:Params: {} 2023-05-24T21:08:56.029 DEBUG:boto:Token: None 2023-05-24T21:08:56.029 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:08:56 GMT /qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:08:56.030 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:9w7wVcgC6GmIgJ+dRAOZYtk+k1s= 2023-05-24T21:08:56.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:56 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:9w7wVcgC6GmIgJ+dRAOZYtk+k1s=', 'Content-Length': '0'} 2023-05-24T21:08:56.032 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:08:14 GMT'), ('x-amz-version-id', 'YniDC1out7TFV1sI.TLfuqiPcVFd.5S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fbd001263abadd0f-00646e7ce8-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:08:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:56.032 DEBUG:boto:path=/qskrps-11/obj-a2 2023-05-24T21:08:56.033 DEBUG:boto:auth_path=/qskrps-11/obj-a2 2023-05-24T21:08:56.033 DEBUG:boto:path=/qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:08:56.033 DEBUG:boto:auth_path=/qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:08:56.034 DEBUG:boto:Method: HEAD 2023-05-24T21:08:56.034 DEBUG:boto:Path: /qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:08:56.034 DEBUG:boto:Data: 2023-05-24T21:08:56.035 DEBUG:boto:Headers: {} 2023-05-24T21:08:56.035 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:08:56.035 DEBUG:boto:Port: 8000 2023-05-24T21:08:56.036 DEBUG:boto:Params: {} 2023-05-24T21:08:56.036 DEBUG:boto:Token: None 2023-05-24T21:08:56.037 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:08:56 GMT /qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:08:56.037 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:9w7wVcgC6GmIgJ+dRAOZYtk+k1s= 2023-05-24T21:08:56.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:56 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:9w7wVcgC6GmIgJ+dRAOZYtk+k1s=', 'Content-Length': '0'} 2023-05-24T21:08:56.039 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:08:14 GMT'), ('x-amz-version-id', 'YniDC1out7TFV1sI.TLfuqiPcVFd.5S'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000013a23d383239f19f-00646e7ce8-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:08:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:56.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-05-24T21:08:56.040 DEBUG:boto:path=/qskrps-11/obj-a2 2023-05-24T21:08:56.040 DEBUG:boto:auth_path=/qskrps-11/obj-a2 2023-05-24T21:08:56.041 DEBUG:boto:path=/qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:08:56.041 DEBUG:boto:auth_path=/qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:08:56.041 DEBUG:boto:Method: GET 2023-05-24T21:08:56.042 DEBUG:boto:Path: /qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:08:56.042 DEBUG:boto:Data: 2023-05-24T21:08:56.043 DEBUG:boto:Headers: {} 2023-05-24T21:08:56.043 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:08:56.044 DEBUG:boto:Port: 8001 2023-05-24T21:08:56.044 DEBUG:boto:Params: {} 2023-05-24T21:08:56.045 DEBUG:boto:Token: None 2023-05-24T21:08:56.045 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:08:56 GMT /qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:08:56.045 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:YIAKQqclANRvfR1X0omE+Lfnp6A= 2023-05-24T21:08:56.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:56 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:YIAKQqclANRvfR1X0omE+Lfnp6A=', 'Content-Length': '0'} 2023-05-24T21:08:56.048 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:08:14 GMT'), ('x-amz-version-id', 'YniDC1out7TFV1sI.TLfuqiPcVFd.5S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000057366d3c8fa2e90a-00646e7ce8-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:08:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:56.092 DEBUG:boto:path=/qskrps-11/obj-a2 2023-05-24T21:08:56.093 DEBUG:boto:auth_path=/qskrps-11/obj-a2 2023-05-24T21:08:56.093 DEBUG:boto:path=/qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:08:56.094 DEBUG:boto:auth_path=/qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:08:56.094 DEBUG:boto:Method: GET 2023-05-24T21:08:56.094 DEBUG:boto:Path: /qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:08:56.095 DEBUG:boto:Data: 2023-05-24T21:08:56.095 DEBUG:boto:Headers: {} 2023-05-24T21:08:56.095 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:08:56.096 DEBUG:boto:Port: 8000 2023-05-24T21:08:56.096 DEBUG:boto:Params: {} 2023-05-24T21:08:56.097 DEBUG:boto:Token: None 2023-05-24T21:08:56.097 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:08:56 GMT /qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:08:56.097 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:YIAKQqclANRvfR1X0omE+Lfnp6A= 2023-05-24T21:08:56.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:56 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:YIAKQqclANRvfR1X0omE+Lfnp6A=', 'Content-Length': '0'} 2023-05-24T21:08:56.100 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:08:14 GMT'), ('x-amz-version-id', 'YniDC1out7TFV1sI.TLfuqiPcVFd.5S'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009e7a2e59550ba078-00646e7ce8-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:08:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:56.101 DEBUG:boto:path=/qskrps-11/ 2023-05-24T21:08:56.101 DEBUG:boto:auth_path=/qskrps-11/ 2023-05-24T21:08:56.101 DEBUG:boto:path=/qskrps-11/?prefix=obj-a2 2023-05-24T21:08:56.102 DEBUG:boto:auth_path=/qskrps-11/?prefix=obj-a2 2023-05-24T21:08:56.102 DEBUG:boto:Method: GET 2023-05-24T21:08:56.102 DEBUG:boto:Path: /qskrps-11/?prefix=obj-a2 2023-05-24T21:08:56.103 DEBUG:boto:Data: 2023-05-24T21:08:56.103 DEBUG:boto:Headers: {} 2023-05-24T21:08:56.103 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:08:56.104 DEBUG:boto:Port: 8001 2023-05-24T21:08:56.104 DEBUG:boto:Params: {} 2023-05-24T21:08:56.105 DEBUG:boto:Token: None 2023-05-24T21:08:56.105 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:08:56 GMT /qskrps-11/ 2023-05-24T21:08:56.105 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:YmBxvgrOAVNEWvd5fnFyuuhgdLc= 2023-05-24T21:08:56.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:56 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:YmBxvgrOAVNEWvd5fnFyuuhgdLc=', 'Content-Length': '0'} 2023-05-24T21:08:56.108 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008bb1a0edc6b467fb-00646e7ce8-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:08:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:56.109 DEBUG:boto:b'qskrps-11obj-a21000falseobj-a22023-05-24T21:08:14.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:08:56.110 DEBUG:boto:path=/qskrps-11/ 2023-05-24T21:08:56.110 DEBUG:boto:auth_path=/qskrps-11/ 2023-05-24T21:08:56.110 DEBUG:boto:path=/qskrps-11/?prefix=obj-a2 2023-05-24T21:08:56.111 DEBUG:boto:auth_path=/qskrps-11/?prefix=obj-a2 2023-05-24T21:08:56.111 DEBUG:boto:Method: GET 2023-05-24T21:08:56.111 DEBUG:boto:Path: /qskrps-11/?prefix=obj-a2 2023-05-24T21:08:56.112 DEBUG:boto:Data: 2023-05-24T21:08:56.112 DEBUG:boto:Headers: {} 2023-05-24T21:08:56.113 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:08:56.113 DEBUG:boto:Port: 8000 2023-05-24T21:08:56.113 DEBUG:boto:Params: {} 2023-05-24T21:08:56.114 DEBUG:boto:Token: None 2023-05-24T21:08:56.114 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:08:56 GMT /qskrps-11/ 2023-05-24T21:08:56.114 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:YmBxvgrOAVNEWvd5fnFyuuhgdLc= 2023-05-24T21:08:56.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:56 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:YmBxvgrOAVNEWvd5fnFyuuhgdLc=', 'Content-Length': '0'} 2023-05-24T21:08:56.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2edf2f77c20ad5c-00646e7ce8-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:08:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:56.117 DEBUG:boto:b'qskrps-11obj-a21000falseobj-a22023-05-24T21:08:14.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:08:56.118 DEBUG:boto:path=/qskrps-11/obj-a2 2023-05-24T21:08:56.118 DEBUG:boto:auth_path=/qskrps-11/obj-a2 2023-05-24T21:08:56.119 DEBUG:boto:Method: HEAD 2023-05-24T21:08:56.119 DEBUG:boto:Path: /qskrps-11/obj-a2 2023-05-24T21:08:56.120 DEBUG:boto:Data: 2023-05-24T21:08:56.120 DEBUG:boto:Headers: {} 2023-05-24T21:08:56.120 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:08:56.121 DEBUG:boto:Port: 8001 2023-05-24T21:08:56.121 DEBUG:boto:Params: {} 2023-05-24T21:08:56.121 DEBUG:boto:Token: None 2023-05-24T21:08:56.122 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:08:56 GMT /qskrps-11/obj-a2 2023-05-24T21:08:56.122 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:UEFmWoEb/ZvxDAMYkzx0awYSVM8= 2023-05-24T21:08:56.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:56 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:UEFmWoEb/ZvxDAMYkzx0awYSVM8=', 'Content-Length': '0'} 2023-05-24T21:08:56.125 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:08:14 GMT'), ('x-amz-version-id', 'YniDC1out7TFV1sI.TLfuqiPcVFd.5S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c82948a8fc20d9a7-00646e7ce8-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:08:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:56.125 DEBUG:boto:path=/qskrps-11/obj-a2 2023-05-24T21:08:56.126 DEBUG:boto:auth_path=/qskrps-11/obj-a2 2023-05-24T21:08:56.126 DEBUG:boto:Method: HEAD 2023-05-24T21:08:56.126 DEBUG:boto:Path: /qskrps-11/obj-a2 2023-05-24T21:08:56.127 DEBUG:boto:Data: 2023-05-24T21:08:56.127 DEBUG:boto:Headers: {} 2023-05-24T21:08:56.127 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:08:56.128 DEBUG:boto:Port: 8000 2023-05-24T21:08:56.128 DEBUG:boto:Params: {} 2023-05-24T21:08:56.128 DEBUG:boto:Token: None 2023-05-24T21:08:56.129 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:08:56 GMT /qskrps-11/obj-a2 2023-05-24T21:08:56.129 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:UEFmWoEb/ZvxDAMYkzx0awYSVM8= 2023-05-24T21:08:56.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:56 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:UEFmWoEb/ZvxDAMYkzx0awYSVM8=', 'Content-Length': '0'} 2023-05-24T21:08:56.132 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:08:14 GMT'), ('x-amz-version-id', 'YniDC1out7TFV1sI.TLfuqiPcVFd.5S'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f623ee74096c1824-00646e7ce8-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:08:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:56.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-05-24T21:08:56.133 DEBUG:boto:path=/qskrps-11/obj-a2 2023-05-24T21:08:56.133 DEBUG:boto:auth_path=/qskrps-11/obj-a2 2023-05-24T21:08:56.134 DEBUG:boto:path=/qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:08:56.134 DEBUG:boto:auth_path=/qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:08:56.134 DEBUG:boto:Method: GET 2023-05-24T21:08:56.135 DEBUG:boto:Path: /qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:08:56.135 DEBUG:boto:Data: 2023-05-24T21:08:56.135 DEBUG:boto:Headers: {} 2023-05-24T21:08:56.136 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:08:56.136 DEBUG:boto:Port: 8001 2023-05-24T21:08:56.137 DEBUG:boto:Params: {} 2023-05-24T21:08:56.137 DEBUG:boto:Token: None 2023-05-24T21:08:56.138 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:08:56 GMT /qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:08:56.138 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:YIAKQqclANRvfR1X0omE+Lfnp6A= 2023-05-24T21:08:56.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:56 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:YIAKQqclANRvfR1X0omE+Lfnp6A=', 'Content-Length': '0'} 2023-05-24T21:08:56.140 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:08:14 GMT'), ('x-amz-version-id', 'YniDC1out7TFV1sI.TLfuqiPcVFd.5S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000024262c71fbca67d-00646e7ce8-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:08:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:56.184 DEBUG:boto:path=/qskrps-11/obj-a2 2023-05-24T21:08:56.185 DEBUG:boto:auth_path=/qskrps-11/obj-a2 2023-05-24T21:08:56.185 DEBUG:boto:path=/qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:08:56.186 DEBUG:boto:auth_path=/qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:08:56.186 DEBUG:boto:Method: GET 2023-05-24T21:08:56.186 DEBUG:boto:Path: /qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:08:56.187 DEBUG:boto:Data: 2023-05-24T21:08:56.187 DEBUG:boto:Headers: {} 2023-05-24T21:08:56.188 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:08:56.188 DEBUG:boto:Port: 8000 2023-05-24T21:08:56.188 DEBUG:boto:Params: {} 2023-05-24T21:08:56.189 DEBUG:boto:Token: None 2023-05-24T21:08:56.189 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:08:56 GMT /qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:08:56.190 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:YIAKQqclANRvfR1X0omE+Lfnp6A= 2023-05-24T21:08:56.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:56 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:YIAKQqclANRvfR1X0omE+Lfnp6A=', 'Content-Length': '0'} 2023-05-24T21:08:56.192 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:08:14 GMT'), ('x-amz-version-id', 'YniDC1out7TFV1sI.TLfuqiPcVFd.5S'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000014ffab6b669fe6c3-00646e7ce8-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:08:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:56.193 DEBUG:boto:path=/qskrps-11/ 2023-05-24T21:08:56.193 DEBUG:boto:auth_path=/qskrps-11/ 2023-05-24T21:08:56.193 DEBUG:boto:path=/qskrps-11/?prefix=obj-a2 2023-05-24T21:08:56.194 DEBUG:boto:auth_path=/qskrps-11/?prefix=obj-a2 2023-05-24T21:08:56.194 DEBUG:boto:Method: GET 2023-05-24T21:08:56.194 DEBUG:boto:Path: /qskrps-11/?prefix=obj-a2 2023-05-24T21:08:56.195 DEBUG:boto:Data: 2023-05-24T21:08:56.195 DEBUG:boto:Headers: {} 2023-05-24T21:08:56.196 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:08:56.196 DEBUG:boto:Port: 8001 2023-05-24T21:08:56.196 DEBUG:boto:Params: {} 2023-05-24T21:08:56.197 DEBUG:boto:Token: None 2023-05-24T21:08:56.197 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:08:56 GMT /qskrps-11/ 2023-05-24T21:08:56.198 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:YmBxvgrOAVNEWvd5fnFyuuhgdLc= 2023-05-24T21:08:56.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:56 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:YmBxvgrOAVNEWvd5fnFyuuhgdLc=', 'Content-Length': '0'} 2023-05-24T21:08:56.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d9042be5ddb268a-00646e7ce8-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:08:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:56.201 DEBUG:boto:b'qskrps-11obj-a21000falseobj-a22023-05-24T21:08:14.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:08:56.202 DEBUG:boto:path=/qskrps-11/ 2023-05-24T21:08:56.202 DEBUG:boto:auth_path=/qskrps-11/ 2023-05-24T21:08:56.203 DEBUG:boto:path=/qskrps-11/?prefix=obj-a2 2023-05-24T21:08:56.203 DEBUG:boto:auth_path=/qskrps-11/?prefix=obj-a2 2023-05-24T21:08:56.203 DEBUG:boto:Method: GET 2023-05-24T21:08:56.204 DEBUG:boto:Path: /qskrps-11/?prefix=obj-a2 2023-05-24T21:08:56.204 DEBUG:boto:Data: 2023-05-24T21:08:56.204 DEBUG:boto:Headers: {} 2023-05-24T21:08:56.205 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:08:56.205 DEBUG:boto:Port: 8000 2023-05-24T21:08:56.206 DEBUG:boto:Params: {} 2023-05-24T21:08:56.206 DEBUG:boto:Token: None 2023-05-24T21:08:56.206 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:08:56 GMT /qskrps-11/ 2023-05-24T21:08:56.207 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:YmBxvgrOAVNEWvd5fnFyuuhgdLc= 2023-05-24T21:08:56.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:08:56 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:YmBxvgrOAVNEWvd5fnFyuuhgdLc=', 'Content-Length': '0'} 2023-05-24T21:08:56.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009efd73ff27a1b4fb-00646e7ce8-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:08:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:08:56.210 DEBUG:boto:b'qskrps-11obj-a21000falseobj-a22023-05-24T21:08:14.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:08:56.211 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qskrps-11 zones={a2, a1} 2023-05-24T21:08:56.211 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=qskrps-12 2023-05-24T21:08:56.211 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-12', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:08:56.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-12', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:08:56.212 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-12 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:08:56.275 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:08:56.276 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:08:56.333 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:08:56.326+0000 7fa611bbc780 1 bucket sync caught up with source: 2023-05-24T21:08:56.334 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000002.17.11, , , 00000000002.5.11, 00000000010.23.6, ] 2023-05-24T21:08:56.334 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000002.17.11, , , 00000000002.5.11, 00000000010.23.6, ] 2023-05-24T21:08:56.334 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:08:56.326+0000 7fa611bbc780 0 bucket checkpoint complete 2023-05-24T21:08:56.357 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=qskrps-12 2023-05-24T21:08:56.357 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:08:56.358 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:08:56.358 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-12 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:08:56.414 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:08:56.415 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:08:56.479 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:08:56.472+0000 7fa8fe280780 1 waiting for incremental sync to catch up: 2023-05-24T21:08:56.480 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000003.36.10, , , 00000000010.28.6, , ] 2023-05-24T21:08:56.480 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000003.36.10, , , 00000000010.28.6, 00000000002.19.11, ] 2023-05-24T21:09:01.481 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:09:01.474+0000 7fa8fe280780 1 waiting for incremental sync to catch up: 2023-05-24T21:09:01.481 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000003.36.10, , , 00000000010.28.6, , ] 2023-05-24T21:09:01.481 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000003.36.10, , , 00000000010.28.6, 00000000002.19.11, ] 2023-05-24T21:09:06.482 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:09:06.475+0000 7fa8fe280780 1 waiting for incremental sync to catch up: 2023-05-24T21:09:06.482 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000003.36.10, , , 00000000010.28.6, , ] 2023-05-24T21:09:06.483 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000003.36.10, , , 00000000010.28.6, 00000000002.19.11, ] 2023-05-24T21:09:11.483 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:09:11.477+0000 7fa8fe280780 1 bucket sync caught up with source: 2023-05-24T21:09:11.484 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000003.36.10, , , 00000000010.28.6, 00000000002.19.11, ] 2023-05-24T21:09:11.484 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000003.36.10, , , 00000000010.28.6, 00000000002.19.11, ] 2023-05-24T21:09:11.484 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:09:11.477+0000 7fa8fe280780 0 bucket checkpoint complete 2023-05-24T21:09:11.499 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qskrps-12 zones={a2, a1} 2023-05-24T21:09:11.499 DEBUG:boto:path=/qskrps-12/ 2023-05-24T21:09:11.500 DEBUG:boto:auth_path=/qskrps-12/ 2023-05-24T21:09:11.500 DEBUG:boto:Method: HEAD 2023-05-24T21:09:11.501 DEBUG:boto:Path: /qskrps-12/ 2023-05-24T21:09:11.501 DEBUG:boto:Data: 2023-05-24T21:09:11.501 DEBUG:boto:Headers: {} 2023-05-24T21:09:11.502 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:11.502 DEBUG:boto:Port: 8001 2023-05-24T21:09:11.502 DEBUG:boto:Params: {} 2023-05-24T21:09:11.503 DEBUG:boto:Token: None 2023-05-24T21:09:11.503 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:11 GMT /qskrps-12/ 2023-05-24T21:09:11.504 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:HmBWxO8WuuPHao4ymcrLWmz+VGQ= 2023-05-24T21:09:11.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:HmBWxO8WuuPHao4ymcrLWmz+VGQ=', 'Content-Length': '0'} 2023-05-24T21:09:11.506 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c97a1e83bb0e0d0-00646e7cf7-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:09:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:11.507 DEBUG:boto:path=/qskrps-12/ 2023-05-24T21:09:11.507 DEBUG:boto:auth_path=/qskrps-12/ 2023-05-24T21:09:11.508 DEBUG:boto:Method: HEAD 2023-05-24T21:09:11.508 DEBUG:boto:Path: /qskrps-12/ 2023-05-24T21:09:11.508 DEBUG:boto:Data: 2023-05-24T21:09:11.509 DEBUG:boto:Headers: {} 2023-05-24T21:09:11.509 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:11.509 DEBUG:boto:Port: 8000 2023-05-24T21:09:11.510 DEBUG:boto:Params: {} 2023-05-24T21:09:11.510 DEBUG:boto:Token: None 2023-05-24T21:09:11.510 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:11 GMT /qskrps-12/ 2023-05-24T21:09:11.511 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:HmBWxO8WuuPHao4ymcrLWmz+VGQ= 2023-05-24T21:09:11.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:HmBWxO8WuuPHao4ymcrLWmz+VGQ=', 'Content-Length': '0'} 2023-05-24T21:09:11.513 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e5f45da0848fefe7-00646e7cf7-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:09:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:11.514 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-24T21:09:11.514 DEBUG:boto:path=/qskrps-12/ 2023-05-24T21:09:11.515 DEBUG:boto:auth_path=/qskrps-12/ 2023-05-24T21:09:11.515 DEBUG:boto:path=/qskrps-12/?versions&max-keys=999 2023-05-24T21:09:11.515 DEBUG:boto:auth_path=/qskrps-12/?versions&max-keys=999 2023-05-24T21:09:11.516 DEBUG:boto:Method: GET 2023-05-24T21:09:11.516 DEBUG:boto:Path: /qskrps-12/?versions&max-keys=999 2023-05-24T21:09:11.516 DEBUG:boto:Data: 2023-05-24T21:09:11.517 DEBUG:boto:Headers: {} 2023-05-24T21:09:11.517 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:11.517 DEBUG:boto:Port: 8001 2023-05-24T21:09:11.518 DEBUG:boto:Params: {} 2023-05-24T21:09:11.518 DEBUG:boto:Token: None 2023-05-24T21:09:11.519 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:11 GMT /qskrps-12/?versions 2023-05-24T21:09:11.519 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:pXSa/yIy0Ikmo62Z9CneaEE6f1I= 2023-05-24T21:09:11.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:pXSa/yIy0Ikmo62Z9CneaEE6f1I=', 'Content-Length': '0'} 2023-05-24T21:09:11.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080effa3c126d5924-00646e7cf7-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:11.522 DEBUG:boto:b'qskrps-12999falsedummykm8zCAetA1TzFWDKQlSimNmDeLQdejHtrue2023-05-24T21:07:54.624Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiVtrue2023-05-24T21:08:15.067Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2PcWX5rYNtc410b9LwKxno3LHHZeAmhmtrue2023-05-24T21:08:15.193Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:11.523 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-05-24T21:09:11.523 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-05-24T21:09:11.524 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-05-24T21:09:11.524 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-24T21:09:11.524 DEBUG:boto:path=/qskrps-12/ 2023-05-24T21:09:11.525 DEBUG:boto:auth_path=/qskrps-12/ 2023-05-24T21:09:11.525 DEBUG:boto:path=/qskrps-12/?versions&max-keys=999 2023-05-24T21:09:11.525 DEBUG:boto:auth_path=/qskrps-12/?versions&max-keys=999 2023-05-24T21:09:11.526 DEBUG:boto:Method: GET 2023-05-24T21:09:11.526 DEBUG:boto:Path: /qskrps-12/?versions&max-keys=999 2023-05-24T21:09:11.526 DEBUG:boto:Data: 2023-05-24T21:09:11.527 DEBUG:boto:Headers: {} 2023-05-24T21:09:11.527 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:11.527 DEBUG:boto:Port: 8000 2023-05-24T21:09:11.528 DEBUG:boto:Params: {} 2023-05-24T21:09:11.528 DEBUG:boto:Token: None 2023-05-24T21:09:11.529 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:11 GMT /qskrps-12/?versions 2023-05-24T21:09:11.529 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:pXSa/yIy0Ikmo62Z9CneaEE6f1I= 2023-05-24T21:09:11.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:pXSa/yIy0Ikmo62Z9CneaEE6f1I=', 'Content-Length': '0'} 2023-05-24T21:09:11.532 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005300bffb60768cef-00646e7cf7-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:11.532 DEBUG:boto:b'qskrps-12999falsedummykm8zCAetA1TzFWDKQlSimNmDeLQdejHtrue2023-05-24T21:07:54.624Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiVtrue2023-05-24T21:08:15.067Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2PcWX5rYNtc410b9LwKxno3LHHZeAmhmtrue2023-05-24T21:08:15.193Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:11.533 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-05-24T21:09:11.533 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-05-24T21:09:11.533 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-05-24T21:09:11.534 DEBUG:boto:path=/qskrps-12/ 2023-05-24T21:09:11.534 DEBUG:boto:auth_path=/qskrps-12/ 2023-05-24T21:09:11.535 DEBUG:boto:path=/qskrps-12/?versions&max-keys=999 2023-05-24T21:09:11.535 DEBUG:boto:auth_path=/qskrps-12/?versions&max-keys=999 2023-05-24T21:09:11.535 DEBUG:boto:Method: GET 2023-05-24T21:09:11.536 DEBUG:boto:Path: /qskrps-12/?versions&max-keys=999 2023-05-24T21:09:11.536 DEBUG:boto:Data: 2023-05-24T21:09:11.536 DEBUG:boto:Headers: {} 2023-05-24T21:09:11.537 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:11.537 DEBUG:boto:Port: 8001 2023-05-24T21:09:11.537 DEBUG:boto:Params: {} 2023-05-24T21:09:11.538 DEBUG:boto:Token: None 2023-05-24T21:09:11.538 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:11 GMT /qskrps-12/?versions 2023-05-24T21:09:11.538 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:pXSa/yIy0Ikmo62Z9CneaEE6f1I= 2023-05-24T21:09:11.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:pXSa/yIy0Ikmo62Z9CneaEE6f1I=', 'Content-Length': '0'} 2023-05-24T21:09:11.541 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3bce2ab7c9b03cf-00646e7cf7-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:11.542 DEBUG:boto:b'qskrps-12999falsedummykm8zCAetA1TzFWDKQlSimNmDeLQdejHtrue2023-05-24T21:07:54.624Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiVtrue2023-05-24T21:08:15.067Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2PcWX5rYNtc410b9LwKxno3LHHZeAmhmtrue2023-05-24T21:08:15.193Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:11.542 DEBUG:boto:path=/qskrps-12/ 2023-05-24T21:09:11.543 DEBUG:boto:auth_path=/qskrps-12/ 2023-05-24T21:09:11.543 DEBUG:boto:path=/qskrps-12/?versions&max-keys=999 2023-05-24T21:09:11.543 DEBUG:boto:auth_path=/qskrps-12/?versions&max-keys=999 2023-05-24T21:09:11.544 DEBUG:boto:Method: GET 2023-05-24T21:09:11.544 DEBUG:boto:Path: /qskrps-12/?versions&max-keys=999 2023-05-24T21:09:11.545 DEBUG:boto:Data: 2023-05-24T21:09:11.545 DEBUG:boto:Headers: {} 2023-05-24T21:09:11.545 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:11.546 DEBUG:boto:Port: 8000 2023-05-24T21:09:11.546 DEBUG:boto:Params: {} 2023-05-24T21:09:11.546 DEBUG:boto:Token: None 2023-05-24T21:09:11.547 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:11 GMT /qskrps-12/?versions 2023-05-24T21:09:11.547 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:pXSa/yIy0Ikmo62Z9CneaEE6f1I= 2023-05-24T21:09:11.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:pXSa/yIy0Ikmo62Z9CneaEE6f1I=', 'Content-Length': '0'} 2023-05-24T21:09:11.550 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2a899ac2b62d849-00646e7cf7-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:11.550 DEBUG:boto:b'qskrps-12999falsedummykm8zCAetA1TzFWDKQlSimNmDeLQdejHtrue2023-05-24T21:07:54.624Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiVtrue2023-05-24T21:08:15.067Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2PcWX5rYNtc410b9LwKxno3LHHZeAmhmtrue2023-05-24T21:08:15.193Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:11.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-05-24T21:09:11.551 DEBUG:boto:path=/qskrps-12/dummy 2023-05-24T21:09:11.552 DEBUG:boto:auth_path=/qskrps-12/dummy 2023-05-24T21:09:11.552 DEBUG:boto:path=/qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:11.553 DEBUG:boto:auth_path=/qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:11.553 DEBUG:boto:Method: GET 2023-05-24T21:09:11.553 DEBUG:boto:Path: /qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:11.554 DEBUG:boto:Data: 2023-05-24T21:09:11.554 DEBUG:boto:Headers: {} 2023-05-24T21:09:11.554 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:11.555 DEBUG:boto:Port: 8001 2023-05-24T21:09:11.555 DEBUG:boto:Params: {} 2023-05-24T21:09:11.555 DEBUG:boto:Token: None 2023-05-24T21:09:11.556 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:11 GMT /qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:11.556 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:T98UbwBvlXeIjI0F5qASjLnNWeE= 2023-05-24T21:09:11.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:T98UbwBvlXeIjI0F5qASjLnNWeE=', 'Content-Length': '0'} 2023-05-24T21:09:11.559 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:07:54 GMT'), ('x-amz-version-id', 'km8zCAetA1TzFWDKQlSimNmDeLQdejH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008c70b5bf4597a1ce-00646e7cf7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:11.559 DEBUG:boto:path=/qskrps-12/dummy 2023-05-24T21:09:11.560 DEBUG:boto:auth_path=/qskrps-12/dummy 2023-05-24T21:09:11.560 DEBUG:boto:path=/qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:11.560 DEBUG:boto:auth_path=/qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:11.561 DEBUG:boto:Method: GET 2023-05-24T21:09:11.561 DEBUG:boto:Path: /qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:11.561 DEBUG:boto:Data: 2023-05-24T21:09:11.562 DEBUG:boto:Headers: {} 2023-05-24T21:09:11.562 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:11.562 DEBUG:boto:Port: 8000 2023-05-24T21:09:11.563 DEBUG:boto:Params: {} 2023-05-24T21:09:11.563 DEBUG:boto:Token: None 2023-05-24T21:09:11.564 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:11 GMT /qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:11.564 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:T98UbwBvlXeIjI0F5qASjLnNWeE= 2023-05-24T21:09:11.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:T98UbwBvlXeIjI0F5qASjLnNWeE=', 'Content-Length': '0'} 2023-05-24T21:09:11.566 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:07:54 GMT'), ('x-amz-version-id', 'km8zCAetA1TzFWDKQlSimNmDeLQdejH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d812798ae98cb14d-00646e7cf7-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:11.567 DEBUG:boto:path=/qskrps-12/dummy 2023-05-24T21:09:11.567 DEBUG:boto:auth_path=/qskrps-12/dummy 2023-05-24T21:09:11.568 DEBUG:boto:path=/qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:11.568 DEBUG:boto:auth_path=/qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:11.568 DEBUG:boto:Method: HEAD 2023-05-24T21:09:11.569 DEBUG:boto:Path: /qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:11.569 DEBUG:boto:Data: 2023-05-24T21:09:11.569 DEBUG:boto:Headers: {} 2023-05-24T21:09:11.570 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:11.570 DEBUG:boto:Port: 8001 2023-05-24T21:09:11.570 DEBUG:boto:Params: {} 2023-05-24T21:09:11.571 DEBUG:boto:Token: None 2023-05-24T21:09:11.571 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:11 GMT /qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:11.571 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ALTe6zB9DeRLrJPRKGlz0Z/y9uk= 2023-05-24T21:09:11.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ALTe6zB9DeRLrJPRKGlz0Z/y9uk=', 'Content-Length': '0'} 2023-05-24T21:09:11.573 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:07:54 GMT'), ('x-amz-version-id', 'km8zCAetA1TzFWDKQlSimNmDeLQdejH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e2d78094591aa198-00646e7cf7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:11.574 DEBUG:boto:path=/qskrps-12/dummy 2023-05-24T21:09:11.574 DEBUG:boto:auth_path=/qskrps-12/dummy 2023-05-24T21:09:11.575 DEBUG:boto:path=/qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:11.575 DEBUG:boto:auth_path=/qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:11.575 DEBUG:boto:Method: HEAD 2023-05-24T21:09:11.576 DEBUG:boto:Path: /qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:11.576 DEBUG:boto:Data: 2023-05-24T21:09:11.576 DEBUG:boto:Headers: {} 2023-05-24T21:09:11.577 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:11.577 DEBUG:boto:Port: 8000 2023-05-24T21:09:11.577 DEBUG:boto:Params: {} 2023-05-24T21:09:11.578 DEBUG:boto:Token: None 2023-05-24T21:09:11.578 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:11 GMT /qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:11.579 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ALTe6zB9DeRLrJPRKGlz0Z/y9uk= 2023-05-24T21:09:11.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ALTe6zB9DeRLrJPRKGlz0Z/y9uk=', 'Content-Length': '0'} 2023-05-24T21:09:11.581 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:07:54 GMT'), ('x-amz-version-id', 'km8zCAetA1TzFWDKQlSimNmDeLQdejH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000023a57bfcfc48f5fe-00646e7cf7-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:11.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-05-24T21:09:11.582 DEBUG:boto:path=/qskrps-12/dummy 2023-05-24T21:09:11.582 DEBUG:boto:auth_path=/qskrps-12/dummy 2023-05-24T21:09:11.582 DEBUG:boto:path=/qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:11.583 DEBUG:boto:auth_path=/qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:11.583 DEBUG:boto:Method: GET 2023-05-24T21:09:11.583 DEBUG:boto:Path: /qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:11.584 DEBUG:boto:Data: 2023-05-24T21:09:11.584 DEBUG:boto:Headers: {} 2023-05-24T21:09:11.585 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:11.585 DEBUG:boto:Port: 8001 2023-05-24T21:09:11.585 DEBUG:boto:Params: {} 2023-05-24T21:09:11.586 DEBUG:boto:Token: None 2023-05-24T21:09:11.586 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:11 GMT /qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:11.587 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:T98UbwBvlXeIjI0F5qASjLnNWeE= 2023-05-24T21:09:11.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:T98UbwBvlXeIjI0F5qASjLnNWeE=', 'Content-Length': '0'} 2023-05-24T21:09:11.589 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:07:54 GMT'), ('x-amz-version-id', 'km8zCAetA1TzFWDKQlSimNmDeLQdejH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000005f06813bcc2651-00646e7cf7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:11.589 DEBUG:boto:path=/qskrps-12/dummy 2023-05-24T21:09:11.590 DEBUG:boto:auth_path=/qskrps-12/dummy 2023-05-24T21:09:11.590 DEBUG:boto:path=/qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:11.590 DEBUG:boto:auth_path=/qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:11.591 DEBUG:boto:Method: GET 2023-05-24T21:09:11.591 DEBUG:boto:Path: /qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:11.591 DEBUG:boto:Data: 2023-05-24T21:09:11.592 DEBUG:boto:Headers: {} 2023-05-24T21:09:11.592 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:11.592 DEBUG:boto:Port: 8000 2023-05-24T21:09:11.593 DEBUG:boto:Params: {} 2023-05-24T21:09:11.593 DEBUG:boto:Token: None 2023-05-24T21:09:11.594 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:11 GMT /qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:11.594 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:T98UbwBvlXeIjI0F5qASjLnNWeE= 2023-05-24T21:09:11.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:T98UbwBvlXeIjI0F5qASjLnNWeE=', 'Content-Length': '0'} 2023-05-24T21:09:11.596 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:07:54 GMT'), ('x-amz-version-id', 'km8zCAetA1TzFWDKQlSimNmDeLQdejH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a7ea1cbad66276d5-00646e7cf7-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:11.597 DEBUG:boto:path=/qskrps-12/ 2023-05-24T21:09:11.597 DEBUG:boto:auth_path=/qskrps-12/ 2023-05-24T21:09:11.597 DEBUG:boto:path=/qskrps-12/?prefix=dummy 2023-05-24T21:09:11.598 DEBUG:boto:auth_path=/qskrps-12/?prefix=dummy 2023-05-24T21:09:11.598 DEBUG:boto:Method: GET 2023-05-24T21:09:11.598 DEBUG:boto:Path: /qskrps-12/?prefix=dummy 2023-05-24T21:09:11.599 DEBUG:boto:Data: 2023-05-24T21:09:11.599 DEBUG:boto:Headers: {} 2023-05-24T21:09:11.600 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:11.600 DEBUG:boto:Port: 8001 2023-05-24T21:09:11.600 DEBUG:boto:Params: {} 2023-05-24T21:09:11.601 DEBUG:boto:Token: None 2023-05-24T21:09:11.601 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:11 GMT /qskrps-12/ 2023-05-24T21:09:11.602 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:SRiYnf/ju70j5q90Eva9Vt45XB8= 2023-05-24T21:09:11.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:SRiYnf/ju70j5q90Eva9Vt45XB8=', 'Content-Length': '0'} 2023-05-24T21:09:11.604 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009be73477e0b0fe25-00646e7cf7-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:11.648 DEBUG:boto:b'qskrps-12dummy1000falsedummy2023-05-24T21:07:54.624Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:11.649 DEBUG:boto:path=/qskrps-12/ 2023-05-24T21:09:11.650 DEBUG:boto:auth_path=/qskrps-12/ 2023-05-24T21:09:11.650 DEBUG:boto:path=/qskrps-12/?prefix=dummy 2023-05-24T21:09:11.650 DEBUG:boto:auth_path=/qskrps-12/?prefix=dummy 2023-05-24T21:09:11.651 DEBUG:boto:Method: GET 2023-05-24T21:09:11.651 DEBUG:boto:Path: /qskrps-12/?prefix=dummy 2023-05-24T21:09:11.652 DEBUG:boto:Data: 2023-05-24T21:09:11.652 DEBUG:boto:Headers: {} 2023-05-24T21:09:11.652 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:11.653 DEBUG:boto:Port: 8000 2023-05-24T21:09:11.653 DEBUG:boto:Params: {} 2023-05-24T21:09:11.653 DEBUG:boto:Token: None 2023-05-24T21:09:11.654 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:11 GMT /qskrps-12/ 2023-05-24T21:09:11.654 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:SRiYnf/ju70j5q90Eva9Vt45XB8= 2023-05-24T21:09:11.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:SRiYnf/ju70j5q90Eva9Vt45XB8=', 'Content-Length': '0'} 2023-05-24T21:09:11.657 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea47c31171941a01-00646e7cf7-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:11.658 DEBUG:boto:b'qskrps-12dummy1000falsedummy2023-05-24T21:07:54.624Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:11.658 DEBUG:boto:path=/qskrps-12/dummy 2023-05-24T21:09:11.659 DEBUG:boto:auth_path=/qskrps-12/dummy 2023-05-24T21:09:11.659 DEBUG:boto:Method: HEAD 2023-05-24T21:09:11.659 DEBUG:boto:Path: /qskrps-12/dummy 2023-05-24T21:09:11.660 DEBUG:boto:Data: 2023-05-24T21:09:11.660 DEBUG:boto:Headers: {} 2023-05-24T21:09:11.660 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:11.661 DEBUG:boto:Port: 8001 2023-05-24T21:09:11.661 DEBUG:boto:Params: {} 2023-05-24T21:09:11.662 DEBUG:boto:Token: None 2023-05-24T21:09:11.662 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:11 GMT /qskrps-12/dummy 2023-05-24T21:09:11.662 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:GiIXx2NK5kngBGMiqlkHIZxguL0= 2023-05-24T21:09:11.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:GiIXx2NK5kngBGMiqlkHIZxguL0=', 'Content-Length': '0'} 2023-05-24T21:09:11.665 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:07:54 GMT'), ('x-amz-version-id', 'km8zCAetA1TzFWDKQlSimNmDeLQdejH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000011ba511cf75d9d37-00646e7cf7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:11.666 DEBUG:boto:path=/qskrps-12/dummy 2023-05-24T21:09:11.666 DEBUG:boto:auth_path=/qskrps-12/dummy 2023-05-24T21:09:11.666 DEBUG:boto:Method: HEAD 2023-05-24T21:09:11.667 DEBUG:boto:Path: /qskrps-12/dummy 2023-05-24T21:09:11.667 DEBUG:boto:Data: 2023-05-24T21:09:11.667 DEBUG:boto:Headers: {} 2023-05-24T21:09:11.668 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:11.668 DEBUG:boto:Port: 8000 2023-05-24T21:09:11.668 DEBUG:boto:Params: {} 2023-05-24T21:09:11.669 DEBUG:boto:Token: None 2023-05-24T21:09:11.669 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:11 GMT /qskrps-12/dummy 2023-05-24T21:09:11.670 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:GiIXx2NK5kngBGMiqlkHIZxguL0= 2023-05-24T21:09:11.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:GiIXx2NK5kngBGMiqlkHIZxguL0=', 'Content-Length': '0'} 2023-05-24T21:09:11.672 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:07:54 GMT'), ('x-amz-version-id', 'km8zCAetA1TzFWDKQlSimNmDeLQdejH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000257cf667c836da2a-00646e7cf7-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:11.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-05-24T21:09:11.673 DEBUG:boto:path=/qskrps-12/dummy 2023-05-24T21:09:11.674 DEBUG:boto:auth_path=/qskrps-12/dummy 2023-05-24T21:09:11.674 DEBUG:boto:path=/qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:11.674 DEBUG:boto:auth_path=/qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:11.675 DEBUG:boto:Method: GET 2023-05-24T21:09:11.675 DEBUG:boto:Path: /qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:11.675 DEBUG:boto:Data: 2023-05-24T21:09:11.676 DEBUG:boto:Headers: {} 2023-05-24T21:09:11.676 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:11.677 DEBUG:boto:Port: 8001 2023-05-24T21:09:11.677 DEBUG:boto:Params: {} 2023-05-24T21:09:11.677 DEBUG:boto:Token: None 2023-05-24T21:09:11.678 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:11 GMT /qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:11.678 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:T98UbwBvlXeIjI0F5qASjLnNWeE= 2023-05-24T21:09:11.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:T98UbwBvlXeIjI0F5qASjLnNWeE=', 'Content-Length': '0'} 2023-05-24T21:09:11.680 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:07:54 GMT'), ('x-amz-version-id', 'km8zCAetA1TzFWDKQlSimNmDeLQdejH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000193c318498bd7955-00646e7cf7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:11.681 DEBUG:boto:path=/qskrps-12/dummy 2023-05-24T21:09:11.682 DEBUG:boto:auth_path=/qskrps-12/dummy 2023-05-24T21:09:11.682 DEBUG:boto:path=/qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:11.682 DEBUG:boto:auth_path=/qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:11.683 DEBUG:boto:Method: GET 2023-05-24T21:09:11.683 DEBUG:boto:Path: /qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:11.683 DEBUG:boto:Data: 2023-05-24T21:09:11.684 DEBUG:boto:Headers: {} 2023-05-24T21:09:11.684 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:11.684 DEBUG:boto:Port: 8000 2023-05-24T21:09:11.685 DEBUG:boto:Params: {} 2023-05-24T21:09:11.685 DEBUG:boto:Token: None 2023-05-24T21:09:11.686 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:11 GMT /qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:11.686 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:T98UbwBvlXeIjI0F5qASjLnNWeE= 2023-05-24T21:09:11.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:T98UbwBvlXeIjI0F5qASjLnNWeE=', 'Content-Length': '0'} 2023-05-24T21:09:11.688 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:07:54 GMT'), ('x-amz-version-id', 'km8zCAetA1TzFWDKQlSimNmDeLQdejH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000aa120046df5b2529-00646e7cf7-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:11.689 DEBUG:boto:path=/qskrps-12/ 2023-05-24T21:09:11.689 DEBUG:boto:auth_path=/qskrps-12/ 2023-05-24T21:09:11.690 DEBUG:boto:path=/qskrps-12/?prefix=dummy 2023-05-24T21:09:11.690 DEBUG:boto:auth_path=/qskrps-12/?prefix=dummy 2023-05-24T21:09:11.690 DEBUG:boto:Method: GET 2023-05-24T21:09:11.691 DEBUG:boto:Path: /qskrps-12/?prefix=dummy 2023-05-24T21:09:11.691 DEBUG:boto:Data: 2023-05-24T21:09:11.691 DEBUG:boto:Headers: {} 2023-05-24T21:09:11.692 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:11.692 DEBUG:boto:Port: 8001 2023-05-24T21:09:11.692 DEBUG:boto:Params: {} 2023-05-24T21:09:11.693 DEBUG:boto:Token: None 2023-05-24T21:09:11.693 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:11 GMT /qskrps-12/ 2023-05-24T21:09:11.694 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:SRiYnf/ju70j5q90Eva9Vt45XB8= 2023-05-24T21:09:11.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:SRiYnf/ju70j5q90Eva9Vt45XB8=', 'Content-Length': '0'} 2023-05-24T21:09:11.696 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097068c5201b4ec28-00646e7cf7-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:11.744 DEBUG:boto:b'qskrps-12dummy1000falsedummy2023-05-24T21:07:54.624Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:11.745 DEBUG:boto:path=/qskrps-12/ 2023-05-24T21:09:11.745 DEBUG:boto:auth_path=/qskrps-12/ 2023-05-24T21:09:11.746 DEBUG:boto:path=/qskrps-12/?prefix=dummy 2023-05-24T21:09:11.746 DEBUG:boto:auth_path=/qskrps-12/?prefix=dummy 2023-05-24T21:09:11.746 DEBUG:boto:Method: GET 2023-05-24T21:09:11.747 DEBUG:boto:Path: /qskrps-12/?prefix=dummy 2023-05-24T21:09:11.747 DEBUG:boto:Data: 2023-05-24T21:09:11.748 DEBUG:boto:Headers: {} 2023-05-24T21:09:11.748 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:11.748 DEBUG:boto:Port: 8000 2023-05-24T21:09:11.749 DEBUG:boto:Params: {} 2023-05-24T21:09:11.749 DEBUG:boto:Token: None 2023-05-24T21:09:11.749 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:11 GMT /qskrps-12/ 2023-05-24T21:09:11.750 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:SRiYnf/ju70j5q90Eva9Vt45XB8= 2023-05-24T21:09:11.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:SRiYnf/ju70j5q90Eva9Vt45XB8=', 'Content-Length': '0'} 2023-05-24T21:09:11.753 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d31e83eba4727df-00646e7cf7-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:11.753 DEBUG:boto:b'qskrps-12dummy1000falsedummy2023-05-24T21:07:54.624Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:11.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-05-24T21:09:11.754 DEBUG:boto:path=/qskrps-12/obj-a1 2023-05-24T21:09:11.755 DEBUG:boto:auth_path=/qskrps-12/obj-a1 2023-05-24T21:09:11.755 DEBUG:boto:path=/qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:11.755 DEBUG:boto:auth_path=/qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:11.756 DEBUG:boto:Method: GET 2023-05-24T21:09:11.756 DEBUG:boto:Path: /qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:11.756 DEBUG:boto:Data: 2023-05-24T21:09:11.757 DEBUG:boto:Headers: {} 2023-05-24T21:09:11.757 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:11.757 DEBUG:boto:Port: 8001 2023-05-24T21:09:11.758 DEBUG:boto:Params: {} 2023-05-24T21:09:11.758 DEBUG:boto:Token: None 2023-05-24T21:09:11.759 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:11 GMT /qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:11.759 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:VkQWIpVRCqw6LZuIMnmY+PHN6KM= 2023-05-24T21:09:11.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:VkQWIpVRCqw6LZuIMnmY+PHN6KM=', 'Content-Length': '0'} 2023-05-24T21:09:11.761 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:08:15 GMT'), ('x-amz-version-id', 'OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d235a30235ee8caf-00646e7cf7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:11.762 DEBUG:boto:path=/qskrps-12/obj-a1 2023-05-24T21:09:11.762 DEBUG:boto:auth_path=/qskrps-12/obj-a1 2023-05-24T21:09:11.763 DEBUG:boto:path=/qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:11.763 DEBUG:boto:auth_path=/qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:11.764 DEBUG:boto:Method: GET 2023-05-24T21:09:11.764 DEBUG:boto:Path: /qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:11.764 DEBUG:boto:Data: 2023-05-24T21:09:11.765 DEBUG:boto:Headers: {} 2023-05-24T21:09:11.765 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:11.765 DEBUG:boto:Port: 8000 2023-05-24T21:09:11.766 DEBUG:boto:Params: {} 2023-05-24T21:09:11.766 DEBUG:boto:Token: None 2023-05-24T21:09:11.767 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:11 GMT /qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:11.767 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:VkQWIpVRCqw6LZuIMnmY+PHN6KM= 2023-05-24T21:09:11.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:VkQWIpVRCqw6LZuIMnmY+PHN6KM=', 'Content-Length': '0'} 2023-05-24T21:09:11.770 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:08:15 GMT'), ('x-amz-version-id', 'OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000061d997b5218925ee-00646e7cf7-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:11.812 DEBUG:boto:path=/qskrps-12/obj-a1 2023-05-24T21:09:11.813 DEBUG:boto:auth_path=/qskrps-12/obj-a1 2023-05-24T21:09:11.813 DEBUG:boto:path=/qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:11.813 DEBUG:boto:auth_path=/qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:11.814 DEBUG:boto:Method: HEAD 2023-05-24T21:09:11.814 DEBUG:boto:Path: /qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:11.814 DEBUG:boto:Data: 2023-05-24T21:09:11.815 DEBUG:boto:Headers: {} 2023-05-24T21:09:11.815 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:11.815 DEBUG:boto:Port: 8001 2023-05-24T21:09:11.816 DEBUG:boto:Params: {} 2023-05-24T21:09:11.816 DEBUG:boto:Token: None 2023-05-24T21:09:11.817 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:11 GMT /qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:11.817 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:lSxbE9whOAnGJFCyK7Y/Fih/gIk= 2023-05-24T21:09:11.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:lSxbE9whOAnGJFCyK7Y/Fih/gIk=', 'Content-Length': '0'} 2023-05-24T21:09:11.819 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:08:15 GMT'), ('x-amz-version-id', 'OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001134a382e8e3a02a-00646e7cf7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:11.820 DEBUG:boto:path=/qskrps-12/obj-a1 2023-05-24T21:09:11.820 DEBUG:boto:auth_path=/qskrps-12/obj-a1 2023-05-24T21:09:11.821 DEBUG:boto:path=/qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:11.821 DEBUG:boto:auth_path=/qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:11.821 DEBUG:boto:Method: HEAD 2023-05-24T21:09:11.822 DEBUG:boto:Path: /qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:11.822 DEBUG:boto:Data: 2023-05-24T21:09:11.822 DEBUG:boto:Headers: {} 2023-05-24T21:09:11.823 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:11.823 DEBUG:boto:Port: 8000 2023-05-24T21:09:11.823 DEBUG:boto:Params: {} 2023-05-24T21:09:11.824 DEBUG:boto:Token: None 2023-05-24T21:09:11.824 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:11 GMT /qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:11.824 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:lSxbE9whOAnGJFCyK7Y/Fih/gIk= 2023-05-24T21:09:11.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:lSxbE9whOAnGJFCyK7Y/Fih/gIk=', 'Content-Length': '0'} 2023-05-24T21:09:11.827 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:08:15 GMT'), ('x-amz-version-id', 'OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007b340dbbc014f2f6-00646e7cf7-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:11.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-05-24T21:09:11.828 DEBUG:boto:path=/qskrps-12/obj-a1 2023-05-24T21:09:11.828 DEBUG:boto:auth_path=/qskrps-12/obj-a1 2023-05-24T21:09:11.828 DEBUG:boto:path=/qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:11.829 DEBUG:boto:auth_path=/qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:11.829 DEBUG:boto:Method: GET 2023-05-24T21:09:11.829 DEBUG:boto:Path: /qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:11.830 DEBUG:boto:Data: 2023-05-24T21:09:11.830 DEBUG:boto:Headers: {} 2023-05-24T21:09:11.830 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:11.831 DEBUG:boto:Port: 8001 2023-05-24T21:09:11.831 DEBUG:boto:Params: {} 2023-05-24T21:09:11.832 DEBUG:boto:Token: None 2023-05-24T21:09:11.832 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:11 GMT /qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:11.832 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:VkQWIpVRCqw6LZuIMnmY+PHN6KM= 2023-05-24T21:09:11.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:VkQWIpVRCqw6LZuIMnmY+PHN6KM=', 'Content-Length': '0'} 2023-05-24T21:09:11.835 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:08:15 GMT'), ('x-amz-version-id', 'OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003218011ec383e6a4-00646e7cf7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:11.835 DEBUG:boto:path=/qskrps-12/obj-a1 2023-05-24T21:09:11.836 DEBUG:boto:auth_path=/qskrps-12/obj-a1 2023-05-24T21:09:11.836 DEBUG:boto:path=/qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:11.836 DEBUG:boto:auth_path=/qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:11.837 DEBUG:boto:Method: GET 2023-05-24T21:09:11.837 DEBUG:boto:Path: /qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:11.837 DEBUG:boto:Data: 2023-05-24T21:09:11.838 DEBUG:boto:Headers: {} 2023-05-24T21:09:11.838 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:11.838 DEBUG:boto:Port: 8000 2023-05-24T21:09:11.839 DEBUG:boto:Params: {} 2023-05-24T21:09:11.839 DEBUG:boto:Token: None 2023-05-24T21:09:11.840 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:11 GMT /qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:11.840 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:VkQWIpVRCqw6LZuIMnmY+PHN6KM= 2023-05-24T21:09:11.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:VkQWIpVRCqw6LZuIMnmY+PHN6KM=', 'Content-Length': '0'} 2023-05-24T21:09:11.842 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:08:15 GMT'), ('x-amz-version-id', 'OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fc6f8de08ddcb976-00646e7cf7-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:11.843 DEBUG:boto:path=/qskrps-12/ 2023-05-24T21:09:11.843 DEBUG:boto:auth_path=/qskrps-12/ 2023-05-24T21:09:11.844 DEBUG:boto:path=/qskrps-12/?prefix=obj-a1 2023-05-24T21:09:11.844 DEBUG:boto:auth_path=/qskrps-12/?prefix=obj-a1 2023-05-24T21:09:11.844 DEBUG:boto:Method: GET 2023-05-24T21:09:11.845 DEBUG:boto:Path: /qskrps-12/?prefix=obj-a1 2023-05-24T21:09:11.845 DEBUG:boto:Data: 2023-05-24T21:09:11.845 DEBUG:boto:Headers: {} 2023-05-24T21:09:11.846 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:11.846 DEBUG:boto:Port: 8001 2023-05-24T21:09:11.847 DEBUG:boto:Params: {} 2023-05-24T21:09:11.847 DEBUG:boto:Token: None 2023-05-24T21:09:11.847 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:11 GMT /qskrps-12/ 2023-05-24T21:09:11.848 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:SRiYnf/ju70j5q90Eva9Vt45XB8= 2023-05-24T21:09:11.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:SRiYnf/ju70j5q90Eva9Vt45XB8=', 'Content-Length': '0'} 2023-05-24T21:09:11.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fe7561b99ef9a78-00646e7cf7-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:11.851 DEBUG:boto:b'qskrps-12obj-a11000falseobj-a12023-05-24T21:08:15.067Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:11.852 DEBUG:boto:path=/qskrps-12/ 2023-05-24T21:09:11.852 DEBUG:boto:auth_path=/qskrps-12/ 2023-05-24T21:09:11.852 DEBUG:boto:path=/qskrps-12/?prefix=obj-a1 2023-05-24T21:09:11.853 DEBUG:boto:auth_path=/qskrps-12/?prefix=obj-a1 2023-05-24T21:09:11.853 DEBUG:boto:Method: GET 2023-05-24T21:09:11.854 DEBUG:boto:Path: /qskrps-12/?prefix=obj-a1 2023-05-24T21:09:11.854 DEBUG:boto:Data: 2023-05-24T21:09:11.854 DEBUG:boto:Headers: {} 2023-05-24T21:09:11.855 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:11.855 DEBUG:boto:Port: 8000 2023-05-24T21:09:11.855 DEBUG:boto:Params: {} 2023-05-24T21:09:11.856 DEBUG:boto:Token: None 2023-05-24T21:09:11.856 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:11 GMT /qskrps-12/ 2023-05-24T21:09:11.857 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:SRiYnf/ju70j5q90Eva9Vt45XB8= 2023-05-24T21:09:11.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:SRiYnf/ju70j5q90Eva9Vt45XB8=', 'Content-Length': '0'} 2023-05-24T21:09:11.859 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030856e63ce0fc9f9-00646e7cf7-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:11.860 DEBUG:boto:b'qskrps-12obj-a11000falseobj-a12023-05-24T21:08:15.067Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:11.860 DEBUG:boto:path=/qskrps-12/obj-a1 2023-05-24T21:09:11.861 DEBUG:boto:auth_path=/qskrps-12/obj-a1 2023-05-24T21:09:11.861 DEBUG:boto:Method: HEAD 2023-05-24T21:09:11.861 DEBUG:boto:Path: /qskrps-12/obj-a1 2023-05-24T21:09:11.862 DEBUG:boto:Data: 2023-05-24T21:09:11.862 DEBUG:boto:Headers: {} 2023-05-24T21:09:11.862 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:11.863 DEBUG:boto:Port: 8001 2023-05-24T21:09:11.863 DEBUG:boto:Params: {} 2023-05-24T21:09:11.863 DEBUG:boto:Token: None 2023-05-24T21:09:11.864 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:11 GMT /qskrps-12/obj-a1 2023-05-24T21:09:11.864 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:3ngzbQsP35Pp0de1+m9VuyZBYEg= 2023-05-24T21:09:11.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:3ngzbQsP35Pp0de1+m9VuyZBYEg=', 'Content-Length': '0'} 2023-05-24T21:09:11.867 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:08:15 GMT'), ('x-amz-version-id', 'OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b54afd3747435fb1-00646e7cf7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:11.867 DEBUG:boto:path=/qskrps-12/obj-a1 2023-05-24T21:09:11.867 DEBUG:boto:auth_path=/qskrps-12/obj-a1 2023-05-24T21:09:11.868 DEBUG:boto:Method: HEAD 2023-05-24T21:09:11.868 DEBUG:boto:Path: /qskrps-12/obj-a1 2023-05-24T21:09:11.869 DEBUG:boto:Data: 2023-05-24T21:09:11.869 DEBUG:boto:Headers: {} 2023-05-24T21:09:11.869 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:11.870 DEBUG:boto:Port: 8000 2023-05-24T21:09:11.870 DEBUG:boto:Params: {} 2023-05-24T21:09:11.870 DEBUG:boto:Token: None 2023-05-24T21:09:11.871 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:11 GMT /qskrps-12/obj-a1 2023-05-24T21:09:11.871 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:3ngzbQsP35Pp0de1+m9VuyZBYEg= 2023-05-24T21:09:11.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:3ngzbQsP35Pp0de1+m9VuyZBYEg=', 'Content-Length': '0'} 2023-05-24T21:09:11.874 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:08:15 GMT'), ('x-amz-version-id', 'OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002e92d29c18376829-00646e7cf7-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:11.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-05-24T21:09:11.875 DEBUG:boto:path=/qskrps-12/obj-a1 2023-05-24T21:09:11.875 DEBUG:boto:auth_path=/qskrps-12/obj-a1 2023-05-24T21:09:11.875 DEBUG:boto:path=/qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:11.876 DEBUG:boto:auth_path=/qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:11.876 DEBUG:boto:Method: GET 2023-05-24T21:09:11.876 DEBUG:boto:Path: /qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:11.877 DEBUG:boto:Data: 2023-05-24T21:09:11.877 DEBUG:boto:Headers: {} 2023-05-24T21:09:11.877 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:11.878 DEBUG:boto:Port: 8001 2023-05-24T21:09:11.878 DEBUG:boto:Params: {} 2023-05-24T21:09:11.879 DEBUG:boto:Token: None 2023-05-24T21:09:11.879 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:11 GMT /qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:11.879 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:VkQWIpVRCqw6LZuIMnmY+PHN6KM= 2023-05-24T21:09:11.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:VkQWIpVRCqw6LZuIMnmY+PHN6KM=', 'Content-Length': '0'} 2023-05-24T21:09:11.882 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:08:15 GMT'), ('x-amz-version-id', 'OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001499e2253334e61f-00646e7cf7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:11.883 DEBUG:boto:path=/qskrps-12/obj-a1 2023-05-24T21:09:11.883 DEBUG:boto:auth_path=/qskrps-12/obj-a1 2023-05-24T21:09:11.883 DEBUG:boto:path=/qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:11.884 DEBUG:boto:auth_path=/qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:11.884 DEBUG:boto:Method: GET 2023-05-24T21:09:11.884 DEBUG:boto:Path: /qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:11.885 DEBUG:boto:Data: 2023-05-24T21:09:11.885 DEBUG:boto:Headers: {} 2023-05-24T21:09:11.886 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:11.886 DEBUG:boto:Port: 8000 2023-05-24T21:09:11.886 DEBUG:boto:Params: {} 2023-05-24T21:09:11.887 DEBUG:boto:Token: None 2023-05-24T21:09:11.887 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:11 GMT /qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:11.888 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:VkQWIpVRCqw6LZuIMnmY+PHN6KM= 2023-05-24T21:09:11.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:VkQWIpVRCqw6LZuIMnmY+PHN6KM=', 'Content-Length': '0'} 2023-05-24T21:09:11.890 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:08:15 GMT'), ('x-amz-version-id', 'OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000412809f152aa7c17-00646e7cf7-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:11.890 DEBUG:boto:path=/qskrps-12/ 2023-05-24T21:09:11.891 DEBUG:boto:auth_path=/qskrps-12/ 2023-05-24T21:09:11.891 DEBUG:boto:path=/qskrps-12/?prefix=obj-a1 2023-05-24T21:09:11.892 DEBUG:boto:auth_path=/qskrps-12/?prefix=obj-a1 2023-05-24T21:09:11.892 DEBUG:boto:Method: GET 2023-05-24T21:09:11.892 DEBUG:boto:Path: /qskrps-12/?prefix=obj-a1 2023-05-24T21:09:11.893 DEBUG:boto:Data: 2023-05-24T21:09:11.893 DEBUG:boto:Headers: {} 2023-05-24T21:09:11.893 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:11.894 DEBUG:boto:Port: 8001 2023-05-24T21:09:11.894 DEBUG:boto:Params: {} 2023-05-24T21:09:11.894 DEBUG:boto:Token: None 2023-05-24T21:09:11.895 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:11 GMT /qskrps-12/ 2023-05-24T21:09:11.895 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:SRiYnf/ju70j5q90Eva9Vt45XB8= 2023-05-24T21:09:11.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:SRiYnf/ju70j5q90Eva9Vt45XB8=', 'Content-Length': '0'} 2023-05-24T21:09:11.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000068e9a05de099e76-00646e7cf7-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:11.898 DEBUG:boto:b'qskrps-12obj-a11000falseobj-a12023-05-24T21:08:15.067Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:11.899 DEBUG:boto:path=/qskrps-12/ 2023-05-24T21:09:11.900 DEBUG:boto:auth_path=/qskrps-12/ 2023-05-24T21:09:11.900 DEBUG:boto:path=/qskrps-12/?prefix=obj-a1 2023-05-24T21:09:11.900 DEBUG:boto:auth_path=/qskrps-12/?prefix=obj-a1 2023-05-24T21:09:11.901 DEBUG:boto:Method: GET 2023-05-24T21:09:11.901 DEBUG:boto:Path: /qskrps-12/?prefix=obj-a1 2023-05-24T21:09:11.901 DEBUG:boto:Data: 2023-05-24T21:09:11.902 DEBUG:boto:Headers: {} 2023-05-24T21:09:11.902 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:11.902 DEBUG:boto:Port: 8000 2023-05-24T21:09:11.903 DEBUG:boto:Params: {} 2023-05-24T21:09:11.903 DEBUG:boto:Token: None 2023-05-24T21:09:11.904 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:11 GMT /qskrps-12/ 2023-05-24T21:09:11.904 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:SRiYnf/ju70j5q90Eva9Vt45XB8= 2023-05-24T21:09:11.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:SRiYnf/ju70j5q90Eva9Vt45XB8=', 'Content-Length': '0'} 2023-05-24T21:09:11.907 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b4c92d793496547-00646e7cf7-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:11.907 DEBUG:boto:b'qskrps-12obj-a11000falseobj-a12023-05-24T21:08:15.067Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:11.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-05-24T21:09:11.908 DEBUG:boto:path=/qskrps-12/obj-a2 2023-05-24T21:09:11.908 DEBUG:boto:auth_path=/qskrps-12/obj-a2 2023-05-24T21:09:11.909 DEBUG:boto:path=/qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:11.909 DEBUG:boto:auth_path=/qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:11.910 DEBUG:boto:Method: GET 2023-05-24T21:09:11.910 DEBUG:boto:Path: /qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:11.910 DEBUG:boto:Data: 2023-05-24T21:09:11.911 DEBUG:boto:Headers: {} 2023-05-24T21:09:11.911 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:11.911 DEBUG:boto:Port: 8001 2023-05-24T21:09:11.912 DEBUG:boto:Params: {} 2023-05-24T21:09:11.912 DEBUG:boto:Token: None 2023-05-24T21:09:11.912 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:11 GMT /qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:11.913 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ISwg4iyn3SojFExeYof6nfL9Vrw= 2023-05-24T21:09:11.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ISwg4iyn3SojFExeYof6nfL9Vrw=', 'Content-Length': '0'} 2023-05-24T21:09:11.915 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:08:15 GMT'), ('x-amz-version-id', 'PcWX5rYNtc410b9LwKxno3LHHZeAmhm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000044f2537c6ec624ce-00646e7cf7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:11.916 DEBUG:boto:path=/qskrps-12/obj-a2 2023-05-24T21:09:11.916 DEBUG:boto:auth_path=/qskrps-12/obj-a2 2023-05-24T21:09:11.917 DEBUG:boto:path=/qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:11.917 DEBUG:boto:auth_path=/qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:11.917 DEBUG:boto:Method: GET 2023-05-24T21:09:11.918 DEBUG:boto:Path: /qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:11.918 DEBUG:boto:Data: 2023-05-24T21:09:11.918 DEBUG:boto:Headers: {} 2023-05-24T21:09:11.919 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:11.919 DEBUG:boto:Port: 8000 2023-05-24T21:09:11.920 DEBUG:boto:Params: {} 2023-05-24T21:09:11.920 DEBUG:boto:Token: None 2023-05-24T21:09:11.920 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:11 GMT /qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:11.921 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ISwg4iyn3SojFExeYof6nfL9Vrw= 2023-05-24T21:09:11.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ISwg4iyn3SojFExeYof6nfL9Vrw=', 'Content-Length': '0'} 2023-05-24T21:09:11.923 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:08:15 GMT'), ('x-amz-version-id', 'PcWX5rYNtc410b9LwKxno3LHHZeAmhm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000043184f8fd5beb965-00646e7cf7-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:11.924 DEBUG:boto:path=/qskrps-12/obj-a2 2023-05-24T21:09:11.924 DEBUG:boto:auth_path=/qskrps-12/obj-a2 2023-05-24T21:09:11.924 DEBUG:boto:path=/qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:11.925 DEBUG:boto:auth_path=/qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:11.925 DEBUG:boto:Method: HEAD 2023-05-24T21:09:11.925 DEBUG:boto:Path: /qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:11.926 DEBUG:boto:Data: 2023-05-24T21:09:11.926 DEBUG:boto:Headers: {} 2023-05-24T21:09:11.926 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:11.927 DEBUG:boto:Port: 8001 2023-05-24T21:09:11.927 DEBUG:boto:Params: {} 2023-05-24T21:09:11.927 DEBUG:boto:Token: None 2023-05-24T21:09:11.928 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:11 GMT /qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:11.928 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:vkyVwZ0exX7IKZXYYFkOPEecDpE= 2023-05-24T21:09:11.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:vkyVwZ0exX7IKZXYYFkOPEecDpE=', 'Content-Length': '0'} 2023-05-24T21:09:11.930 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:08:15 GMT'), ('x-amz-version-id', 'PcWX5rYNtc410b9LwKxno3LHHZeAmhm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003e6a61dbfad4a0e9-00646e7cf7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:11.931 DEBUG:boto:path=/qskrps-12/obj-a2 2023-05-24T21:09:11.931 DEBUG:boto:auth_path=/qskrps-12/obj-a2 2023-05-24T21:09:11.932 DEBUG:boto:path=/qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:11.932 DEBUG:boto:auth_path=/qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:11.932 DEBUG:boto:Method: HEAD 2023-05-24T21:09:11.933 DEBUG:boto:Path: /qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:11.933 DEBUG:boto:Data: 2023-05-24T21:09:11.933 DEBUG:boto:Headers: {} 2023-05-24T21:09:11.934 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:11.934 DEBUG:boto:Port: 8000 2023-05-24T21:09:11.934 DEBUG:boto:Params: {} 2023-05-24T21:09:11.935 DEBUG:boto:Token: None 2023-05-24T21:09:11.935 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:11 GMT /qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:11.936 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:vkyVwZ0exX7IKZXYYFkOPEecDpE= 2023-05-24T21:09:11.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:vkyVwZ0exX7IKZXYYFkOPEecDpE=', 'Content-Length': '0'} 2023-05-24T21:09:11.938 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:08:15 GMT'), ('x-amz-version-id', 'PcWX5rYNtc410b9LwKxno3LHHZeAmhm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008153d3344750f93a-00646e7cf7-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:11.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-05-24T21:09:11.939 DEBUG:boto:path=/qskrps-12/obj-a2 2023-05-24T21:09:11.939 DEBUG:boto:auth_path=/qskrps-12/obj-a2 2023-05-24T21:09:11.939 DEBUG:boto:path=/qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:11.940 DEBUG:boto:auth_path=/qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:11.940 DEBUG:boto:Method: GET 2023-05-24T21:09:11.941 DEBUG:boto:Path: /qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:11.941 DEBUG:boto:Data: 2023-05-24T21:09:11.941 DEBUG:boto:Headers: {} 2023-05-24T21:09:11.942 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:11.942 DEBUG:boto:Port: 8001 2023-05-24T21:09:11.942 DEBUG:boto:Params: {} 2023-05-24T21:09:11.943 DEBUG:boto:Token: None 2023-05-24T21:09:11.943 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:11 GMT /qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:11.943 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ISwg4iyn3SojFExeYof6nfL9Vrw= 2023-05-24T21:09:11.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ISwg4iyn3SojFExeYof6nfL9Vrw=', 'Content-Length': '0'} 2023-05-24T21:09:11.946 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:08:15 GMT'), ('x-amz-version-id', 'PcWX5rYNtc410b9LwKxno3LHHZeAmhm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f5394f51362b6fbb-00646e7cf7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:11.946 DEBUG:boto:path=/qskrps-12/obj-a2 2023-05-24T21:09:11.947 DEBUG:boto:auth_path=/qskrps-12/obj-a2 2023-05-24T21:09:11.947 DEBUG:boto:path=/qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:11.948 DEBUG:boto:auth_path=/qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:11.948 DEBUG:boto:Method: GET 2023-05-24T21:09:11.948 DEBUG:boto:Path: /qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:11.949 DEBUG:boto:Data: 2023-05-24T21:09:11.949 DEBUG:boto:Headers: {} 2023-05-24T21:09:11.949 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:11.950 DEBUG:boto:Port: 8000 2023-05-24T21:09:11.950 DEBUG:boto:Params: {} 2023-05-24T21:09:11.950 DEBUG:boto:Token: None 2023-05-24T21:09:11.951 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:11 GMT /qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:11.951 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ISwg4iyn3SojFExeYof6nfL9Vrw= 2023-05-24T21:09:11.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ISwg4iyn3SojFExeYof6nfL9Vrw=', 'Content-Length': '0'} 2023-05-24T21:09:11.953 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:08:15 GMT'), ('x-amz-version-id', 'PcWX5rYNtc410b9LwKxno3LHHZeAmhm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c1219f27082c4cdd-00646e7cf7-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:11.954 DEBUG:boto:path=/qskrps-12/ 2023-05-24T21:09:11.955 DEBUG:boto:auth_path=/qskrps-12/ 2023-05-24T21:09:11.955 DEBUG:boto:path=/qskrps-12/?prefix=obj-a2 2023-05-24T21:09:11.955 DEBUG:boto:auth_path=/qskrps-12/?prefix=obj-a2 2023-05-24T21:09:11.956 DEBUG:boto:Method: GET 2023-05-24T21:09:11.956 DEBUG:boto:Path: /qskrps-12/?prefix=obj-a2 2023-05-24T21:09:11.956 DEBUG:boto:Data: 2023-05-24T21:09:11.957 DEBUG:boto:Headers: {} 2023-05-24T21:09:11.957 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:11.957 DEBUG:boto:Port: 8001 2023-05-24T21:09:11.958 DEBUG:boto:Params: {} 2023-05-24T21:09:11.958 DEBUG:boto:Token: None 2023-05-24T21:09:11.959 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:11 GMT /qskrps-12/ 2023-05-24T21:09:11.959 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:SRiYnf/ju70j5q90Eva9Vt45XB8= 2023-05-24T21:09:11.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:SRiYnf/ju70j5q90Eva9Vt45XB8=', 'Content-Length': '0'} 2023-05-24T21:09:11.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009cabc43a2f1314a8-00646e7cf7-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:11.962 DEBUG:boto:b'qskrps-12obj-a21000falseobj-a22023-05-24T21:08:15.193Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:11.963 DEBUG:boto:path=/qskrps-12/ 2023-05-24T21:09:11.963 DEBUG:boto:auth_path=/qskrps-12/ 2023-05-24T21:09:11.964 DEBUG:boto:path=/qskrps-12/?prefix=obj-a2 2023-05-24T21:09:11.964 DEBUG:boto:auth_path=/qskrps-12/?prefix=obj-a2 2023-05-24T21:09:11.964 DEBUG:boto:Method: GET 2023-05-24T21:09:11.965 DEBUG:boto:Path: /qskrps-12/?prefix=obj-a2 2023-05-24T21:09:11.965 DEBUG:boto:Data: 2023-05-24T21:09:11.966 DEBUG:boto:Headers: {} 2023-05-24T21:09:11.966 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:11.966 DEBUG:boto:Port: 8000 2023-05-24T21:09:11.967 DEBUG:boto:Params: {} 2023-05-24T21:09:11.967 DEBUG:boto:Token: None 2023-05-24T21:09:11.967 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:11 GMT /qskrps-12/ 2023-05-24T21:09:11.968 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:SRiYnf/ju70j5q90Eva9Vt45XB8= 2023-05-24T21:09:11.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:SRiYnf/ju70j5q90Eva9Vt45XB8=', 'Content-Length': '0'} 2023-05-24T21:09:11.970 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008ff9f01684205d0-00646e7cf7-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:11.971 DEBUG:boto:b'qskrps-12obj-a21000falseobj-a22023-05-24T21:08:15.193Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:11.971 DEBUG:boto:path=/qskrps-12/obj-a2 2023-05-24T21:09:11.972 DEBUG:boto:auth_path=/qskrps-12/obj-a2 2023-05-24T21:09:11.972 DEBUG:boto:Method: HEAD 2023-05-24T21:09:11.972 DEBUG:boto:Path: /qskrps-12/obj-a2 2023-05-24T21:09:11.973 DEBUG:boto:Data: 2023-05-24T21:09:11.973 DEBUG:boto:Headers: {} 2023-05-24T21:09:11.973 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:11.974 DEBUG:boto:Port: 8001 2023-05-24T21:09:11.974 DEBUG:boto:Params: {} 2023-05-24T21:09:11.974 DEBUG:boto:Token: None 2023-05-24T21:09:11.975 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:11 GMT /qskrps-12/obj-a2 2023-05-24T21:09:11.975 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:UavhwEdv9zk9Ycw64/8SXvBDmcI= 2023-05-24T21:09:11.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:UavhwEdv9zk9Ycw64/8SXvBDmcI=', 'Content-Length': '0'} 2023-05-24T21:09:11.978 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:08:15 GMT'), ('x-amz-version-id', 'PcWX5rYNtc410b9LwKxno3LHHZeAmhm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a8297a25d5813904-00646e7cf7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:11.978 DEBUG:boto:path=/qskrps-12/obj-a2 2023-05-24T21:09:11.979 DEBUG:boto:auth_path=/qskrps-12/obj-a2 2023-05-24T21:09:11.979 DEBUG:boto:Method: HEAD 2023-05-24T21:09:11.979 DEBUG:boto:Path: /qskrps-12/obj-a2 2023-05-24T21:09:11.980 DEBUG:boto:Data: 2023-05-24T21:09:11.980 DEBUG:boto:Headers: {} 2023-05-24T21:09:11.980 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:11.981 DEBUG:boto:Port: 8000 2023-05-24T21:09:11.981 DEBUG:boto:Params: {} 2023-05-24T21:09:11.981 DEBUG:boto:Token: None 2023-05-24T21:09:11.982 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:11 GMT /qskrps-12/obj-a2 2023-05-24T21:09:11.982 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:UavhwEdv9zk9Ycw64/8SXvBDmcI= 2023-05-24T21:09:11.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:UavhwEdv9zk9Ycw64/8SXvBDmcI=', 'Content-Length': '0'} 2023-05-24T21:09:11.985 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:08:15 GMT'), ('x-amz-version-id', 'PcWX5rYNtc410b9LwKxno3LHHZeAmhm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000034e7cab9e7be8154-00646e7cf7-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:11.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-05-24T21:09:11.985 DEBUG:boto:path=/qskrps-12/obj-a2 2023-05-24T21:09:11.986 DEBUG:boto:auth_path=/qskrps-12/obj-a2 2023-05-24T21:09:11.986 DEBUG:boto:path=/qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:11.987 DEBUG:boto:auth_path=/qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:11.987 DEBUG:boto:Method: GET 2023-05-24T21:09:11.987 DEBUG:boto:Path: /qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:11.988 DEBUG:boto:Data: 2023-05-24T21:09:11.988 DEBUG:boto:Headers: {} 2023-05-24T21:09:11.988 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:11.989 DEBUG:boto:Port: 8001 2023-05-24T21:09:11.989 DEBUG:boto:Params: {} 2023-05-24T21:09:11.989 DEBUG:boto:Token: None 2023-05-24T21:09:11.990 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:11 GMT /qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:11.990 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ISwg4iyn3SojFExeYof6nfL9Vrw= 2023-05-24T21:09:11.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ISwg4iyn3SojFExeYof6nfL9Vrw=', 'Content-Length': '0'} 2023-05-24T21:09:11.993 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:08:15 GMT'), ('x-amz-version-id', 'PcWX5rYNtc410b9LwKxno3LHHZeAmhm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000067082fdb78dcd9e0-00646e7cf7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:11.993 DEBUG:boto:path=/qskrps-12/obj-a2 2023-05-24T21:09:11.994 DEBUG:boto:auth_path=/qskrps-12/obj-a2 2023-05-24T21:09:11.994 DEBUG:boto:path=/qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:11.994 DEBUG:boto:auth_path=/qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:11.995 DEBUG:boto:Method: GET 2023-05-24T21:09:11.995 DEBUG:boto:Path: /qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:11.995 DEBUG:boto:Data: 2023-05-24T21:09:11.996 DEBUG:boto:Headers: {} 2023-05-24T21:09:11.996 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:11.996 DEBUG:boto:Port: 8000 2023-05-24T21:09:11.997 DEBUG:boto:Params: {} 2023-05-24T21:09:11.997 DEBUG:boto:Token: None 2023-05-24T21:09:11.998 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:11 GMT /qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:11.998 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ISwg4iyn3SojFExeYof6nfL9Vrw= 2023-05-24T21:09:11.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ISwg4iyn3SojFExeYof6nfL9Vrw=', 'Content-Length': '0'} 2023-05-24T21:09:12.000 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:08:15 GMT'), ('x-amz-version-id', 'PcWX5rYNtc410b9LwKxno3LHHZeAmhm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007c4aa43b1f9fb36f-00646e7cf7-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:12.001 DEBUG:boto:path=/qskrps-12/ 2023-05-24T21:09:12.001 DEBUG:boto:auth_path=/qskrps-12/ 2023-05-24T21:09:12.002 DEBUG:boto:path=/qskrps-12/?prefix=obj-a2 2023-05-24T21:09:12.002 DEBUG:boto:auth_path=/qskrps-12/?prefix=obj-a2 2023-05-24T21:09:12.002 DEBUG:boto:Method: GET 2023-05-24T21:09:12.003 DEBUG:boto:Path: /qskrps-12/?prefix=obj-a2 2023-05-24T21:09:12.003 DEBUG:boto:Data: 2023-05-24T21:09:12.003 DEBUG:boto:Headers: {} 2023-05-24T21:09:12.004 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:12.004 DEBUG:boto:Port: 8001 2023-05-24T21:09:12.004 DEBUG:boto:Params: {} 2023-05-24T21:09:12.005 DEBUG:boto:Token: None 2023-05-24T21:09:12.005 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:12 GMT /qskrps-12/ 2023-05-24T21:09:12.006 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:rCrSycqMSw+dDovc1xnEKH4HBtA= 2023-05-24T21:09:12.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:rCrSycqMSw+dDovc1xnEKH4HBtA=', 'Content-Length': '0'} 2023-05-24T21:09:12.008 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c56239c85212884-00646e7cf8-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:12.052 DEBUG:boto:b'qskrps-12obj-a21000falseobj-a22023-05-24T21:08:15.193Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:12.053 DEBUG:boto:path=/qskrps-12/ 2023-05-24T21:09:12.053 DEBUG:boto:auth_path=/qskrps-12/ 2023-05-24T21:09:12.054 DEBUG:boto:path=/qskrps-12/?prefix=obj-a2 2023-05-24T21:09:12.054 DEBUG:boto:auth_path=/qskrps-12/?prefix=obj-a2 2023-05-24T21:09:12.055 DEBUG:boto:Method: GET 2023-05-24T21:09:12.055 DEBUG:boto:Path: /qskrps-12/?prefix=obj-a2 2023-05-24T21:09:12.055 DEBUG:boto:Data: 2023-05-24T21:09:12.056 DEBUG:boto:Headers: {} 2023-05-24T21:09:12.056 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:12.056 DEBUG:boto:Port: 8000 2023-05-24T21:09:12.057 DEBUG:boto:Params: {} 2023-05-24T21:09:12.057 DEBUG:boto:Token: None 2023-05-24T21:09:12.058 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:12 GMT /qskrps-12/ 2023-05-24T21:09:12.058 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:rCrSycqMSw+dDovc1xnEKH4HBtA= 2023-05-24T21:09:12.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:rCrSycqMSw+dDovc1xnEKH4HBtA=', 'Content-Length': '0'} 2023-05-24T21:09:12.061 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4648cb459147406-00646e7cf8-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:12.061 DEBUG:boto:b'qskrps-12obj-a21000falseobj-a22023-05-24T21:08:15.193Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:12.062 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qskrps-12 zones={a2, a1} 2023-05-24T21:09:12.062 DEBUG:boto:path=/qskrps-11/ 2023-05-24T21:09:12.063 DEBUG:boto:auth_path=/qskrps-11/ 2023-05-24T21:09:12.063 DEBUG:boto:Method: HEAD 2023-05-24T21:09:12.063 DEBUG:boto:Path: /qskrps-11/ 2023-05-24T21:09:12.064 DEBUG:boto:Data: 2023-05-24T21:09:12.064 DEBUG:boto:Headers: {} 2023-05-24T21:09:12.064 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:12.065 DEBUG:boto:Port: 8000 2023-05-24T21:09:12.065 DEBUG:boto:Params: {} 2023-05-24T21:09:12.066 DEBUG:boto:Token: None 2023-05-24T21:09:12.066 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:12 GMT /qskrps-11/ 2023-05-24T21:09:12.066 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:B/L81ONargzO8/0B7t0PsaVLQyg= 2023-05-24T21:09:12.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:B/L81ONargzO8/0B7t0PsaVLQyg=', 'Content-Length': '0'} 2023-05-24T21:09:12.069 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f5d1c2a76541c933-00646e7cf8-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:09:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:12.069 DEBUG:boto:path=/qskrps-11/ 2023-05-24T21:09:12.070 DEBUG:boto:auth_path=/qskrps-11/ 2023-05-24T21:09:12.070 DEBUG:boto:path=/qskrps-11/?versions&max-keys=999&prefix=obj-a1 2023-05-24T21:09:12.070 DEBUG:boto:auth_path=/qskrps-11/?versions&max-keys=999&prefix=obj-a1 2023-05-24T21:09:12.071 DEBUG:boto:Method: GET 2023-05-24T21:09:12.071 DEBUG:boto:Path: /qskrps-11/?versions&max-keys=999&prefix=obj-a1 2023-05-24T21:09:12.071 DEBUG:boto:Data: 2023-05-24T21:09:12.072 DEBUG:boto:Headers: {} 2023-05-24T21:09:12.072 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:12.072 DEBUG:boto:Port: 8000 2023-05-24T21:09:12.073 DEBUG:boto:Params: {} 2023-05-24T21:09:12.073 DEBUG:boto:Token: None 2023-05-24T21:09:12.074 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:12 GMT /qskrps-11/?versions 2023-05-24T21:09:12.074 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:TQcHQfe4wpMWkvYQaFKfwJRwcMc= 2023-05-24T21:09:12.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:TQcHQfe4wpMWkvYQaFKfwJRwcMc=', 'Content-Length': '0'} 2023-05-24T21:09:12.077 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030cca6ae54742f4b-00646e7cf8-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:12.120 DEBUG:boto:b'qskrps-11obj-a1999falseobj-a154a54vsjHFa46aVpWzOVMoAJ0PQA-.vtrue2023-05-24T21:08:14.786Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:12.121 DEBUG:boto:path=/qskrps-11/obj-a1 2023-05-24T21:09:12.122 DEBUG:boto:auth_path=/qskrps-11/obj-a1 2023-05-24T21:09:12.122 DEBUG:boto:path=/qskrps-11/obj-a1?acl 2023-05-24T21:09:12.122 DEBUG:boto:auth_path=/qskrps-11/obj-a1?acl 2023-05-24T21:09:12.123 DEBUG:boto:Method: PUT 2023-05-24T21:09:12.123 DEBUG:boto:Path: /qskrps-11/obj-a1?acl 2023-05-24T21:09:12.123 DEBUG:boto:Data: 2023-05-24T21:09:12.124 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-05-24T21:09:12.124 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:12.124 DEBUG:boto:Port: 8000 2023-05-24T21:09:12.125 DEBUG:boto:Params: {} 2023-05-24T21:09:12.125 DEBUG:boto:Token: None 2023-05-24T21:09:12.125 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:09:12 GMT x-amz-acl:public-read /qskrps-11/obj-a1?acl 2023-05-24T21:09:12.126 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Zqte1IBE/kO2SrMIen+ByBeCB5g= 2023-05-24T21:09:12.126 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Zqte1IBE/kO2SrMIen+ByBeCB5g=', 'Content-Length': '0'} 2023-05-24T21:09:12.133 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005510969993dd72b8-00646e7cf8-110d-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:09:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:12.134 DEBUG:boto:path=/qskrps-11/ 2023-05-24T21:09:12.134 DEBUG:boto:auth_path=/qskrps-11/ 2023-05-24T21:09:12.134 DEBUG:boto:Method: HEAD 2023-05-24T21:09:12.135 DEBUG:boto:Path: /qskrps-11/ 2023-05-24T21:09:12.135 DEBUG:boto:Data: 2023-05-24T21:09:12.135 DEBUG:boto:Headers: {} 2023-05-24T21:09:12.136 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:12.136 DEBUG:boto:Port: 8001 2023-05-24T21:09:12.136 DEBUG:boto:Params: {} 2023-05-24T21:09:12.137 DEBUG:boto:Token: None 2023-05-24T21:09:12.137 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:12 GMT /qskrps-11/ 2023-05-24T21:09:12.137 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:B/L81ONargzO8/0B7t0PsaVLQyg= 2023-05-24T21:09:12.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:B/L81ONargzO8/0B7t0PsaVLQyg=', 'Content-Length': '0'} 2023-05-24T21:09:12.140 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000900336dcc6035225-00646e7cf8-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:09:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:12.141 DEBUG:boto:path=/qskrps-11/ 2023-05-24T21:09:12.141 DEBUG:boto:auth_path=/qskrps-11/ 2023-05-24T21:09:12.141 DEBUG:boto:path=/qskrps-11/?versions&max-keys=999&prefix=obj-a2 2023-05-24T21:09:12.142 DEBUG:boto:auth_path=/qskrps-11/?versions&max-keys=999&prefix=obj-a2 2023-05-24T21:09:12.142 DEBUG:boto:Method: GET 2023-05-24T21:09:12.142 DEBUG:boto:Path: /qskrps-11/?versions&max-keys=999&prefix=obj-a2 2023-05-24T21:09:12.143 DEBUG:boto:Data: 2023-05-24T21:09:12.143 DEBUG:boto:Headers: {} 2023-05-24T21:09:12.143 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:12.144 DEBUG:boto:Port: 8001 2023-05-24T21:09:12.144 DEBUG:boto:Params: {} 2023-05-24T21:09:12.144 DEBUG:boto:Token: None 2023-05-24T21:09:12.145 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:12 GMT /qskrps-11/?versions 2023-05-24T21:09:12.145 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:TQcHQfe4wpMWkvYQaFKfwJRwcMc= 2023-05-24T21:09:12.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:TQcHQfe4wpMWkvYQaFKfwJRwcMc=', 'Content-Length': '0'} 2023-05-24T21:09:12.148 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003467a1862d9cfbc8-00646e7cf8-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:12.149 DEBUG:boto:b'qskrps-11obj-a2999falseobj-a2YniDC1out7TFV1sI.TLfuqiPcVFd.5Strue2023-05-24T21:08:14.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:12.149 DEBUG:boto:path=/qskrps-11/obj-a2 2023-05-24T21:09:12.150 DEBUG:boto:auth_path=/qskrps-11/obj-a2 2023-05-24T21:09:12.150 DEBUG:boto:path=/qskrps-11/obj-a2?acl 2023-05-24T21:09:12.150 DEBUG:boto:auth_path=/qskrps-11/obj-a2?acl 2023-05-24T21:09:12.151 DEBUG:boto:Method: PUT 2023-05-24T21:09:12.151 DEBUG:boto:Path: /qskrps-11/obj-a2?acl 2023-05-24T21:09:12.151 DEBUG:boto:Data: 2023-05-24T21:09:12.152 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-05-24T21:09:12.152 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:12.152 DEBUG:boto:Port: 8001 2023-05-24T21:09:12.153 DEBUG:boto:Params: {} 2023-05-24T21:09:12.153 DEBUG:boto:Token: None 2023-05-24T21:09:12.153 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:09:12 GMT x-amz-acl:public-read /qskrps-11/obj-a2?acl 2023-05-24T21:09:12.154 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:CqPRq82fnLjE4ayeCBrpbk9zyaM= 2023-05-24T21:09:12.154 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:CqPRq82fnLjE4ayeCBrpbk9zyaM=', 'Content-Length': '0'} 2023-05-24T21:09:12.162 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000716707d780cd475b-00646e7cf8-1111-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:09:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:12.162 DEBUG:boto:path=/qskrps-12/ 2023-05-24T21:09:12.163 DEBUG:boto:auth_path=/qskrps-12/ 2023-05-24T21:09:12.163 DEBUG:boto:Method: HEAD 2023-05-24T21:09:12.163 DEBUG:boto:Path: /qskrps-12/ 2023-05-24T21:09:12.164 DEBUG:boto:Data: 2023-05-24T21:09:12.164 DEBUG:boto:Headers: {} 2023-05-24T21:09:12.164 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:12.165 DEBUG:boto:Port: 8000 2023-05-24T21:09:12.165 DEBUG:boto:Params: {} 2023-05-24T21:09:12.165 DEBUG:boto:Token: None 2023-05-24T21:09:12.166 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:12 GMT /qskrps-12/ 2023-05-24T21:09:12.166 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:j10x7LiDmavOU1Pg4vNQ+CNfCLA= 2023-05-24T21:09:12.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:j10x7LiDmavOU1Pg4vNQ+CNfCLA=', 'Content-Length': '0'} 2023-05-24T21:09:12.169 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001dfb184ea805d43d-00646e7cf8-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:09:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:12.169 DEBUG:boto:path=/qskrps-12/ 2023-05-24T21:09:12.170 DEBUG:boto:auth_path=/qskrps-12/ 2023-05-24T21:09:12.170 DEBUG:boto:path=/qskrps-12/?versions&max-keys=999&prefix=obj-a1 2023-05-24T21:09:12.171 DEBUG:boto:auth_path=/qskrps-12/?versions&max-keys=999&prefix=obj-a1 2023-05-24T21:09:12.171 DEBUG:boto:Method: GET 2023-05-24T21:09:12.171 DEBUG:boto:Path: /qskrps-12/?versions&max-keys=999&prefix=obj-a1 2023-05-24T21:09:12.172 DEBUG:boto:Data: 2023-05-24T21:09:12.172 DEBUG:boto:Headers: {} 2023-05-24T21:09:12.172 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:12.173 DEBUG:boto:Port: 8000 2023-05-24T21:09:12.173 DEBUG:boto:Params: {} 2023-05-24T21:09:12.173 DEBUG:boto:Token: None 2023-05-24T21:09:12.174 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:12 GMT /qskrps-12/?versions 2023-05-24T21:09:12.174 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:UzZSnF4W9VN6wKcKa2R8u7seQK0= 2023-05-24T21:09:12.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:UzZSnF4W9VN6wKcKa2R8u7seQK0=', 'Content-Length': '0'} 2023-05-24T21:09:12.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c212b3afab5fe25-00646e7cf8-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:12.220 DEBUG:boto:b'qskrps-12obj-a1999falseobj-a1OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiVtrue2023-05-24T21:08:15.067Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:12.221 DEBUG:boto:path=/qskrps-12/obj-a1 2023-05-24T21:09:12.222 DEBUG:boto:auth_path=/qskrps-12/obj-a1 2023-05-24T21:09:12.222 DEBUG:boto:path=/qskrps-12/obj-a1?acl 2023-05-24T21:09:12.222 DEBUG:boto:auth_path=/qskrps-12/obj-a1?acl 2023-05-24T21:09:12.223 DEBUG:boto:Method: PUT 2023-05-24T21:09:12.223 DEBUG:boto:Path: /qskrps-12/obj-a1?acl 2023-05-24T21:09:12.223 DEBUG:boto:Data: 2023-05-24T21:09:12.224 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-05-24T21:09:12.224 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:12.224 DEBUG:boto:Port: 8000 2023-05-24T21:09:12.225 DEBUG:boto:Params: {} 2023-05-24T21:09:12.225 DEBUG:boto:Token: None 2023-05-24T21:09:12.226 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:09:12 GMT x-amz-acl:public-read /qskrps-12/obj-a1?acl 2023-05-24T21:09:12.226 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Z2bwBfVfo1PUn6RurneRk+eC854= 2023-05-24T21:09:12.226 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Z2bwBfVfo1PUn6RurneRk+eC854=', 'Content-Length': '0'} 2023-05-24T21:09:12.233 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c024e1c3ea33a296-00646e7cf8-110d-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:09:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:12.234 DEBUG:boto:path=/qskrps-12/ 2023-05-24T21:09:12.234 DEBUG:boto:auth_path=/qskrps-12/ 2023-05-24T21:09:12.235 DEBUG:boto:Method: HEAD 2023-05-24T21:09:12.235 DEBUG:boto:Path: /qskrps-12/ 2023-05-24T21:09:12.235 DEBUG:boto:Data: 2023-05-24T21:09:12.236 DEBUG:boto:Headers: {} 2023-05-24T21:09:12.236 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:12.236 DEBUG:boto:Port: 8001 2023-05-24T21:09:12.237 DEBUG:boto:Params: {} 2023-05-24T21:09:12.237 DEBUG:boto:Token: None 2023-05-24T21:09:12.237 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:12 GMT /qskrps-12/ 2023-05-24T21:09:12.238 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:j10x7LiDmavOU1Pg4vNQ+CNfCLA= 2023-05-24T21:09:12.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:j10x7LiDmavOU1Pg4vNQ+CNfCLA=', 'Content-Length': '0'} 2023-05-24T21:09:12.241 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f4b369658acf34f9-00646e7cf8-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:09:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:12.241 DEBUG:boto:path=/qskrps-12/ 2023-05-24T21:09:12.242 DEBUG:boto:auth_path=/qskrps-12/ 2023-05-24T21:09:12.242 DEBUG:boto:path=/qskrps-12/?versions&max-keys=999&prefix=obj-a2 2023-05-24T21:09:12.243 DEBUG:boto:auth_path=/qskrps-12/?versions&max-keys=999&prefix=obj-a2 2023-05-24T21:09:12.243 DEBUG:boto:Method: GET 2023-05-24T21:09:12.243 DEBUG:boto:Path: /qskrps-12/?versions&max-keys=999&prefix=obj-a2 2023-05-24T21:09:12.244 DEBUG:boto:Data: 2023-05-24T21:09:12.244 DEBUG:boto:Headers: {} 2023-05-24T21:09:12.245 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:12.245 DEBUG:boto:Port: 8001 2023-05-24T21:09:12.245 DEBUG:boto:Params: {} 2023-05-24T21:09:12.246 DEBUG:boto:Token: None 2023-05-24T21:09:12.246 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:12 GMT /qskrps-12/?versions 2023-05-24T21:09:12.247 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:UzZSnF4W9VN6wKcKa2R8u7seQK0= 2023-05-24T21:09:12.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:UzZSnF4W9VN6wKcKa2R8u7seQK0=', 'Content-Length': '0'} 2023-05-24T21:09:12.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000479e968adae1b397-00646e7cf8-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:12.250 DEBUG:boto:b'qskrps-12obj-a2999falseobj-a2PcWX5rYNtc410b9LwKxno3LHHZeAmhmtrue2023-05-24T21:08:15.193Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:12.251 DEBUG:boto:path=/qskrps-12/obj-a2 2023-05-24T21:09:12.251 DEBUG:boto:auth_path=/qskrps-12/obj-a2 2023-05-24T21:09:12.252 DEBUG:boto:path=/qskrps-12/obj-a2?acl 2023-05-24T21:09:12.252 DEBUG:boto:auth_path=/qskrps-12/obj-a2?acl 2023-05-24T21:09:12.252 DEBUG:boto:Method: PUT 2023-05-24T21:09:12.253 DEBUG:boto:Path: /qskrps-12/obj-a2?acl 2023-05-24T21:09:12.253 DEBUG:boto:Data: 2023-05-24T21:09:12.253 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-05-24T21:09:12.254 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:12.254 DEBUG:boto:Port: 8001 2023-05-24T21:09:12.254 DEBUG:boto:Params: {} 2023-05-24T21:09:12.255 DEBUG:boto:Token: None 2023-05-24T21:09:12.255 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:09:12 GMT x-amz-acl:public-read /qskrps-12/obj-a2?acl 2023-05-24T21:09:12.255 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:+TWFv+RgqwSskDs+M4xEy79gYm0= 2023-05-24T21:09:12.256 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:+TWFv+RgqwSskDs+M4xEy79gYm0=', 'Content-Length': '0'} 2023-05-24T21:09:12.263 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000db85a118d613cde3-00646e7cf8-1111-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:09:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:12.263 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=qskrps-11 2023-05-24T21:09:12.263 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:09:12.264 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:09:12.264 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-11 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:09:12.328 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:09:12.328 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:09:12.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:09:12.379+0000 7f79d3980780 1 waiting for incremental sync to catch up: 2023-05-24T21:09:12.385 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000003.8.10, , , 00000000002.11.11, 00000000010.25.6, ] 2023-05-24T21:09:12.386 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000003.8.10, , , 00000000002.11.11, 00000000011.28.6, ] 2023-05-24T21:09:17.386 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:09:17.381+0000 7f79d3980780 1 waiting for incremental sync to catch up: 2023-05-24T21:09:17.387 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000003.8.10, , , 00000000002.11.11, 00000000010.25.6, ] 2023-05-24T21:09:17.387 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000003.8.10, , , 00000000002.11.11, 00000000011.28.6, ] 2023-05-24T21:09:22.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:09:22.382+0000 7f79d3980780 1 waiting for incremental sync to catch up: 2023-05-24T21:09:22.388 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000003.8.10, , , 00000000002.11.11, 00000000010.25.6, ] 2023-05-24T21:09:22.388 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000003.8.10, , , 00000000002.11.11, 00000000011.28.6, ] 2023-05-24T21:09:27.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:09:27.383+0000 7f79d3980780 1 waiting for incremental sync to catch up: 2023-05-24T21:09:27.389 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000003.8.10, , , 00000000002.11.11, 00000000010.25.6, ] 2023-05-24T21:09:27.390 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000003.8.10, , , 00000000002.11.11, 00000000011.28.6, ] 2023-05-24T21:09:32.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:09:32.384+0000 7f79d3980780 1 bucket sync caught up with source: 2023-05-24T21:09:32.391 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000003.8.10, , , 00000000003.14.6, 00000000011.28.6, ] 2023-05-24T21:09:32.391 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000003.8.10, , , 00000000002.11.11, 00000000011.28.6, ] 2023-05-24T21:09:32.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:09:32.384+0000 7f79d3980780 0 bucket checkpoint complete 2023-05-24T21:09:32.407 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=qskrps-11 2023-05-24T21:09:32.408 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-11', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:09:32.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-11', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:09:32.409 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-11 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:09:32.471 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:09:32.471 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:09:32.515 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:09:32.509+0000 7f0ed4a2f780 1 waiting for incremental sync to catch up: 2023-05-24T21:09:32.515 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000002.19.11, , , 00000000011.44.6, 00000000002.26.11, ] 2023-05-24T21:09:32.515 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000002.19.11, , , 00000000011.44.6, 00000000003.29.6, ] 2023-05-24T21:09:37.516 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:09:37.510+0000 7f0ed4a2f780 1 waiting for incremental sync to catch up: 2023-05-24T21:09:37.517 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000002.19.11, , , 00000000011.44.6, 00000000002.26.11, ] 2023-05-24T21:09:37.517 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000002.19.11, , , 00000000011.44.6, 00000000003.29.6, ] 2023-05-24T21:09:42.518 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:09:42.511+0000 7f0ed4a2f780 1 waiting for incremental sync to catch up: 2023-05-24T21:09:42.518 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000002.19.11, , , 00000000011.44.6, 00000000002.26.11, ] 2023-05-24T21:09:42.518 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000002.19.11, , , 00000000011.44.6, 00000000003.29.6, ] 2023-05-24T21:09:47.519 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:09:47.512+0000 7f0ed4a2f780 1 waiting for incremental sync to catch up: 2023-05-24T21:09:47.520 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000002.19.11, , , 00000000011.44.6, 00000000002.26.11, ] 2023-05-24T21:09:47.520 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000002.19.11, , , 00000000011.44.6, 00000000003.29.6, ] 2023-05-24T21:09:52.520 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:09:52.514+0000 7f0ed4a2f780 1 bucket sync caught up with source: 2023-05-24T21:09:52.521 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000002.19.11, , , 00000000011.44.6, 00000000003.29.6, ] 2023-05-24T21:09:52.521 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000002.19.11, , , 00000000011.44.6, 00000000003.29.6, ] 2023-05-24T21:09:52.521 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:09:52.514+0000 7f0ed4a2f780 0 bucket checkpoint complete 2023-05-24T21:09:52.546 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qskrps-11 zones={a2, a1} 2023-05-24T21:09:52.546 DEBUG:boto:path=/qskrps-11/ 2023-05-24T21:09:52.547 DEBUG:boto:auth_path=/qskrps-11/ 2023-05-24T21:09:52.547 DEBUG:boto:Method: HEAD 2023-05-24T21:09:52.547 DEBUG:boto:Path: /qskrps-11/ 2023-05-24T21:09:52.548 DEBUG:boto:Data: 2023-05-24T21:09:52.548 DEBUG:boto:Headers: {} 2023-05-24T21:09:52.549 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:52.549 DEBUG:boto:Port: 8001 2023-05-24T21:09:52.549 DEBUG:boto:Params: {} 2023-05-24T21:09:52.550 DEBUG:boto:Token: None 2023-05-24T21:09:52.550 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:52 GMT /qskrps-11/ 2023-05-24T21:09:52.551 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ERj/cuxVpSM5qFbibhR8yc02EEA= 2023-05-24T21:09:52.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ERj/cuxVpSM5qFbibhR8yc02EEA=', 'Content-Length': '0'} 2023-05-24T21:09:52.554 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000043e50e74569cd5c4-00646e7d20-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:52.554 DEBUG:boto:path=/qskrps-11/ 2023-05-24T21:09:52.554 DEBUG:boto:auth_path=/qskrps-11/ 2023-05-24T21:09:52.555 DEBUG:boto:Method: HEAD 2023-05-24T21:09:52.555 DEBUG:boto:Path: /qskrps-11/ 2023-05-24T21:09:52.555 DEBUG:boto:Data: 2023-05-24T21:09:52.556 DEBUG:boto:Headers: {} 2023-05-24T21:09:52.556 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:52.557 DEBUG:boto:Port: 8000 2023-05-24T21:09:52.557 DEBUG:boto:Params: {} 2023-05-24T21:09:52.557 DEBUG:boto:Token: None 2023-05-24T21:09:52.558 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:52 GMT /qskrps-11/ 2023-05-24T21:09:52.558 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ERj/cuxVpSM5qFbibhR8yc02EEA= 2023-05-24T21:09:52.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ERj/cuxVpSM5qFbibhR8yc02EEA=', 'Content-Length': '0'} 2023-05-24T21:09:52.561 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d44f0bd2d1e9ce41-00646e7d20-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:52.561 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-24T21:09:52.562 DEBUG:boto:path=/qskrps-11/ 2023-05-24T21:09:52.562 DEBUG:boto:auth_path=/qskrps-11/ 2023-05-24T21:09:52.562 DEBUG:boto:path=/qskrps-11/?versions&max-keys=999 2023-05-24T21:09:52.563 DEBUG:boto:auth_path=/qskrps-11/?versions&max-keys=999 2023-05-24T21:09:52.563 DEBUG:boto:Method: GET 2023-05-24T21:09:52.564 DEBUG:boto:Path: /qskrps-11/?versions&max-keys=999 2023-05-24T21:09:52.564 DEBUG:boto:Data: 2023-05-24T21:09:52.565 DEBUG:boto:Headers: {} 2023-05-24T21:09:52.566 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:52.567 DEBUG:boto:Port: 8001 2023-05-24T21:09:52.567 DEBUG:boto:Params: {} 2023-05-24T21:09:52.568 DEBUG:boto:Token: None 2023-05-24T21:09:52.569 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:52 GMT /qskrps-11/?versions 2023-05-24T21:09:52.570 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ZCOim4i22BXW/WRWqBBKNQiFpPo= 2023-05-24T21:09:52.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ZCOim4i22BXW/WRWqBBKNQiFpPo=', 'Content-Length': '0'} 2023-05-24T21:09:52.573 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f4081b58b080f81-00646e7d20-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:52.574 DEBUG:boto:b'qskrps-11999falsedummyK0LgvK2trkm-uHhxLikH3sxyZSZDazNtrue2023-05-24T21:07:39.470Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a154a54vsjHFa46aVpWzOVMoAJ0PQA-.vtrue2023-05-24T21:08:14.786Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2YniDC1out7TFV1sI.TLfuqiPcVFd.5Strue2023-05-24T21:08:14.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:52.575 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-05-24T21:09:52.575 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-05-24T21:09:52.575 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-05-24T21:09:52.576 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-24T21:09:52.576 DEBUG:boto:path=/qskrps-11/ 2023-05-24T21:09:52.576 DEBUG:boto:auth_path=/qskrps-11/ 2023-05-24T21:09:52.577 DEBUG:boto:path=/qskrps-11/?versions&max-keys=999 2023-05-24T21:09:52.577 DEBUG:boto:auth_path=/qskrps-11/?versions&max-keys=999 2023-05-24T21:09:52.577 DEBUG:boto:Method: GET 2023-05-24T21:09:52.578 DEBUG:boto:Path: /qskrps-11/?versions&max-keys=999 2023-05-24T21:09:52.578 DEBUG:boto:Data: 2023-05-24T21:09:52.579 DEBUG:boto:Headers: {} 2023-05-24T21:09:52.579 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:52.579 DEBUG:boto:Port: 8000 2023-05-24T21:09:52.580 DEBUG:boto:Params: {} 2023-05-24T21:09:52.580 DEBUG:boto:Token: None 2023-05-24T21:09:52.581 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:52 GMT /qskrps-11/?versions 2023-05-24T21:09:52.581 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ZCOim4i22BXW/WRWqBBKNQiFpPo= 2023-05-24T21:09:52.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ZCOim4i22BXW/WRWqBBKNQiFpPo=', 'Content-Length': '0'} 2023-05-24T21:09:52.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080e907d502609257-00646e7d20-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:52.585 DEBUG:boto:b'qskrps-11999falsedummyK0LgvK2trkm-uHhxLikH3sxyZSZDazNtrue2023-05-24T21:07:39.470Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a154a54vsjHFa46aVpWzOVMoAJ0PQA-.vtrue2023-05-24T21:08:14.786Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2YniDC1out7TFV1sI.TLfuqiPcVFd.5Strue2023-05-24T21:08:14.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:52.585 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-05-24T21:09:52.586 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-05-24T21:09:52.586 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-05-24T21:09:52.586 DEBUG:boto:path=/qskrps-11/ 2023-05-24T21:09:52.587 DEBUG:boto:auth_path=/qskrps-11/ 2023-05-24T21:09:52.587 DEBUG:boto:path=/qskrps-11/?versions&max-keys=999 2023-05-24T21:09:52.588 DEBUG:boto:auth_path=/qskrps-11/?versions&max-keys=999 2023-05-24T21:09:52.588 DEBUG:boto:Method: GET 2023-05-24T21:09:52.588 DEBUG:boto:Path: /qskrps-11/?versions&max-keys=999 2023-05-24T21:09:52.589 DEBUG:boto:Data: 2023-05-24T21:09:52.589 DEBUG:boto:Headers: {} 2023-05-24T21:09:52.589 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:52.590 DEBUG:boto:Port: 8001 2023-05-24T21:09:52.590 DEBUG:boto:Params: {} 2023-05-24T21:09:52.591 DEBUG:boto:Token: None 2023-05-24T21:09:52.591 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:52 GMT /qskrps-11/?versions 2023-05-24T21:09:52.592 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ZCOim4i22BXW/WRWqBBKNQiFpPo= 2023-05-24T21:09:52.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ZCOim4i22BXW/WRWqBBKNQiFpPo=', 'Content-Length': '0'} 2023-05-24T21:09:52.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077e6065d5dc172c1-00646e7d20-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:52.596 DEBUG:boto:b'qskrps-11999falsedummyK0LgvK2trkm-uHhxLikH3sxyZSZDazNtrue2023-05-24T21:07:39.470Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a154a54vsjHFa46aVpWzOVMoAJ0PQA-.vtrue2023-05-24T21:08:14.786Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2YniDC1out7TFV1sI.TLfuqiPcVFd.5Strue2023-05-24T21:08:14.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:52.597 DEBUG:boto:path=/qskrps-11/ 2023-05-24T21:09:52.597 DEBUG:boto:auth_path=/qskrps-11/ 2023-05-24T21:09:52.598 DEBUG:boto:path=/qskrps-11/?versions&max-keys=999 2023-05-24T21:09:52.598 DEBUG:boto:auth_path=/qskrps-11/?versions&max-keys=999 2023-05-24T21:09:52.598 DEBUG:boto:Method: GET 2023-05-24T21:09:52.599 DEBUG:boto:Path: /qskrps-11/?versions&max-keys=999 2023-05-24T21:09:52.599 DEBUG:boto:Data: 2023-05-24T21:09:52.600 DEBUG:boto:Headers: {} 2023-05-24T21:09:52.600 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:52.601 DEBUG:boto:Port: 8000 2023-05-24T21:09:52.601 DEBUG:boto:Params: {} 2023-05-24T21:09:52.602 DEBUG:boto:Token: None 2023-05-24T21:09:52.603 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:52 GMT /qskrps-11/?versions 2023-05-24T21:09:52.603 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ZCOim4i22BXW/WRWqBBKNQiFpPo= 2023-05-24T21:09:52.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ZCOim4i22BXW/WRWqBBKNQiFpPo=', 'Content-Length': '0'} 2023-05-24T21:09:52.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7fd2b9c3c827aa7-00646e7d20-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:52.607 DEBUG:boto:b'qskrps-11999falsedummyK0LgvK2trkm-uHhxLikH3sxyZSZDazNtrue2023-05-24T21:07:39.470Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a154a54vsjHFa46aVpWzOVMoAJ0PQA-.vtrue2023-05-24T21:08:14.786Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2YniDC1out7TFV1sI.TLfuqiPcVFd.5Strue2023-05-24T21:08:14.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:52.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-05-24T21:09:52.608 DEBUG:boto:path=/qskrps-11/dummy 2023-05-24T21:09:52.609 DEBUG:boto:auth_path=/qskrps-11/dummy 2023-05-24T21:09:52.609 DEBUG:boto:path=/qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:09:52.609 DEBUG:boto:auth_path=/qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:09:52.610 DEBUG:boto:Method: GET 2023-05-24T21:09:52.610 DEBUG:boto:Path: /qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:09:52.611 DEBUG:boto:Data: 2023-05-24T21:09:52.611 DEBUG:boto:Headers: {} 2023-05-24T21:09:52.612 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:52.612 DEBUG:boto:Port: 8001 2023-05-24T21:09:52.612 DEBUG:boto:Params: {} 2023-05-24T21:09:52.613 DEBUG:boto:Token: None 2023-05-24T21:09:52.613 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:52 GMT /qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:09:52.614 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:WFl533bpo7BXPVBFZZneD7xY7Qo= 2023-05-24T21:09:52.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WFl533bpo7BXPVBFZZneD7xY7Qo=', 'Content-Length': '0'} 2023-05-24T21:09:52.616 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:07:39 GMT'), ('x-amz-version-id', 'K0LgvK2trkm-uHhxLikH3sxyZSZDazN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e555fa7aca0ad448-00646e7d20-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:52.617 DEBUG:boto:path=/qskrps-11/dummy 2023-05-24T21:09:52.617 DEBUG:boto:auth_path=/qskrps-11/dummy 2023-05-24T21:09:52.618 DEBUG:boto:path=/qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:09:52.618 DEBUG:boto:auth_path=/qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:09:52.619 DEBUG:boto:Method: GET 2023-05-24T21:09:52.619 DEBUG:boto:Path: /qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:09:52.620 DEBUG:boto:Data: 2023-05-24T21:09:52.620 DEBUG:boto:Headers: {} 2023-05-24T21:09:52.620 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:52.621 DEBUG:boto:Port: 8000 2023-05-24T21:09:52.621 DEBUG:boto:Params: {} 2023-05-24T21:09:52.622 DEBUG:boto:Token: None 2023-05-24T21:09:52.622 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:52 GMT /qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:09:52.623 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:WFl533bpo7BXPVBFZZneD7xY7Qo= 2023-05-24T21:09:52.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WFl533bpo7BXPVBFZZneD7xY7Qo=', 'Content-Length': '0'} 2023-05-24T21:09:52.625 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:07:39 GMT'), ('x-amz-version-id', 'K0LgvK2trkm-uHhxLikH3sxyZSZDazN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000601f11b3af0ca9d-00646e7d20-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:52.625 DEBUG:boto:path=/qskrps-11/dummy 2023-05-24T21:09:52.626 DEBUG:boto:auth_path=/qskrps-11/dummy 2023-05-24T21:09:52.626 DEBUG:boto:path=/qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:09:52.627 DEBUG:boto:auth_path=/qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:09:52.627 DEBUG:boto:Method: HEAD 2023-05-24T21:09:52.628 DEBUG:boto:Path: /qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:09:52.628 DEBUG:boto:Data: 2023-05-24T21:09:52.628 DEBUG:boto:Headers: {} 2023-05-24T21:09:52.629 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:52.629 DEBUG:boto:Port: 8001 2023-05-24T21:09:52.630 DEBUG:boto:Params: {} 2023-05-24T21:09:52.630 DEBUG:boto:Token: None 2023-05-24T21:09:52.631 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:52 GMT /qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:09:52.631 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:8YwQ3jfSGIRE3rI4zdmJKWBNxCw= 2023-05-24T21:09:52.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:8YwQ3jfSGIRE3rI4zdmJKWBNxCw=', 'Content-Length': '0'} 2023-05-24T21:09:52.634 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:07:39 GMT'), ('x-amz-version-id', 'K0LgvK2trkm-uHhxLikH3sxyZSZDazN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a914a0f7ee582a4f-00646e7d20-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:52.634 DEBUG:boto:path=/qskrps-11/dummy 2023-05-24T21:09:52.635 DEBUG:boto:auth_path=/qskrps-11/dummy 2023-05-24T21:09:52.635 DEBUG:boto:path=/qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:09:52.636 DEBUG:boto:auth_path=/qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:09:52.636 DEBUG:boto:Method: HEAD 2023-05-24T21:09:52.637 DEBUG:boto:Path: /qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:09:52.637 DEBUG:boto:Data: 2023-05-24T21:09:52.637 DEBUG:boto:Headers: {} 2023-05-24T21:09:52.638 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:52.638 DEBUG:boto:Port: 8000 2023-05-24T21:09:52.638 DEBUG:boto:Params: {} 2023-05-24T21:09:52.639 DEBUG:boto:Token: None 2023-05-24T21:09:52.639 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:52 GMT /qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:09:52.639 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:8YwQ3jfSGIRE3rI4zdmJKWBNxCw= 2023-05-24T21:09:52.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:8YwQ3jfSGIRE3rI4zdmJKWBNxCw=', 'Content-Length': '0'} 2023-05-24T21:09:52.642 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:07:39 GMT'), ('x-amz-version-id', 'K0LgvK2trkm-uHhxLikH3sxyZSZDazN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b413e7a73d4e72d2-00646e7d20-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:52.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-05-24T21:09:52.642 DEBUG:boto:path=/qskrps-11/dummy 2023-05-24T21:09:52.643 DEBUG:boto:auth_path=/qskrps-11/dummy 2023-05-24T21:09:52.643 DEBUG:boto:path=/qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:09:52.644 DEBUG:boto:auth_path=/qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:09:52.644 DEBUG:boto:Method: GET 2023-05-24T21:09:52.644 DEBUG:boto:Path: /qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:09:52.645 DEBUG:boto:Data: 2023-05-24T21:09:52.645 DEBUG:boto:Headers: {} 2023-05-24T21:09:52.645 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:52.646 DEBUG:boto:Port: 8001 2023-05-24T21:09:52.646 DEBUG:boto:Params: {} 2023-05-24T21:09:52.647 DEBUG:boto:Token: None 2023-05-24T21:09:52.647 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:52 GMT /qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:09:52.647 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:WFl533bpo7BXPVBFZZneD7xY7Qo= 2023-05-24T21:09:52.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WFl533bpo7BXPVBFZZneD7xY7Qo=', 'Content-Length': '0'} 2023-05-24T21:09:52.650 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:07:39 GMT'), ('x-amz-version-id', 'K0LgvK2trkm-uHhxLikH3sxyZSZDazN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000130d510a525221ba-00646e7d20-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:52.650 DEBUG:boto:path=/qskrps-11/dummy 2023-05-24T21:09:52.651 DEBUG:boto:auth_path=/qskrps-11/dummy 2023-05-24T21:09:52.651 DEBUG:boto:path=/qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:09:52.651 DEBUG:boto:auth_path=/qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:09:52.652 DEBUG:boto:Method: GET 2023-05-24T21:09:52.652 DEBUG:boto:Path: /qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:09:52.652 DEBUG:boto:Data: 2023-05-24T21:09:52.653 DEBUG:boto:Headers: {} 2023-05-24T21:09:52.653 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:52.653 DEBUG:boto:Port: 8000 2023-05-24T21:09:52.654 DEBUG:boto:Params: {} 2023-05-24T21:09:52.654 DEBUG:boto:Token: None 2023-05-24T21:09:52.655 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:52 GMT /qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:09:52.655 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:WFl533bpo7BXPVBFZZneD7xY7Qo= 2023-05-24T21:09:52.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WFl533bpo7BXPVBFZZneD7xY7Qo=', 'Content-Length': '0'} 2023-05-24T21:09:52.657 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:07:39 GMT'), ('x-amz-version-id', 'K0LgvK2trkm-uHhxLikH3sxyZSZDazN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009ea95ae0067794af-00646e7d20-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:52.658 DEBUG:boto:path=/qskrps-11/ 2023-05-24T21:09:52.658 DEBUG:boto:auth_path=/qskrps-11/ 2023-05-24T21:09:52.659 DEBUG:boto:path=/qskrps-11/?prefix=dummy 2023-05-24T21:09:52.659 DEBUG:boto:auth_path=/qskrps-11/?prefix=dummy 2023-05-24T21:09:52.659 DEBUG:boto:Method: GET 2023-05-24T21:09:52.660 DEBUG:boto:Path: /qskrps-11/?prefix=dummy 2023-05-24T21:09:52.660 DEBUG:boto:Data: 2023-05-24T21:09:52.660 DEBUG:boto:Headers: {} 2023-05-24T21:09:52.661 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:52.661 DEBUG:boto:Port: 8001 2023-05-24T21:09:52.661 DEBUG:boto:Params: {} 2023-05-24T21:09:52.662 DEBUG:boto:Token: None 2023-05-24T21:09:52.662 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:52 GMT /qskrps-11/ 2023-05-24T21:09:52.663 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:WOCIs7+w7kza/qojJ2xVtP+fH7g= 2023-05-24T21:09:52.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WOCIs7+w7kza/qojJ2xVtP+fH7g=', 'Content-Length': '0'} 2023-05-24T21:09:52.665 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfc5d2fb6a292538-00646e7d20-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:52.708 DEBUG:boto:b'qskrps-11dummy1000falsedummy2023-05-24T21:07:39.470Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:52.710 DEBUG:boto:path=/qskrps-11/ 2023-05-24T21:09:52.710 DEBUG:boto:auth_path=/qskrps-11/ 2023-05-24T21:09:52.711 DEBUG:boto:path=/qskrps-11/?prefix=dummy 2023-05-24T21:09:52.711 DEBUG:boto:auth_path=/qskrps-11/?prefix=dummy 2023-05-24T21:09:52.712 DEBUG:boto:Method: GET 2023-05-24T21:09:52.713 DEBUG:boto:Path: /qskrps-11/?prefix=dummy 2023-05-24T21:09:52.713 DEBUG:boto:Data: 2023-05-24T21:09:52.714 DEBUG:boto:Headers: {} 2023-05-24T21:09:52.715 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:52.715 DEBUG:boto:Port: 8000 2023-05-24T21:09:52.716 DEBUG:boto:Params: {} 2023-05-24T21:09:52.717 DEBUG:boto:Token: None 2023-05-24T21:09:52.717 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:52 GMT /qskrps-11/ 2023-05-24T21:09:52.718 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:WOCIs7+w7kza/qojJ2xVtP+fH7g= 2023-05-24T21:09:52.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WOCIs7+w7kza/qojJ2xVtP+fH7g=', 'Content-Length': '0'} 2023-05-24T21:09:52.722 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d31718a1564b2c32-00646e7d20-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:52.722 DEBUG:boto:b'qskrps-11dummy1000falsedummy2023-05-24T21:07:39.470Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:52.723 DEBUG:boto:path=/qskrps-11/dummy 2023-05-24T21:09:52.723 DEBUG:boto:auth_path=/qskrps-11/dummy 2023-05-24T21:09:52.724 DEBUG:boto:Method: HEAD 2023-05-24T21:09:52.724 DEBUG:boto:Path: /qskrps-11/dummy 2023-05-24T21:09:52.724 DEBUG:boto:Data: 2023-05-24T21:09:52.725 DEBUG:boto:Headers: {} 2023-05-24T21:09:52.725 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:52.725 DEBUG:boto:Port: 8001 2023-05-24T21:09:52.726 DEBUG:boto:Params: {} 2023-05-24T21:09:52.726 DEBUG:boto:Token: None 2023-05-24T21:09:52.726 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:52 GMT /qskrps-11/dummy 2023-05-24T21:09:52.727 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:As3EHppEPugI29VE6WBzgPpnyrA= 2023-05-24T21:09:52.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:As3EHppEPugI29VE6WBzgPpnyrA=', 'Content-Length': '0'} 2023-05-24T21:09:52.730 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:07:39 GMT'), ('x-amz-version-id', 'K0LgvK2trkm-uHhxLikH3sxyZSZDazN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bf0e506f0d4f1fe5-00646e7d20-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:52.731 DEBUG:boto:path=/qskrps-11/dummy 2023-05-24T21:09:52.731 DEBUG:boto:auth_path=/qskrps-11/dummy 2023-05-24T21:09:52.731 DEBUG:boto:Method: HEAD 2023-05-24T21:09:52.732 DEBUG:boto:Path: /qskrps-11/dummy 2023-05-24T21:09:52.732 DEBUG:boto:Data: 2023-05-24T21:09:52.732 DEBUG:boto:Headers: {} 2023-05-24T21:09:52.733 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:52.733 DEBUG:boto:Port: 8000 2023-05-24T21:09:52.733 DEBUG:boto:Params: {} 2023-05-24T21:09:52.734 DEBUG:boto:Token: None 2023-05-24T21:09:52.734 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:52 GMT /qskrps-11/dummy 2023-05-24T21:09:52.735 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:As3EHppEPugI29VE6WBzgPpnyrA= 2023-05-24T21:09:52.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:As3EHppEPugI29VE6WBzgPpnyrA=', 'Content-Length': '0'} 2023-05-24T21:09:52.737 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:07:39 GMT'), ('x-amz-version-id', 'K0LgvK2trkm-uHhxLikH3sxyZSZDazN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004c1691818afca893-00646e7d20-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:52.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-05-24T21:09:52.738 DEBUG:boto:path=/qskrps-11/dummy 2023-05-24T21:09:52.739 DEBUG:boto:auth_path=/qskrps-11/dummy 2023-05-24T21:09:52.739 DEBUG:boto:path=/qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:09:52.740 DEBUG:boto:auth_path=/qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:09:52.740 DEBUG:boto:Method: GET 2023-05-24T21:09:52.740 DEBUG:boto:Path: /qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:09:52.741 DEBUG:boto:Data: 2023-05-24T21:09:52.741 DEBUG:boto:Headers: {} 2023-05-24T21:09:52.741 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:52.742 DEBUG:boto:Port: 8001 2023-05-24T21:09:52.742 DEBUG:boto:Params: {} 2023-05-24T21:09:52.742 DEBUG:boto:Token: None 2023-05-24T21:09:52.743 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:52 GMT /qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:09:52.743 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:WFl533bpo7BXPVBFZZneD7xY7Qo= 2023-05-24T21:09:52.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WFl533bpo7BXPVBFZZneD7xY7Qo=', 'Content-Length': '0'} 2023-05-24T21:09:52.745 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:07:39 GMT'), ('x-amz-version-id', 'K0LgvK2trkm-uHhxLikH3sxyZSZDazN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002d3b38eb613fa844-00646e7d20-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:52.746 DEBUG:boto:path=/qskrps-11/dummy 2023-05-24T21:09:52.746 DEBUG:boto:auth_path=/qskrps-11/dummy 2023-05-24T21:09:52.747 DEBUG:boto:path=/qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:09:52.747 DEBUG:boto:auth_path=/qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:09:52.748 DEBUG:boto:Method: GET 2023-05-24T21:09:52.748 DEBUG:boto:Path: /qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:09:52.748 DEBUG:boto:Data: 2023-05-24T21:09:52.749 DEBUG:boto:Headers: {} 2023-05-24T21:09:52.749 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:52.749 DEBUG:boto:Port: 8000 2023-05-24T21:09:52.750 DEBUG:boto:Params: {} 2023-05-24T21:09:52.750 DEBUG:boto:Token: None 2023-05-24T21:09:52.751 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:52 GMT /qskrps-11/dummy?versionId=K0LgvK2trkm-uHhxLikH3sxyZSZDazN 2023-05-24T21:09:52.751 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:WFl533bpo7BXPVBFZZneD7xY7Qo= 2023-05-24T21:09:52.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WFl533bpo7BXPVBFZZneD7xY7Qo=', 'Content-Length': '0'} 2023-05-24T21:09:52.753 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:07:39 GMT'), ('x-amz-version-id', 'K0LgvK2trkm-uHhxLikH3sxyZSZDazN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e5914dd836d9c60c-00646e7d20-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:52.754 DEBUG:boto:path=/qskrps-11/ 2023-05-24T21:09:52.754 DEBUG:boto:auth_path=/qskrps-11/ 2023-05-24T21:09:52.755 DEBUG:boto:path=/qskrps-11/?prefix=dummy 2023-05-24T21:09:52.755 DEBUG:boto:auth_path=/qskrps-11/?prefix=dummy 2023-05-24T21:09:52.755 DEBUG:boto:Method: GET 2023-05-24T21:09:52.756 DEBUG:boto:Path: /qskrps-11/?prefix=dummy 2023-05-24T21:09:52.756 DEBUG:boto:Data: 2023-05-24T21:09:52.757 DEBUG:boto:Headers: {} 2023-05-24T21:09:52.757 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:52.757 DEBUG:boto:Port: 8001 2023-05-24T21:09:52.758 DEBUG:boto:Params: {} 2023-05-24T21:09:52.758 DEBUG:boto:Token: None 2023-05-24T21:09:52.758 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:52 GMT /qskrps-11/ 2023-05-24T21:09:52.759 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:WOCIs7+w7kza/qojJ2xVtP+fH7g= 2023-05-24T21:09:52.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WOCIs7+w7kza/qojJ2xVtP+fH7g=', 'Content-Length': '0'} 2023-05-24T21:09:52.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b990b80d3206e7ef-00646e7d20-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:52.762 DEBUG:boto:b'qskrps-11dummy1000falsedummy2023-05-24T21:07:39.470Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:52.763 DEBUG:boto:path=/qskrps-11/ 2023-05-24T21:09:52.763 DEBUG:boto:auth_path=/qskrps-11/ 2023-05-24T21:09:52.764 DEBUG:boto:path=/qskrps-11/?prefix=dummy 2023-05-24T21:09:52.764 DEBUG:boto:auth_path=/qskrps-11/?prefix=dummy 2023-05-24T21:09:52.764 DEBUG:boto:Method: GET 2023-05-24T21:09:52.765 DEBUG:boto:Path: /qskrps-11/?prefix=dummy 2023-05-24T21:09:52.765 DEBUG:boto:Data: 2023-05-24T21:09:52.765 DEBUG:boto:Headers: {} 2023-05-24T21:09:52.766 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:52.766 DEBUG:boto:Port: 8000 2023-05-24T21:09:52.766 DEBUG:boto:Params: {} 2023-05-24T21:09:52.767 DEBUG:boto:Token: None 2023-05-24T21:09:52.767 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:52 GMT /qskrps-11/ 2023-05-24T21:09:52.768 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:WOCIs7+w7kza/qojJ2xVtP+fH7g= 2023-05-24T21:09:52.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WOCIs7+w7kza/qojJ2xVtP+fH7g=', 'Content-Length': '0'} 2023-05-24T21:09:52.770 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccc5068f8352b682-00646e7d20-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:52.812 DEBUG:boto:b'qskrps-11dummy1000falsedummy2023-05-24T21:07:39.470Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:52.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-05-24T21:09:52.813 DEBUG:boto:path=/qskrps-11/obj-a1 2023-05-24T21:09:52.814 DEBUG:boto:auth_path=/qskrps-11/obj-a1 2023-05-24T21:09:52.814 DEBUG:boto:path=/qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:09:52.814 DEBUG:boto:auth_path=/qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:09:52.815 DEBUG:boto:Method: GET 2023-05-24T21:09:52.815 DEBUG:boto:Path: /qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:09:52.816 DEBUG:boto:Data: 2023-05-24T21:09:52.816 DEBUG:boto:Headers: {} 2023-05-24T21:09:52.816 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:52.817 DEBUG:boto:Port: 8001 2023-05-24T21:09:52.817 DEBUG:boto:Params: {} 2023-05-24T21:09:52.817 DEBUG:boto:Token: None 2023-05-24T21:09:52.818 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:52 GMT /qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:09:52.818 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:L9xjykaxx0G4o1EsKKu7Hp2wsMA= 2023-05-24T21:09:52.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:L9xjykaxx0G4o1EsKKu7Hp2wsMA=', 'Content-Length': '0'} 2023-05-24T21:09:52.821 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:12 GMT'), ('x-amz-version-id', '54a54vsjHFa46aVpWzOVMoAJ0PQA-.v'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008a563368193b6d80-00646e7d20-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:52.822 DEBUG:boto:path=/qskrps-11/obj-a1 2023-05-24T21:09:52.822 DEBUG:boto:auth_path=/qskrps-11/obj-a1 2023-05-24T21:09:52.822 DEBUG:boto:path=/qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:09:52.823 DEBUG:boto:auth_path=/qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:09:52.823 DEBUG:boto:Method: GET 2023-05-24T21:09:52.824 DEBUG:boto:Path: /qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:09:52.824 DEBUG:boto:Data: 2023-05-24T21:09:52.824 DEBUG:boto:Headers: {} 2023-05-24T21:09:52.825 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:52.825 DEBUG:boto:Port: 8000 2023-05-24T21:09:52.825 DEBUG:boto:Params: {} 2023-05-24T21:09:52.826 DEBUG:boto:Token: None 2023-05-24T21:09:52.826 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:52 GMT /qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:09:52.827 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:L9xjykaxx0G4o1EsKKu7Hp2wsMA= 2023-05-24T21:09:52.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:L9xjykaxx0G4o1EsKKu7Hp2wsMA=', 'Content-Length': '0'} 2023-05-24T21:09:52.829 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:12 GMT'), ('x-amz-version-id', '54a54vsjHFa46aVpWzOVMoAJ0PQA-.v'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000fdd4341cf608900-00646e7d20-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:52.830 DEBUG:boto:path=/qskrps-11/obj-a1 2023-05-24T21:09:52.830 DEBUG:boto:auth_path=/qskrps-11/obj-a1 2023-05-24T21:09:52.830 DEBUG:boto:path=/qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:09:52.831 DEBUG:boto:auth_path=/qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:09:52.831 DEBUG:boto:Method: HEAD 2023-05-24T21:09:52.831 DEBUG:boto:Path: /qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:09:52.832 DEBUG:boto:Data: 2023-05-24T21:09:52.832 DEBUG:boto:Headers: {} 2023-05-24T21:09:52.832 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:52.833 DEBUG:boto:Port: 8001 2023-05-24T21:09:52.833 DEBUG:boto:Params: {} 2023-05-24T21:09:52.833 DEBUG:boto:Token: None 2023-05-24T21:09:52.834 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:52 GMT /qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:09:52.834 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:yV+zMnL8DQsMlG5xfsnYpDxi3r0= 2023-05-24T21:09:52.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:yV+zMnL8DQsMlG5xfsnYpDxi3r0=', 'Content-Length': '0'} 2023-05-24T21:09:52.836 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:12 GMT'), ('x-amz-version-id', '54a54vsjHFa46aVpWzOVMoAJ0PQA-.v'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f65c46eb6efad169-00646e7d20-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:52.837 DEBUG:boto:path=/qskrps-11/obj-a1 2023-05-24T21:09:52.837 DEBUG:boto:auth_path=/qskrps-11/obj-a1 2023-05-24T21:09:52.838 DEBUG:boto:path=/qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:09:52.838 DEBUG:boto:auth_path=/qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:09:52.838 DEBUG:boto:Method: HEAD 2023-05-24T21:09:52.839 DEBUG:boto:Path: /qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:09:52.839 DEBUG:boto:Data: 2023-05-24T21:09:52.839 DEBUG:boto:Headers: {} 2023-05-24T21:09:52.840 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:52.840 DEBUG:boto:Port: 8000 2023-05-24T21:09:52.840 DEBUG:boto:Params: {} 2023-05-24T21:09:52.841 DEBUG:boto:Token: None 2023-05-24T21:09:52.841 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:52 GMT /qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:09:52.841 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:yV+zMnL8DQsMlG5xfsnYpDxi3r0= 2023-05-24T21:09:52.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:yV+zMnL8DQsMlG5xfsnYpDxi3r0=', 'Content-Length': '0'} 2023-05-24T21:09:52.844 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:12 GMT'), ('x-amz-version-id', '54a54vsjHFa46aVpWzOVMoAJ0PQA-.v'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004053808d52ec9855-00646e7d20-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:52.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-05-24T21:09:52.845 DEBUG:boto:path=/qskrps-11/obj-a1 2023-05-24T21:09:52.845 DEBUG:boto:auth_path=/qskrps-11/obj-a1 2023-05-24T21:09:52.845 DEBUG:boto:path=/qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:09:52.846 DEBUG:boto:auth_path=/qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:09:52.846 DEBUG:boto:Method: GET 2023-05-24T21:09:52.846 DEBUG:boto:Path: /qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:09:52.847 DEBUG:boto:Data: 2023-05-24T21:09:52.847 DEBUG:boto:Headers: {} 2023-05-24T21:09:52.847 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:52.848 DEBUG:boto:Port: 8001 2023-05-24T21:09:52.848 DEBUG:boto:Params: {} 2023-05-24T21:09:52.849 DEBUG:boto:Token: None 2023-05-24T21:09:52.849 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:52 GMT /qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:09:52.849 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:L9xjykaxx0G4o1EsKKu7Hp2wsMA= 2023-05-24T21:09:52.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:L9xjykaxx0G4o1EsKKu7Hp2wsMA=', 'Content-Length': '0'} 2023-05-24T21:09:52.852 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:12 GMT'), ('x-amz-version-id', '54a54vsjHFa46aVpWzOVMoAJ0PQA-.v'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005592cfcb5cc691b2-00646e7d20-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:52.896 DEBUG:boto:path=/qskrps-11/obj-a1 2023-05-24T21:09:52.897 DEBUG:boto:auth_path=/qskrps-11/obj-a1 2023-05-24T21:09:52.897 DEBUG:boto:path=/qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:09:52.898 DEBUG:boto:auth_path=/qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:09:52.898 DEBUG:boto:Method: GET 2023-05-24T21:09:52.898 DEBUG:boto:Path: /qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:09:52.899 DEBUG:boto:Data: 2023-05-24T21:09:52.899 DEBUG:boto:Headers: {} 2023-05-24T21:09:52.899 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:52.900 DEBUG:boto:Port: 8000 2023-05-24T21:09:52.900 DEBUG:boto:Params: {} 2023-05-24T21:09:52.900 DEBUG:boto:Token: None 2023-05-24T21:09:52.901 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:52 GMT /qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:09:52.901 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:L9xjykaxx0G4o1EsKKu7Hp2wsMA= 2023-05-24T21:09:52.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:L9xjykaxx0G4o1EsKKu7Hp2wsMA=', 'Content-Length': '0'} 2023-05-24T21:09:52.904 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:12 GMT'), ('x-amz-version-id', '54a54vsjHFa46aVpWzOVMoAJ0PQA-.v'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bd3024f852d8a894-00646e7d20-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:52.905 DEBUG:boto:path=/qskrps-11/ 2023-05-24T21:09:52.905 DEBUG:boto:auth_path=/qskrps-11/ 2023-05-24T21:09:52.905 DEBUG:boto:path=/qskrps-11/?prefix=obj-a1 2023-05-24T21:09:52.906 DEBUG:boto:auth_path=/qskrps-11/?prefix=obj-a1 2023-05-24T21:09:52.906 DEBUG:boto:Method: GET 2023-05-24T21:09:52.906 DEBUG:boto:Path: /qskrps-11/?prefix=obj-a1 2023-05-24T21:09:52.907 DEBUG:boto:Data: 2023-05-24T21:09:52.907 DEBUG:boto:Headers: {} 2023-05-24T21:09:52.907 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:52.908 DEBUG:boto:Port: 8001 2023-05-24T21:09:52.908 DEBUG:boto:Params: {} 2023-05-24T21:09:52.909 DEBUG:boto:Token: None 2023-05-24T21:09:52.909 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:52 GMT /qskrps-11/ 2023-05-24T21:09:52.909 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:WOCIs7+w7kza/qojJ2xVtP+fH7g= 2023-05-24T21:09:52.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WOCIs7+w7kza/qojJ2xVtP+fH7g=', 'Content-Length': '0'} 2023-05-24T21:09:52.912 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b68958618bf0391b-00646e7d20-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:52.913 DEBUG:boto:b'qskrps-11obj-a11000falseobj-a12023-05-24T21:08:14.786Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:52.913 DEBUG:boto:path=/qskrps-11/ 2023-05-24T21:09:52.914 DEBUG:boto:auth_path=/qskrps-11/ 2023-05-24T21:09:52.914 DEBUG:boto:path=/qskrps-11/?prefix=obj-a1 2023-05-24T21:09:52.915 DEBUG:boto:auth_path=/qskrps-11/?prefix=obj-a1 2023-05-24T21:09:52.915 DEBUG:boto:Method: GET 2023-05-24T21:09:52.915 DEBUG:boto:Path: /qskrps-11/?prefix=obj-a1 2023-05-24T21:09:52.916 DEBUG:boto:Data: 2023-05-24T21:09:52.916 DEBUG:boto:Headers: {} 2023-05-24T21:09:52.916 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:52.917 DEBUG:boto:Port: 8000 2023-05-24T21:09:52.917 DEBUG:boto:Params: {} 2023-05-24T21:09:52.917 DEBUG:boto:Token: None 2023-05-24T21:09:52.918 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:52 GMT /qskrps-11/ 2023-05-24T21:09:52.918 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:WOCIs7+w7kza/qojJ2xVtP+fH7g= 2023-05-24T21:09:52.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WOCIs7+w7kza/qojJ2xVtP+fH7g=', 'Content-Length': '0'} 2023-05-24T21:09:52.921 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048e54bdb681a9151-00646e7d20-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:52.921 DEBUG:boto:b'qskrps-11obj-a11000falseobj-a12023-05-24T21:08:14.786Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:52.922 DEBUG:boto:path=/qskrps-11/obj-a1 2023-05-24T21:09:52.922 DEBUG:boto:auth_path=/qskrps-11/obj-a1 2023-05-24T21:09:52.923 DEBUG:boto:Method: HEAD 2023-05-24T21:09:52.923 DEBUG:boto:Path: /qskrps-11/obj-a1 2023-05-24T21:09:52.923 DEBUG:boto:Data: 2023-05-24T21:09:52.924 DEBUG:boto:Headers: {} 2023-05-24T21:09:52.924 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:52.924 DEBUG:boto:Port: 8001 2023-05-24T21:09:52.925 DEBUG:boto:Params: {} 2023-05-24T21:09:52.925 DEBUG:boto:Token: None 2023-05-24T21:09:52.925 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:52 GMT /qskrps-11/obj-a1 2023-05-24T21:09:52.926 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:IEzPEdJkpqWRac/UZuX6BGzbmQI= 2023-05-24T21:09:52.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:IEzPEdJkpqWRac/UZuX6BGzbmQI=', 'Content-Length': '0'} 2023-05-24T21:09:52.928 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:12 GMT'), ('x-amz-version-id', '54a54vsjHFa46aVpWzOVMoAJ0PQA-.v'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000c0af906989a9d61-00646e7d20-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:52.929 DEBUG:boto:path=/qskrps-11/obj-a1 2023-05-24T21:09:52.929 DEBUG:boto:auth_path=/qskrps-11/obj-a1 2023-05-24T21:09:52.930 DEBUG:boto:Method: HEAD 2023-05-24T21:09:52.930 DEBUG:boto:Path: /qskrps-11/obj-a1 2023-05-24T21:09:52.930 DEBUG:boto:Data: 2023-05-24T21:09:52.931 DEBUG:boto:Headers: {} 2023-05-24T21:09:52.931 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:52.931 DEBUG:boto:Port: 8000 2023-05-24T21:09:52.932 DEBUG:boto:Params: {} 2023-05-24T21:09:52.932 DEBUG:boto:Token: None 2023-05-24T21:09:52.932 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:52 GMT /qskrps-11/obj-a1 2023-05-24T21:09:52.933 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:IEzPEdJkpqWRac/UZuX6BGzbmQI= 2023-05-24T21:09:52.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:IEzPEdJkpqWRac/UZuX6BGzbmQI=', 'Content-Length': '0'} 2023-05-24T21:09:52.935 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:12 GMT'), ('x-amz-version-id', '54a54vsjHFa46aVpWzOVMoAJ0PQA-.v'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000049ccee8d33506e0d-00646e7d20-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:52.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-05-24T21:09:52.936 DEBUG:boto:path=/qskrps-11/obj-a1 2023-05-24T21:09:52.937 DEBUG:boto:auth_path=/qskrps-11/obj-a1 2023-05-24T21:09:52.937 DEBUG:boto:path=/qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:09:52.937 DEBUG:boto:auth_path=/qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:09:52.938 DEBUG:boto:Method: GET 2023-05-24T21:09:52.938 DEBUG:boto:Path: /qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:09:52.939 DEBUG:boto:Data: 2023-05-24T21:09:52.939 DEBUG:boto:Headers: {} 2023-05-24T21:09:52.939 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:52.940 DEBUG:boto:Port: 8001 2023-05-24T21:09:52.940 DEBUG:boto:Params: {} 2023-05-24T21:09:52.940 DEBUG:boto:Token: None 2023-05-24T21:09:52.941 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:52 GMT /qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:09:52.941 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:L9xjykaxx0G4o1EsKKu7Hp2wsMA= 2023-05-24T21:09:52.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:L9xjykaxx0G4o1EsKKu7Hp2wsMA=', 'Content-Length': '0'} 2023-05-24T21:09:52.943 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:12 GMT'), ('x-amz-version-id', '54a54vsjHFa46aVpWzOVMoAJ0PQA-.v'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e06ed3b0165ff938-00646e7d20-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:52.984 DEBUG:boto:path=/qskrps-11/obj-a1 2023-05-24T21:09:52.985 DEBUG:boto:auth_path=/qskrps-11/obj-a1 2023-05-24T21:09:52.985 DEBUG:boto:path=/qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:09:52.985 DEBUG:boto:auth_path=/qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:09:52.986 DEBUG:boto:Method: GET 2023-05-24T21:09:52.986 DEBUG:boto:Path: /qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:09:52.987 DEBUG:boto:Data: 2023-05-24T21:09:52.987 DEBUG:boto:Headers: {} 2023-05-24T21:09:52.987 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:52.988 DEBUG:boto:Port: 8000 2023-05-24T21:09:52.988 DEBUG:boto:Params: {} 2023-05-24T21:09:52.988 DEBUG:boto:Token: None 2023-05-24T21:09:52.989 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:52 GMT /qskrps-11/obj-a1?versionId=54a54vsjHFa46aVpWzOVMoAJ0PQA-.v 2023-05-24T21:09:52.989 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:L9xjykaxx0G4o1EsKKu7Hp2wsMA= 2023-05-24T21:09:52.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:L9xjykaxx0G4o1EsKKu7Hp2wsMA=', 'Content-Length': '0'} 2023-05-24T21:09:52.992 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:12 GMT'), ('x-amz-version-id', '54a54vsjHFa46aVpWzOVMoAJ0PQA-.v'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008898dfebb6c8943a-00646e7d20-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:52.993 DEBUG:boto:path=/qskrps-11/ 2023-05-24T21:09:52.993 DEBUG:boto:auth_path=/qskrps-11/ 2023-05-24T21:09:52.993 DEBUG:boto:path=/qskrps-11/?prefix=obj-a1 2023-05-24T21:09:52.994 DEBUG:boto:auth_path=/qskrps-11/?prefix=obj-a1 2023-05-24T21:09:52.994 DEBUG:boto:Method: GET 2023-05-24T21:09:52.994 DEBUG:boto:Path: /qskrps-11/?prefix=obj-a1 2023-05-24T21:09:52.995 DEBUG:boto:Data: 2023-05-24T21:09:52.995 DEBUG:boto:Headers: {} 2023-05-24T21:09:52.995 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:52.996 DEBUG:boto:Port: 8001 2023-05-24T21:09:52.996 DEBUG:boto:Params: {} 2023-05-24T21:09:52.997 DEBUG:boto:Token: None 2023-05-24T21:09:52.997 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:52 GMT /qskrps-11/ 2023-05-24T21:09:52.997 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:WOCIs7+w7kza/qojJ2xVtP+fH7g= 2023-05-24T21:09:52.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WOCIs7+w7kza/qojJ2xVtP+fH7g=', 'Content-Length': '0'} 2023-05-24T21:09:53.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000436cf87040a26c21-00646e7d20-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.001 DEBUG:boto:b'qskrps-11obj-a11000falseobj-a12023-05-24T21:08:14.786Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:53.002 DEBUG:boto:path=/qskrps-11/ 2023-05-24T21:09:53.002 DEBUG:boto:auth_path=/qskrps-11/ 2023-05-24T21:09:53.002 DEBUG:boto:path=/qskrps-11/?prefix=obj-a1 2023-05-24T21:09:53.003 DEBUG:boto:auth_path=/qskrps-11/?prefix=obj-a1 2023-05-24T21:09:53.003 DEBUG:boto:Method: GET 2023-05-24T21:09:53.003 DEBUG:boto:Path: /qskrps-11/?prefix=obj-a1 2023-05-24T21:09:53.004 DEBUG:boto:Data: 2023-05-24T21:09:53.004 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.005 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:53.005 DEBUG:boto:Port: 8000 2023-05-24T21:09:53.005 DEBUG:boto:Params: {} 2023-05-24T21:09:53.006 DEBUG:boto:Token: None 2023-05-24T21:09:53.006 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:53 GMT /qskrps-11/ 2023-05-24T21:09:53.007 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:8ba+Lk5je6c9IpfhfpUCGl1L9qQ= 2023-05-24T21:09:53.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:8ba+Lk5je6c9IpfhfpUCGl1L9qQ=', 'Content-Length': '0'} 2023-05-24T21:09:53.009 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001acb506248e8011b-00646e7d21-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.010 DEBUG:boto:b'qskrps-11obj-a11000falseobj-a12023-05-24T21:08:14.786Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:53.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-05-24T21:09:53.011 DEBUG:boto:path=/qskrps-11/obj-a2 2023-05-24T21:09:53.011 DEBUG:boto:auth_path=/qskrps-11/obj-a2 2023-05-24T21:09:53.012 DEBUG:boto:path=/qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:09:53.012 DEBUG:boto:auth_path=/qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:09:53.012 DEBUG:boto:Method: GET 2023-05-24T21:09:53.013 DEBUG:boto:Path: /qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:09:53.013 DEBUG:boto:Data: 2023-05-24T21:09:53.013 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.014 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:53.014 DEBUG:boto:Port: 8001 2023-05-24T21:09:53.014 DEBUG:boto:Params: {} 2023-05-24T21:09:53.015 DEBUG:boto:Token: None 2023-05-24T21:09:53.015 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:53 GMT /qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:09:53.016 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:wkN58CO9nnf6U7tiB0XtmYvUn4o= 2023-05-24T21:09:53.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wkN58CO9nnf6U7tiB0XtmYvUn4o=', 'Content-Length': '0'} 2023-05-24T21:09:53.018 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:12 GMT'), ('x-amz-version-id', 'YniDC1out7TFV1sI.TLfuqiPcVFd.5S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007cdc605c5d04f88e-00646e7d21-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.019 DEBUG:boto:path=/qskrps-11/obj-a2 2023-05-24T21:09:53.019 DEBUG:boto:auth_path=/qskrps-11/obj-a2 2023-05-24T21:09:53.020 DEBUG:boto:path=/qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:09:53.020 DEBUG:boto:auth_path=/qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:09:53.020 DEBUG:boto:Method: GET 2023-05-24T21:09:53.021 DEBUG:boto:Path: /qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:09:53.021 DEBUG:boto:Data: 2023-05-24T21:09:53.021 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.022 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:53.022 DEBUG:boto:Port: 8000 2023-05-24T21:09:53.022 DEBUG:boto:Params: {} 2023-05-24T21:09:53.023 DEBUG:boto:Token: None 2023-05-24T21:09:53.023 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:53 GMT /qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:09:53.024 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:wkN58CO9nnf6U7tiB0XtmYvUn4o= 2023-05-24T21:09:53.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wkN58CO9nnf6U7tiB0XtmYvUn4o=', 'Content-Length': '0'} 2023-05-24T21:09:53.026 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:12 GMT'), ('x-amz-version-id', 'YniDC1out7TFV1sI.TLfuqiPcVFd.5S'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000041b4dc8e8b25f6af-00646e7d21-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.027 DEBUG:boto:path=/qskrps-11/obj-a2 2023-05-24T21:09:53.027 DEBUG:boto:auth_path=/qskrps-11/obj-a2 2023-05-24T21:09:53.027 DEBUG:boto:path=/qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:09:53.028 DEBUG:boto:auth_path=/qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:09:53.028 DEBUG:boto:Method: HEAD 2023-05-24T21:09:53.029 DEBUG:boto:Path: /qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:09:53.029 DEBUG:boto:Data: 2023-05-24T21:09:53.029 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.030 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:53.030 DEBUG:boto:Port: 8001 2023-05-24T21:09:53.030 DEBUG:boto:Params: {} 2023-05-24T21:09:53.031 DEBUG:boto:Token: None 2023-05-24T21:09:53.031 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:53 GMT /qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:09:53.031 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:/voMJ4n9LTCx/WqKxMpdfRadn0s= 2023-05-24T21:09:53.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:/voMJ4n9LTCx/WqKxMpdfRadn0s=', 'Content-Length': '0'} 2023-05-24T21:09:53.033 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:12 GMT'), ('x-amz-version-id', 'YniDC1out7TFV1sI.TLfuqiPcVFd.5S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000590ce60ad773821e-00646e7d21-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.034 DEBUG:boto:path=/qskrps-11/obj-a2 2023-05-24T21:09:53.034 DEBUG:boto:auth_path=/qskrps-11/obj-a2 2023-05-24T21:09:53.035 DEBUG:boto:path=/qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:09:53.035 DEBUG:boto:auth_path=/qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:09:53.035 DEBUG:boto:Method: HEAD 2023-05-24T21:09:53.036 DEBUG:boto:Path: /qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:09:53.036 DEBUG:boto:Data: 2023-05-24T21:09:53.036 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.037 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:53.037 DEBUG:boto:Port: 8000 2023-05-24T21:09:53.037 DEBUG:boto:Params: {} 2023-05-24T21:09:53.038 DEBUG:boto:Token: None 2023-05-24T21:09:53.038 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:53 GMT /qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:09:53.039 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:/voMJ4n9LTCx/WqKxMpdfRadn0s= 2023-05-24T21:09:53.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:/voMJ4n9LTCx/WqKxMpdfRadn0s=', 'Content-Length': '0'} 2023-05-24T21:09:53.041 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:12 GMT'), ('x-amz-version-id', 'YniDC1out7TFV1sI.TLfuqiPcVFd.5S'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ffe930a44d6ac11-00646e7d21-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-05-24T21:09:53.042 DEBUG:boto:path=/qskrps-11/obj-a2 2023-05-24T21:09:53.042 DEBUG:boto:auth_path=/qskrps-11/obj-a2 2023-05-24T21:09:53.042 DEBUG:boto:path=/qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:09:53.043 DEBUG:boto:auth_path=/qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:09:53.043 DEBUG:boto:Method: GET 2023-05-24T21:09:53.043 DEBUG:boto:Path: /qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:09:53.044 DEBUG:boto:Data: 2023-05-24T21:09:53.044 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.044 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:53.045 DEBUG:boto:Port: 8001 2023-05-24T21:09:53.045 DEBUG:boto:Params: {} 2023-05-24T21:09:53.046 DEBUG:boto:Token: None 2023-05-24T21:09:53.046 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:53 GMT /qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:09:53.046 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:wkN58CO9nnf6U7tiB0XtmYvUn4o= 2023-05-24T21:09:53.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wkN58CO9nnf6U7tiB0XtmYvUn4o=', 'Content-Length': '0'} 2023-05-24T21:09:53.049 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:12 GMT'), ('x-amz-version-id', 'YniDC1out7TFV1sI.TLfuqiPcVFd.5S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e0e337eb476a941-00646e7d21-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.049 DEBUG:boto:path=/qskrps-11/obj-a2 2023-05-24T21:09:53.050 DEBUG:boto:auth_path=/qskrps-11/obj-a2 2023-05-24T21:09:53.050 DEBUG:boto:path=/qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:09:53.050 DEBUG:boto:auth_path=/qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:09:53.051 DEBUG:boto:Method: GET 2023-05-24T21:09:53.051 DEBUG:boto:Path: /qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:09:53.051 DEBUG:boto:Data: 2023-05-24T21:09:53.052 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.052 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:53.053 DEBUG:boto:Port: 8000 2023-05-24T21:09:53.053 DEBUG:boto:Params: {} 2023-05-24T21:09:53.053 DEBUG:boto:Token: None 2023-05-24T21:09:53.054 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:53 GMT /qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:09:53.054 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:wkN58CO9nnf6U7tiB0XtmYvUn4o= 2023-05-24T21:09:53.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wkN58CO9nnf6U7tiB0XtmYvUn4o=', 'Content-Length': '0'} 2023-05-24T21:09:53.056 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:12 GMT'), ('x-amz-version-id', 'YniDC1out7TFV1sI.TLfuqiPcVFd.5S'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000091afe4b99ee56b19-00646e7d21-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.057 DEBUG:boto:path=/qskrps-11/ 2023-05-24T21:09:53.057 DEBUG:boto:auth_path=/qskrps-11/ 2023-05-24T21:09:53.058 DEBUG:boto:path=/qskrps-11/?prefix=obj-a2 2023-05-24T21:09:53.058 DEBUG:boto:auth_path=/qskrps-11/?prefix=obj-a2 2023-05-24T21:09:53.058 DEBUG:boto:Method: GET 2023-05-24T21:09:53.059 DEBUG:boto:Path: /qskrps-11/?prefix=obj-a2 2023-05-24T21:09:53.059 DEBUG:boto:Data: 2023-05-24T21:09:53.059 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.060 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:53.060 DEBUG:boto:Port: 8001 2023-05-24T21:09:53.060 DEBUG:boto:Params: {} 2023-05-24T21:09:53.061 DEBUG:boto:Token: None 2023-05-24T21:09:53.061 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:53 GMT /qskrps-11/ 2023-05-24T21:09:53.062 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:8ba+Lk5je6c9IpfhfpUCGl1L9qQ= 2023-05-24T21:09:53.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:8ba+Lk5je6c9IpfhfpUCGl1L9qQ=', 'Content-Length': '0'} 2023-05-24T21:09:53.064 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087ae74397de95bd6-00646e7d21-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.112 DEBUG:boto:b'qskrps-11obj-a21000falseobj-a22023-05-24T21:08:14.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:53.113 DEBUG:boto:path=/qskrps-11/ 2023-05-24T21:09:53.114 DEBUG:boto:auth_path=/qskrps-11/ 2023-05-24T21:09:53.114 DEBUG:boto:path=/qskrps-11/?prefix=obj-a2 2023-05-24T21:09:53.114 DEBUG:boto:auth_path=/qskrps-11/?prefix=obj-a2 2023-05-24T21:09:53.115 DEBUG:boto:Method: GET 2023-05-24T21:09:53.115 DEBUG:boto:Path: /qskrps-11/?prefix=obj-a2 2023-05-24T21:09:53.115 DEBUG:boto:Data: 2023-05-24T21:09:53.116 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.116 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:53.116 DEBUG:boto:Port: 8000 2023-05-24T21:09:53.117 DEBUG:boto:Params: {} 2023-05-24T21:09:53.117 DEBUG:boto:Token: None 2023-05-24T21:09:53.118 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:53 GMT /qskrps-11/ 2023-05-24T21:09:53.118 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:8ba+Lk5je6c9IpfhfpUCGl1L9qQ= 2023-05-24T21:09:53.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:8ba+Lk5je6c9IpfhfpUCGl1L9qQ=', 'Content-Length': '0'} 2023-05-24T21:09:53.121 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e3d80dfb6bc4e9b-00646e7d21-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.121 DEBUG:boto:b'qskrps-11obj-a21000falseobj-a22023-05-24T21:08:14.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:53.122 DEBUG:boto:path=/qskrps-11/obj-a2 2023-05-24T21:09:53.122 DEBUG:boto:auth_path=/qskrps-11/obj-a2 2023-05-24T21:09:53.123 DEBUG:boto:Method: HEAD 2023-05-24T21:09:53.123 DEBUG:boto:Path: /qskrps-11/obj-a2 2023-05-24T21:09:53.123 DEBUG:boto:Data: 2023-05-24T21:09:53.124 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.124 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:53.124 DEBUG:boto:Port: 8001 2023-05-24T21:09:53.125 DEBUG:boto:Params: {} 2023-05-24T21:09:53.125 DEBUG:boto:Token: None 2023-05-24T21:09:53.126 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:53 GMT /qskrps-11/obj-a2 2023-05-24T21:09:53.126 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:XWyqymWgEE3EGo3upjmXkQ/EMts= 2023-05-24T21:09:53.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:XWyqymWgEE3EGo3upjmXkQ/EMts=', 'Content-Length': '0'} 2023-05-24T21:09:53.129 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:12 GMT'), ('x-amz-version-id', 'YniDC1out7TFV1sI.TLfuqiPcVFd.5S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ee63fe901447ec2-00646e7d21-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.129 DEBUG:boto:path=/qskrps-11/obj-a2 2023-05-24T21:09:53.130 DEBUG:boto:auth_path=/qskrps-11/obj-a2 2023-05-24T21:09:53.130 DEBUG:boto:Method: HEAD 2023-05-24T21:09:53.130 DEBUG:boto:Path: /qskrps-11/obj-a2 2023-05-24T21:09:53.131 DEBUG:boto:Data: 2023-05-24T21:09:53.131 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.131 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:53.132 DEBUG:boto:Port: 8000 2023-05-24T21:09:53.132 DEBUG:boto:Params: {} 2023-05-24T21:09:53.132 DEBUG:boto:Token: None 2023-05-24T21:09:53.133 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:53 GMT /qskrps-11/obj-a2 2023-05-24T21:09:53.133 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:XWyqymWgEE3EGo3upjmXkQ/EMts= 2023-05-24T21:09:53.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:XWyqymWgEE3EGo3upjmXkQ/EMts=', 'Content-Length': '0'} 2023-05-24T21:09:53.136 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:12 GMT'), ('x-amz-version-id', 'YniDC1out7TFV1sI.TLfuqiPcVFd.5S'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007579e44bd510a2b6-00646e7d21-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-05-24T21:09:53.137 DEBUG:boto:path=/qskrps-11/obj-a2 2023-05-24T21:09:53.137 DEBUG:boto:auth_path=/qskrps-11/obj-a2 2023-05-24T21:09:53.137 DEBUG:boto:path=/qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:09:53.138 DEBUG:boto:auth_path=/qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:09:53.138 DEBUG:boto:Method: GET 2023-05-24T21:09:53.138 DEBUG:boto:Path: /qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:09:53.139 DEBUG:boto:Data: 2023-05-24T21:09:53.139 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.139 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:53.140 DEBUG:boto:Port: 8001 2023-05-24T21:09:53.140 DEBUG:boto:Params: {} 2023-05-24T21:09:53.141 DEBUG:boto:Token: None 2023-05-24T21:09:53.141 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:53 GMT /qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:09:53.141 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:wkN58CO9nnf6U7tiB0XtmYvUn4o= 2023-05-24T21:09:53.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wkN58CO9nnf6U7tiB0XtmYvUn4o=', 'Content-Length': '0'} 2023-05-24T21:09:53.144 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:12 GMT'), ('x-amz-version-id', 'YniDC1out7TFV1sI.TLfuqiPcVFd.5S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000deeec6df20cd5044-00646e7d21-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.144 DEBUG:boto:path=/qskrps-11/obj-a2 2023-05-24T21:09:53.145 DEBUG:boto:auth_path=/qskrps-11/obj-a2 2023-05-24T21:09:53.145 DEBUG:boto:path=/qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:09:53.145 DEBUG:boto:auth_path=/qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:09:53.146 DEBUG:boto:Method: GET 2023-05-24T21:09:53.146 DEBUG:boto:Path: /qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:09:53.146 DEBUG:boto:Data: 2023-05-24T21:09:53.147 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.147 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:53.147 DEBUG:boto:Port: 8000 2023-05-24T21:09:53.148 DEBUG:boto:Params: {} 2023-05-24T21:09:53.148 DEBUG:boto:Token: None 2023-05-24T21:09:53.149 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:53 GMT /qskrps-11/obj-a2?versionId=YniDC1out7TFV1sI.TLfuqiPcVFd.5S 2023-05-24T21:09:53.149 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:wkN58CO9nnf6U7tiB0XtmYvUn4o= 2023-05-24T21:09:53.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wkN58CO9nnf6U7tiB0XtmYvUn4o=', 'Content-Length': '0'} 2023-05-24T21:09:53.151 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:12 GMT'), ('x-amz-version-id', 'YniDC1out7TFV1sI.TLfuqiPcVFd.5S'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f6475468b0d3f01b-00646e7d21-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.196 DEBUG:boto:path=/qskrps-11/ 2023-05-24T21:09:53.197 DEBUG:boto:auth_path=/qskrps-11/ 2023-05-24T21:09:53.197 DEBUG:boto:path=/qskrps-11/?prefix=obj-a2 2023-05-24T21:09:53.198 DEBUG:boto:auth_path=/qskrps-11/?prefix=obj-a2 2023-05-24T21:09:53.198 DEBUG:boto:Method: GET 2023-05-24T21:09:53.199 DEBUG:boto:Path: /qskrps-11/?prefix=obj-a2 2023-05-24T21:09:53.199 DEBUG:boto:Data: 2023-05-24T21:09:53.199 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.200 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:53.200 DEBUG:boto:Port: 8001 2023-05-24T21:09:53.200 DEBUG:boto:Params: {} 2023-05-24T21:09:53.201 DEBUG:boto:Token: None 2023-05-24T21:09:53.201 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:53 GMT /qskrps-11/ 2023-05-24T21:09:53.202 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:8ba+Lk5je6c9IpfhfpUCGl1L9qQ= 2023-05-24T21:09:53.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:8ba+Lk5je6c9IpfhfpUCGl1L9qQ=', 'Content-Length': '0'} 2023-05-24T21:09:53.205 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000554273bda670cb53-00646e7d21-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.205 DEBUG:boto:b'qskrps-11obj-a21000falseobj-a22023-05-24T21:08:14.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:53.206 DEBUG:boto:path=/qskrps-11/ 2023-05-24T21:09:53.206 DEBUG:boto:auth_path=/qskrps-11/ 2023-05-24T21:09:53.207 DEBUG:boto:path=/qskrps-11/?prefix=obj-a2 2023-05-24T21:09:53.207 DEBUG:boto:auth_path=/qskrps-11/?prefix=obj-a2 2023-05-24T21:09:53.207 DEBUG:boto:Method: GET 2023-05-24T21:09:53.208 DEBUG:boto:Path: /qskrps-11/?prefix=obj-a2 2023-05-24T21:09:53.208 DEBUG:boto:Data: 2023-05-24T21:09:53.208 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.209 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:53.209 DEBUG:boto:Port: 8000 2023-05-24T21:09:53.210 DEBUG:boto:Params: {} 2023-05-24T21:09:53.210 DEBUG:boto:Token: None 2023-05-24T21:09:53.210 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:53 GMT /qskrps-11/ 2023-05-24T21:09:53.211 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:8ba+Lk5je6c9IpfhfpUCGl1L9qQ= 2023-05-24T21:09:53.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:8ba+Lk5je6c9IpfhfpUCGl1L9qQ=', 'Content-Length': '0'} 2023-05-24T21:09:53.213 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1475af8e230cc12-00646e7d21-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.214 DEBUG:boto:b'qskrps-11obj-a21000falseobj-a22023-05-24T21:08:14.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:53.215 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qskrps-11 zones={a2, a1} 2023-05-24T21:09:53.215 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=qskrps-12 2023-05-24T21:09:53.215 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-12', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:09:53.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-12', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:09:53.216 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-12 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:09:53.279 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:09:53.280 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:09:53.333 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:09:53.326+0000 7fd7771b1780 1 bucket sync caught up with source: 2023-05-24T21:09:53.334 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000002.17.11, , , 00000000003.8.6, 00000000011.26.6, ] 2023-05-24T21:09:53.334 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000002.17.11, , , 00000000003.8.6, 00000000011.26.6, ] 2023-05-24T21:09:53.334 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:09:53.326+0000 7fd7771b1780 0 bucket checkpoint complete 2023-05-24T21:09:53.355 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=qskrps-12 2023-05-24T21:09:53.355 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:09:53.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:09:53.356 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-12 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:09:53.412 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:09:53.412 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:09:53.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:09:53.455+0000 7f61d96e5780 1 bucket sync caught up with source: 2023-05-24T21:09:53.462 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000003.36.10, , , 00000000011.31.6, 00000000003.22.6, ] 2023-05-24T21:09:53.462 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000003.36.10, , , 00000000011.31.6, 00000000003.22.6, ] 2023-05-24T21:09:53.463 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:09:53.455+0000 7f61d96e5780 0 bucket checkpoint complete 2023-05-24T21:09:53.479 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qskrps-12 zones={a2, a1} 2023-05-24T21:09:53.479 DEBUG:boto:path=/qskrps-12/ 2023-05-24T21:09:53.480 DEBUG:boto:auth_path=/qskrps-12/ 2023-05-24T21:09:53.480 DEBUG:boto:Method: HEAD 2023-05-24T21:09:53.480 DEBUG:boto:Path: /qskrps-12/ 2023-05-24T21:09:53.481 DEBUG:boto:Data: 2023-05-24T21:09:53.481 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.481 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:53.482 DEBUG:boto:Port: 8001 2023-05-24T21:09:53.482 DEBUG:boto:Params: {} 2023-05-24T21:09:53.482 DEBUG:boto:Token: None 2023-05-24T21:09:53.483 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:53 GMT /qskrps-12/ 2023-05-24T21:09:53.483 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:KBI0NGkJYoo/OtAyTbMjSRvYWi4= 2023-05-24T21:09:53.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:KBI0NGkJYoo/OtAyTbMjSRvYWi4=', 'Content-Length': '0'} 2023-05-24T21:09:53.486 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e31c27ba0e7b64db-00646e7d21-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.486 DEBUG:boto:path=/qskrps-12/ 2023-05-24T21:09:53.487 DEBUG:boto:auth_path=/qskrps-12/ 2023-05-24T21:09:53.487 DEBUG:boto:Method: HEAD 2023-05-24T21:09:53.487 DEBUG:boto:Path: /qskrps-12/ 2023-05-24T21:09:53.488 DEBUG:boto:Data: 2023-05-24T21:09:53.488 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.489 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:53.489 DEBUG:boto:Port: 8000 2023-05-24T21:09:53.489 DEBUG:boto:Params: {} 2023-05-24T21:09:53.490 DEBUG:boto:Token: None 2023-05-24T21:09:53.490 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:53 GMT /qskrps-12/ 2023-05-24T21:09:53.490 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:KBI0NGkJYoo/OtAyTbMjSRvYWi4= 2023-05-24T21:09:53.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:KBI0NGkJYoo/OtAyTbMjSRvYWi4=', 'Content-Length': '0'} 2023-05-24T21:09:53.493 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d53aaea7aa599e1f-00646e7d21-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.494 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-24T21:09:53.494 DEBUG:boto:path=/qskrps-12/ 2023-05-24T21:09:53.494 DEBUG:boto:auth_path=/qskrps-12/ 2023-05-24T21:09:53.495 DEBUG:boto:path=/qskrps-12/?versions&max-keys=999 2023-05-24T21:09:53.495 DEBUG:boto:auth_path=/qskrps-12/?versions&max-keys=999 2023-05-24T21:09:53.495 DEBUG:boto:Method: GET 2023-05-24T21:09:53.496 DEBUG:boto:Path: /qskrps-12/?versions&max-keys=999 2023-05-24T21:09:53.496 DEBUG:boto:Data: 2023-05-24T21:09:53.496 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.497 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:53.497 DEBUG:boto:Port: 8001 2023-05-24T21:09:53.497 DEBUG:boto:Params: {} 2023-05-24T21:09:53.498 DEBUG:boto:Token: None 2023-05-24T21:09:53.498 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:53 GMT /qskrps-12/?versions 2023-05-24T21:09:53.499 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:pepM6kc4SkDYbmMS6mf6VtG6Fn0= 2023-05-24T21:09:53.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:pepM6kc4SkDYbmMS6mf6VtG6Fn0=', 'Content-Length': '0'} 2023-05-24T21:09:53.501 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070b0c2e986d78fd4-00646e7d21-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.501 DEBUG:boto:b'qskrps-12999falsedummykm8zCAetA1TzFWDKQlSimNmDeLQdejHtrue2023-05-24T21:07:54.624Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiVtrue2023-05-24T21:08:15.067Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2PcWX5rYNtc410b9LwKxno3LHHZeAmhmtrue2023-05-24T21:08:15.193Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:53.502 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-05-24T21:09:53.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-05-24T21:09:53.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-05-24T21:09:53.503 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-24T21:09:53.504 DEBUG:boto:path=/qskrps-12/ 2023-05-24T21:09:53.504 DEBUG:boto:auth_path=/qskrps-12/ 2023-05-24T21:09:53.504 DEBUG:boto:path=/qskrps-12/?versions&max-keys=999 2023-05-24T21:09:53.505 DEBUG:boto:auth_path=/qskrps-12/?versions&max-keys=999 2023-05-24T21:09:53.505 DEBUG:boto:Method: GET 2023-05-24T21:09:53.505 DEBUG:boto:Path: /qskrps-12/?versions&max-keys=999 2023-05-24T21:09:53.506 DEBUG:boto:Data: 2023-05-24T21:09:53.506 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.506 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:53.507 DEBUG:boto:Port: 8000 2023-05-24T21:09:53.507 DEBUG:boto:Params: {} 2023-05-24T21:09:53.507 DEBUG:boto:Token: None 2023-05-24T21:09:53.508 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:53 GMT /qskrps-12/?versions 2023-05-24T21:09:53.508 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:pepM6kc4SkDYbmMS6mf6VtG6Fn0= 2023-05-24T21:09:53.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:pepM6kc4SkDYbmMS6mf6VtG6Fn0=', 'Content-Length': '0'} 2023-05-24T21:09:53.511 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008dc5de4fbbf74e71-00646e7d21-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.511 DEBUG:boto:b'qskrps-12999falsedummykm8zCAetA1TzFWDKQlSimNmDeLQdejHtrue2023-05-24T21:07:54.624Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiVtrue2023-05-24T21:08:15.067Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2PcWX5rYNtc410b9LwKxno3LHHZeAmhmtrue2023-05-24T21:08:15.193Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:53.512 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-05-24T21:09:53.513 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-05-24T21:09:53.513 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-05-24T21:09:53.513 DEBUG:boto:path=/qskrps-12/ 2023-05-24T21:09:53.514 DEBUG:boto:auth_path=/qskrps-12/ 2023-05-24T21:09:53.514 DEBUG:boto:path=/qskrps-12/?versions&max-keys=999 2023-05-24T21:09:53.514 DEBUG:boto:auth_path=/qskrps-12/?versions&max-keys=999 2023-05-24T21:09:53.515 DEBUG:boto:Method: GET 2023-05-24T21:09:53.515 DEBUG:boto:Path: /qskrps-12/?versions&max-keys=999 2023-05-24T21:09:53.515 DEBUG:boto:Data: 2023-05-24T21:09:53.516 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.516 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:53.516 DEBUG:boto:Port: 8001 2023-05-24T21:09:53.517 DEBUG:boto:Params: {} 2023-05-24T21:09:53.517 DEBUG:boto:Token: None 2023-05-24T21:09:53.517 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:53 GMT /qskrps-12/?versions 2023-05-24T21:09:53.518 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:pepM6kc4SkDYbmMS6mf6VtG6Fn0= 2023-05-24T21:09:53.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:pepM6kc4SkDYbmMS6mf6VtG6Fn0=', 'Content-Length': '0'} 2023-05-24T21:09:53.520 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039346aa6dbbb073f-00646e7d21-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.521 DEBUG:boto:b'qskrps-12999falsedummykm8zCAetA1TzFWDKQlSimNmDeLQdejHtrue2023-05-24T21:07:54.624Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiVtrue2023-05-24T21:08:15.067Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2PcWX5rYNtc410b9LwKxno3LHHZeAmhmtrue2023-05-24T21:08:15.193Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:53.521 DEBUG:boto:path=/qskrps-12/ 2023-05-24T21:09:53.522 DEBUG:boto:auth_path=/qskrps-12/ 2023-05-24T21:09:53.522 DEBUG:boto:path=/qskrps-12/?versions&max-keys=999 2023-05-24T21:09:53.522 DEBUG:boto:auth_path=/qskrps-12/?versions&max-keys=999 2023-05-24T21:09:53.523 DEBUG:boto:Method: GET 2023-05-24T21:09:53.523 DEBUG:boto:Path: /qskrps-12/?versions&max-keys=999 2023-05-24T21:09:53.524 DEBUG:boto:Data: 2023-05-24T21:09:53.524 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.524 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:53.525 DEBUG:boto:Port: 8000 2023-05-24T21:09:53.525 DEBUG:boto:Params: {} 2023-05-24T21:09:53.525 DEBUG:boto:Token: None 2023-05-24T21:09:53.526 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:53 GMT /qskrps-12/?versions 2023-05-24T21:09:53.526 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:pepM6kc4SkDYbmMS6mf6VtG6Fn0= 2023-05-24T21:09:53.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:pepM6kc4SkDYbmMS6mf6VtG6Fn0=', 'Content-Length': '0'} 2023-05-24T21:09:53.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086448188de18a038-00646e7d21-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.529 DEBUG:boto:b'qskrps-12999falsedummykm8zCAetA1TzFWDKQlSimNmDeLQdejHtrue2023-05-24T21:07:54.624Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiVtrue2023-05-24T21:08:15.067Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2PcWX5rYNtc410b9LwKxno3LHHZeAmhmtrue2023-05-24T21:08:15.193Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:53.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-05-24T21:09:53.530 DEBUG:boto:path=/qskrps-12/dummy 2023-05-24T21:09:53.530 DEBUG:boto:auth_path=/qskrps-12/dummy 2023-05-24T21:09:53.531 DEBUG:boto:path=/qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:53.531 DEBUG:boto:auth_path=/qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:53.531 DEBUG:boto:Method: GET 2023-05-24T21:09:53.532 DEBUG:boto:Path: /qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:53.532 DEBUG:boto:Data: 2023-05-24T21:09:53.533 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.533 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:53.533 DEBUG:boto:Port: 8001 2023-05-24T21:09:53.534 DEBUG:boto:Params: {} 2023-05-24T21:09:53.534 DEBUG:boto:Token: None 2023-05-24T21:09:53.534 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:53 GMT /qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:53.535 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:jy3cPOAGX70MikmBafJiLoL3ISs= 2023-05-24T21:09:53.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:jy3cPOAGX70MikmBafJiLoL3ISs=', 'Content-Length': '0'} 2023-05-24T21:09:53.537 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:07:54 GMT'), ('x-amz-version-id', 'km8zCAetA1TzFWDKQlSimNmDeLQdejH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cee32bbc0d0f3d0b-00646e7d21-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.537 DEBUG:boto:path=/qskrps-12/dummy 2023-05-24T21:09:53.538 DEBUG:boto:auth_path=/qskrps-12/dummy 2023-05-24T21:09:53.538 DEBUG:boto:path=/qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:53.539 DEBUG:boto:auth_path=/qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:53.539 DEBUG:boto:Method: GET 2023-05-24T21:09:53.539 DEBUG:boto:Path: /qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:53.540 DEBUG:boto:Data: 2023-05-24T21:09:53.540 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.540 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:53.541 DEBUG:boto:Port: 8000 2023-05-24T21:09:53.541 DEBUG:boto:Params: {} 2023-05-24T21:09:53.542 DEBUG:boto:Token: None 2023-05-24T21:09:53.542 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:53 GMT /qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:53.542 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:jy3cPOAGX70MikmBafJiLoL3ISs= 2023-05-24T21:09:53.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:jy3cPOAGX70MikmBafJiLoL3ISs=', 'Content-Length': '0'} 2023-05-24T21:09:53.544 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:07:54 GMT'), ('x-amz-version-id', 'km8zCAetA1TzFWDKQlSimNmDeLQdejH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ea5485047c83834c-00646e7d21-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.545 DEBUG:boto:path=/qskrps-12/dummy 2023-05-24T21:09:53.545 DEBUG:boto:auth_path=/qskrps-12/dummy 2023-05-24T21:09:53.546 DEBUG:boto:path=/qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:53.546 DEBUG:boto:auth_path=/qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:53.547 DEBUG:boto:Method: HEAD 2023-05-24T21:09:53.547 DEBUG:boto:Path: /qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:53.547 DEBUG:boto:Data: 2023-05-24T21:09:53.548 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.548 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:53.548 DEBUG:boto:Port: 8001 2023-05-24T21:09:53.549 DEBUG:boto:Params: {} 2023-05-24T21:09:53.549 DEBUG:boto:Token: None 2023-05-24T21:09:53.549 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:53 GMT /qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:53.550 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:IJCzA4ngszKw/WgDeZItHU7V7RM= 2023-05-24T21:09:53.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:IJCzA4ngszKw/WgDeZItHU7V7RM=', 'Content-Length': '0'} 2023-05-24T21:09:53.551 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:07:54 GMT'), ('x-amz-version-id', 'km8zCAetA1TzFWDKQlSimNmDeLQdejH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000744cac405a508bae-00646e7d21-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.552 DEBUG:boto:path=/qskrps-12/dummy 2023-05-24T21:09:53.552 DEBUG:boto:auth_path=/qskrps-12/dummy 2023-05-24T21:09:53.553 DEBUG:boto:path=/qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:53.553 DEBUG:boto:auth_path=/qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:53.553 DEBUG:boto:Method: HEAD 2023-05-24T21:09:53.554 DEBUG:boto:Path: /qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:53.554 DEBUG:boto:Data: 2023-05-24T21:09:53.554 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.555 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:53.555 DEBUG:boto:Port: 8000 2023-05-24T21:09:53.555 DEBUG:boto:Params: {} 2023-05-24T21:09:53.556 DEBUG:boto:Token: None 2023-05-24T21:09:53.556 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:53 GMT /qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:53.557 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:IJCzA4ngszKw/WgDeZItHU7V7RM= 2023-05-24T21:09:53.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:IJCzA4ngszKw/WgDeZItHU7V7RM=', 'Content-Length': '0'} 2023-05-24T21:09:53.560 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:07:54 GMT'), ('x-amz-version-id', 'km8zCAetA1TzFWDKQlSimNmDeLQdejH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a97d3ec11553a1d4-00646e7d21-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-05-24T21:09:53.562 DEBUG:boto:path=/qskrps-12/dummy 2023-05-24T21:09:53.562 DEBUG:boto:auth_path=/qskrps-12/dummy 2023-05-24T21:09:53.562 DEBUG:boto:path=/qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:53.563 DEBUG:boto:auth_path=/qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:53.563 DEBUG:boto:Method: GET 2023-05-24T21:09:53.563 DEBUG:boto:Path: /qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:53.564 DEBUG:boto:Data: 2023-05-24T21:09:53.564 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.565 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:53.565 DEBUG:boto:Port: 8001 2023-05-24T21:09:53.565 DEBUG:boto:Params: {} 2023-05-24T21:09:53.566 DEBUG:boto:Token: None 2023-05-24T21:09:53.566 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:53 GMT /qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:53.567 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:jy3cPOAGX70MikmBafJiLoL3ISs= 2023-05-24T21:09:53.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:jy3cPOAGX70MikmBafJiLoL3ISs=', 'Content-Length': '0'} 2023-05-24T21:09:53.569 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:07:54 GMT'), ('x-amz-version-id', 'km8zCAetA1TzFWDKQlSimNmDeLQdejH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bea37eb8451ddb91-00646e7d21-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.569 DEBUG:boto:path=/qskrps-12/dummy 2023-05-24T21:09:53.570 DEBUG:boto:auth_path=/qskrps-12/dummy 2023-05-24T21:09:53.570 DEBUG:boto:path=/qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:53.570 DEBUG:boto:auth_path=/qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:53.571 DEBUG:boto:Method: GET 2023-05-24T21:09:53.571 DEBUG:boto:Path: /qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:53.572 DEBUG:boto:Data: 2023-05-24T21:09:53.573 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.573 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:53.573 DEBUG:boto:Port: 8000 2023-05-24T21:09:53.574 DEBUG:boto:Params: {} 2023-05-24T21:09:53.574 DEBUG:boto:Token: None 2023-05-24T21:09:53.575 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:53 GMT /qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:53.575 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:jy3cPOAGX70MikmBafJiLoL3ISs= 2023-05-24T21:09:53.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:jy3cPOAGX70MikmBafJiLoL3ISs=', 'Content-Length': '0'} 2023-05-24T21:09:53.578 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:07:54 GMT'), ('x-amz-version-id', 'km8zCAetA1TzFWDKQlSimNmDeLQdejH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005a1244f76293f907-00646e7d21-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.578 DEBUG:boto:path=/qskrps-12/ 2023-05-24T21:09:53.579 DEBUG:boto:auth_path=/qskrps-12/ 2023-05-24T21:09:53.579 DEBUG:boto:path=/qskrps-12/?prefix=dummy 2023-05-24T21:09:53.580 DEBUG:boto:auth_path=/qskrps-12/?prefix=dummy 2023-05-24T21:09:53.580 DEBUG:boto:Method: GET 2023-05-24T21:09:53.581 DEBUG:boto:Path: /qskrps-12/?prefix=dummy 2023-05-24T21:09:53.581 DEBUG:boto:Data: 2023-05-24T21:09:53.581 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.582 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:53.582 DEBUG:boto:Port: 8001 2023-05-24T21:09:53.582 DEBUG:boto:Params: {} 2023-05-24T21:09:53.583 DEBUG:boto:Token: None 2023-05-24T21:09:53.583 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:53 GMT /qskrps-12/ 2023-05-24T21:09:53.584 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:7ulOuSFyzfND0cttgSz4vnt94vc= 2023-05-24T21:09:53.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:7ulOuSFyzfND0cttgSz4vnt94vc=', 'Content-Length': '0'} 2023-05-24T21:09:53.586 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a4e1486ef1456d6-00646e7d21-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.632 DEBUG:boto:b'qskrps-12dummy1000falsedummy2023-05-24T21:07:54.624Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:53.633 DEBUG:boto:path=/qskrps-12/ 2023-05-24T21:09:53.634 DEBUG:boto:auth_path=/qskrps-12/ 2023-05-24T21:09:53.634 DEBUG:boto:path=/qskrps-12/?prefix=dummy 2023-05-24T21:09:53.634 DEBUG:boto:auth_path=/qskrps-12/?prefix=dummy 2023-05-24T21:09:53.635 DEBUG:boto:Method: GET 2023-05-24T21:09:53.635 DEBUG:boto:Path: /qskrps-12/?prefix=dummy 2023-05-24T21:09:53.635 DEBUG:boto:Data: 2023-05-24T21:09:53.636 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.636 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:53.636 DEBUG:boto:Port: 8000 2023-05-24T21:09:53.637 DEBUG:boto:Params: {} 2023-05-24T21:09:53.637 DEBUG:boto:Token: None 2023-05-24T21:09:53.638 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:53 GMT /qskrps-12/ 2023-05-24T21:09:53.638 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:7ulOuSFyzfND0cttgSz4vnt94vc= 2023-05-24T21:09:53.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:7ulOuSFyzfND0cttgSz4vnt94vc=', 'Content-Length': '0'} 2023-05-24T21:09:53.641 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008a0676f934c3f3a-00646e7d21-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.641 DEBUG:boto:b'qskrps-12dummy1000falsedummy2023-05-24T21:07:54.624Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:53.642 DEBUG:boto:path=/qskrps-12/dummy 2023-05-24T21:09:53.642 DEBUG:boto:auth_path=/qskrps-12/dummy 2023-05-24T21:09:53.642 DEBUG:boto:Method: HEAD 2023-05-24T21:09:53.643 DEBUG:boto:Path: /qskrps-12/dummy 2023-05-24T21:09:53.643 DEBUG:boto:Data: 2023-05-24T21:09:53.643 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.644 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:53.644 DEBUG:boto:Port: 8001 2023-05-24T21:09:53.644 DEBUG:boto:Params: {} 2023-05-24T21:09:53.645 DEBUG:boto:Token: None 2023-05-24T21:09:53.645 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:53 GMT /qskrps-12/dummy 2023-05-24T21:09:53.646 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:AXKi2YY9LRgb/v0poxMFX9TAqlI= 2023-05-24T21:09:53.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:AXKi2YY9LRgb/v0poxMFX9TAqlI=', 'Content-Length': '0'} 2023-05-24T21:09:53.648 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:07:54 GMT'), ('x-amz-version-id', 'km8zCAetA1TzFWDKQlSimNmDeLQdejH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e335088850b5ca80-00646e7d21-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.649 DEBUG:boto:path=/qskrps-12/dummy 2023-05-24T21:09:53.649 DEBUG:boto:auth_path=/qskrps-12/dummy 2023-05-24T21:09:53.649 DEBUG:boto:Method: HEAD 2023-05-24T21:09:53.650 DEBUG:boto:Path: /qskrps-12/dummy 2023-05-24T21:09:53.650 DEBUG:boto:Data: 2023-05-24T21:09:53.650 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.651 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:53.651 DEBUG:boto:Port: 8000 2023-05-24T21:09:53.651 DEBUG:boto:Params: {} 2023-05-24T21:09:53.652 DEBUG:boto:Token: None 2023-05-24T21:09:53.652 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:53 GMT /qskrps-12/dummy 2023-05-24T21:09:53.652 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:AXKi2YY9LRgb/v0poxMFX9TAqlI= 2023-05-24T21:09:53.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:AXKi2YY9LRgb/v0poxMFX9TAqlI=', 'Content-Length': '0'} 2023-05-24T21:09:53.655 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:07:54 GMT'), ('x-amz-version-id', 'km8zCAetA1TzFWDKQlSimNmDeLQdejH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000190752c0fb770a1c-00646e7d21-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-05-24T21:09:53.656 DEBUG:boto:path=/qskrps-12/dummy 2023-05-24T21:09:53.656 DEBUG:boto:auth_path=/qskrps-12/dummy 2023-05-24T21:09:53.656 DEBUG:boto:path=/qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:53.657 DEBUG:boto:auth_path=/qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:53.657 DEBUG:boto:Method: GET 2023-05-24T21:09:53.657 DEBUG:boto:Path: /qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:53.658 DEBUG:boto:Data: 2023-05-24T21:09:53.658 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.659 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:53.659 DEBUG:boto:Port: 8001 2023-05-24T21:09:53.659 DEBUG:boto:Params: {} 2023-05-24T21:09:53.660 DEBUG:boto:Token: None 2023-05-24T21:09:53.660 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:53 GMT /qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:53.661 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:jy3cPOAGX70MikmBafJiLoL3ISs= 2023-05-24T21:09:53.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:jy3cPOAGX70MikmBafJiLoL3ISs=', 'Content-Length': '0'} 2023-05-24T21:09:53.663 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:07:54 GMT'), ('x-amz-version-id', 'km8zCAetA1TzFWDKQlSimNmDeLQdejH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000493e3e3185d2a088-00646e7d21-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.663 DEBUG:boto:path=/qskrps-12/dummy 2023-05-24T21:09:53.664 DEBUG:boto:auth_path=/qskrps-12/dummy 2023-05-24T21:09:53.664 DEBUG:boto:path=/qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:53.664 DEBUG:boto:auth_path=/qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:53.665 DEBUG:boto:Method: GET 2023-05-24T21:09:53.665 DEBUG:boto:Path: /qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:53.666 DEBUG:boto:Data: 2023-05-24T21:09:53.666 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.666 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:53.667 DEBUG:boto:Port: 8000 2023-05-24T21:09:53.667 DEBUG:boto:Params: {} 2023-05-24T21:09:53.667 DEBUG:boto:Token: None 2023-05-24T21:09:53.668 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:53 GMT /qskrps-12/dummy?versionId=km8zCAetA1TzFWDKQlSimNmDeLQdejH 2023-05-24T21:09:53.668 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:jy3cPOAGX70MikmBafJiLoL3ISs= 2023-05-24T21:09:53.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:jy3cPOAGX70MikmBafJiLoL3ISs=', 'Content-Length': '0'} 2023-05-24T21:09:53.670 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:07:54 GMT'), ('x-amz-version-id', 'km8zCAetA1TzFWDKQlSimNmDeLQdejH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000e487200b34785f1-00646e7d21-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.671 DEBUG:boto:path=/qskrps-12/ 2023-05-24T21:09:53.671 DEBUG:boto:auth_path=/qskrps-12/ 2023-05-24T21:09:53.671 DEBUG:boto:path=/qskrps-12/?prefix=dummy 2023-05-24T21:09:53.672 DEBUG:boto:auth_path=/qskrps-12/?prefix=dummy 2023-05-24T21:09:53.672 DEBUG:boto:Method: GET 2023-05-24T21:09:53.672 DEBUG:boto:Path: /qskrps-12/?prefix=dummy 2023-05-24T21:09:53.673 DEBUG:boto:Data: 2023-05-24T21:09:53.673 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.673 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:53.674 DEBUG:boto:Port: 8001 2023-05-24T21:09:53.674 DEBUG:boto:Params: {} 2023-05-24T21:09:53.675 DEBUG:boto:Token: None 2023-05-24T21:09:53.675 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:53 GMT /qskrps-12/ 2023-05-24T21:09:53.675 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:7ulOuSFyzfND0cttgSz4vnt94vc= 2023-05-24T21:09:53.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:7ulOuSFyzfND0cttgSz4vnt94vc=', 'Content-Length': '0'} 2023-05-24T21:09:53.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a06b427ec20aa5cd-00646e7d21-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.678 DEBUG:boto:b'qskrps-12dummy1000falsedummy2023-05-24T21:07:54.624Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:53.679 DEBUG:boto:path=/qskrps-12/ 2023-05-24T21:09:53.679 DEBUG:boto:auth_path=/qskrps-12/ 2023-05-24T21:09:53.680 DEBUG:boto:path=/qskrps-12/?prefix=dummy 2023-05-24T21:09:53.680 DEBUG:boto:auth_path=/qskrps-12/?prefix=dummy 2023-05-24T21:09:53.680 DEBUG:boto:Method: GET 2023-05-24T21:09:53.681 DEBUG:boto:Path: /qskrps-12/?prefix=dummy 2023-05-24T21:09:53.681 DEBUG:boto:Data: 2023-05-24T21:09:53.682 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.682 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:53.682 DEBUG:boto:Port: 8000 2023-05-24T21:09:53.683 DEBUG:boto:Params: {} 2023-05-24T21:09:53.683 DEBUG:boto:Token: None 2023-05-24T21:09:53.683 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:53 GMT /qskrps-12/ 2023-05-24T21:09:53.684 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:7ulOuSFyzfND0cttgSz4vnt94vc= 2023-05-24T21:09:53.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:7ulOuSFyzfND0cttgSz4vnt94vc=', 'Content-Length': '0'} 2023-05-24T21:09:53.686 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084f1edb93a96b662-00646e7d21-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.687 DEBUG:boto:b'qskrps-12dummy1000falsedummy2023-05-24T21:07:54.624Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:53.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-05-24T21:09:53.687 DEBUG:boto:path=/qskrps-12/obj-a1 2023-05-24T21:09:53.688 DEBUG:boto:auth_path=/qskrps-12/obj-a1 2023-05-24T21:09:53.688 DEBUG:boto:path=/qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:53.689 DEBUG:boto:auth_path=/qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:53.689 DEBUG:boto:Method: GET 2023-05-24T21:09:53.689 DEBUG:boto:Path: /qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:53.690 DEBUG:boto:Data: 2023-05-24T21:09:53.690 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.690 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:53.691 DEBUG:boto:Port: 8001 2023-05-24T21:09:53.691 DEBUG:boto:Params: {} 2023-05-24T21:09:53.692 DEBUG:boto:Token: None 2023-05-24T21:09:53.692 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:53 GMT /qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:53.692 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:AKrTGpKgJj/cwLkcWcLKiJLlixI= 2023-05-24T21:09:53.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:AKrTGpKgJj/cwLkcWcLKiJLlixI=', 'Content-Length': '0'} 2023-05-24T21:09:53.695 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:12 GMT'), ('x-amz-version-id', 'OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000001b66aefd773bb22-00646e7d21-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.695 DEBUG:boto:path=/qskrps-12/obj-a1 2023-05-24T21:09:53.696 DEBUG:boto:auth_path=/qskrps-12/obj-a1 2023-05-24T21:09:53.696 DEBUG:boto:path=/qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:53.696 DEBUG:boto:auth_path=/qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:53.697 DEBUG:boto:Method: GET 2023-05-24T21:09:53.697 DEBUG:boto:Path: /qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:53.698 DEBUG:boto:Data: 2023-05-24T21:09:53.698 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.698 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:53.699 DEBUG:boto:Port: 8000 2023-05-24T21:09:53.699 DEBUG:boto:Params: {} 2023-05-24T21:09:53.699 DEBUG:boto:Token: None 2023-05-24T21:09:53.700 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:53 GMT /qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:53.700 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:AKrTGpKgJj/cwLkcWcLKiJLlixI= 2023-05-24T21:09:53.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:AKrTGpKgJj/cwLkcWcLKiJLlixI=', 'Content-Length': '0'} 2023-05-24T21:09:53.702 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:12 GMT'), ('x-amz-version-id', 'OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000083fdc389826c352d-00646e7d21-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.703 DEBUG:boto:path=/qskrps-12/obj-a1 2023-05-24T21:09:53.703 DEBUG:boto:auth_path=/qskrps-12/obj-a1 2023-05-24T21:09:53.704 DEBUG:boto:path=/qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:53.704 DEBUG:boto:auth_path=/qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:53.704 DEBUG:boto:Method: HEAD 2023-05-24T21:09:53.705 DEBUG:boto:Path: /qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:53.705 DEBUG:boto:Data: 2023-05-24T21:09:53.705 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.706 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:53.706 DEBUG:boto:Port: 8001 2023-05-24T21:09:53.706 DEBUG:boto:Params: {} 2023-05-24T21:09:53.707 DEBUG:boto:Token: None 2023-05-24T21:09:53.707 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:53 GMT /qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:53.707 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:t6KxhGYrDjkuxEjQuepo9OuSd1I= 2023-05-24T21:09:53.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:t6KxhGYrDjkuxEjQuepo9OuSd1I=', 'Content-Length': '0'} 2023-05-24T21:09:53.709 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:12 GMT'), ('x-amz-version-id', 'OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005465fc209580e00f-00646e7d21-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.710 DEBUG:boto:path=/qskrps-12/obj-a1 2023-05-24T21:09:53.710 DEBUG:boto:auth_path=/qskrps-12/obj-a1 2023-05-24T21:09:53.711 DEBUG:boto:path=/qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:53.711 DEBUG:boto:auth_path=/qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:53.711 DEBUG:boto:Method: HEAD 2023-05-24T21:09:53.712 DEBUG:boto:Path: /qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:53.712 DEBUG:boto:Data: 2023-05-24T21:09:53.713 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.713 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:53.713 DEBUG:boto:Port: 8000 2023-05-24T21:09:53.714 DEBUG:boto:Params: {} 2023-05-24T21:09:53.714 DEBUG:boto:Token: None 2023-05-24T21:09:53.714 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:53 GMT /qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:53.715 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:t6KxhGYrDjkuxEjQuepo9OuSd1I= 2023-05-24T21:09:53.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:t6KxhGYrDjkuxEjQuepo9OuSd1I=', 'Content-Length': '0'} 2023-05-24T21:09:53.717 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:12 GMT'), ('x-amz-version-id', 'OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001277dbb92c158311-00646e7d21-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-05-24T21:09:53.718 DEBUG:boto:path=/qskrps-12/obj-a1 2023-05-24T21:09:53.718 DEBUG:boto:auth_path=/qskrps-12/obj-a1 2023-05-24T21:09:53.719 DEBUG:boto:path=/qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:53.719 DEBUG:boto:auth_path=/qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:53.719 DEBUG:boto:Method: GET 2023-05-24T21:09:53.720 DEBUG:boto:Path: /qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:53.720 DEBUG:boto:Data: 2023-05-24T21:09:53.720 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.721 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:53.721 DEBUG:boto:Port: 8001 2023-05-24T21:09:53.722 DEBUG:boto:Params: {} 2023-05-24T21:09:53.722 DEBUG:boto:Token: None 2023-05-24T21:09:53.723 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:53 GMT /qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:53.723 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:AKrTGpKgJj/cwLkcWcLKiJLlixI= 2023-05-24T21:09:53.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:AKrTGpKgJj/cwLkcWcLKiJLlixI=', 'Content-Length': '0'} 2023-05-24T21:09:53.725 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:12 GMT'), ('x-amz-version-id', 'OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e1e773903d0dd17a-00646e7d21-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.726 DEBUG:boto:path=/qskrps-12/obj-a1 2023-05-24T21:09:53.726 DEBUG:boto:auth_path=/qskrps-12/obj-a1 2023-05-24T21:09:53.727 DEBUG:boto:path=/qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:53.727 DEBUG:boto:auth_path=/qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:53.727 DEBUG:boto:Method: GET 2023-05-24T21:09:53.728 DEBUG:boto:Path: /qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:53.728 DEBUG:boto:Data: 2023-05-24T21:09:53.728 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.729 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:53.729 DEBUG:boto:Port: 8000 2023-05-24T21:09:53.730 DEBUG:boto:Params: {} 2023-05-24T21:09:53.730 DEBUG:boto:Token: None 2023-05-24T21:09:53.730 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:53 GMT /qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:53.731 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:AKrTGpKgJj/cwLkcWcLKiJLlixI= 2023-05-24T21:09:53.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:AKrTGpKgJj/cwLkcWcLKiJLlixI=', 'Content-Length': '0'} 2023-05-24T21:09:53.733 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:12 GMT'), ('x-amz-version-id', 'OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000010a3998e86dea168-00646e7d21-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.734 DEBUG:boto:path=/qskrps-12/ 2023-05-24T21:09:53.734 DEBUG:boto:auth_path=/qskrps-12/ 2023-05-24T21:09:53.734 DEBUG:boto:path=/qskrps-12/?prefix=obj-a1 2023-05-24T21:09:53.735 DEBUG:boto:auth_path=/qskrps-12/?prefix=obj-a1 2023-05-24T21:09:53.735 DEBUG:boto:Method: GET 2023-05-24T21:09:53.735 DEBUG:boto:Path: /qskrps-12/?prefix=obj-a1 2023-05-24T21:09:53.736 DEBUG:boto:Data: 2023-05-24T21:09:53.736 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.736 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:53.737 DEBUG:boto:Port: 8001 2023-05-24T21:09:53.737 DEBUG:boto:Params: {} 2023-05-24T21:09:53.738 DEBUG:boto:Token: None 2023-05-24T21:09:53.738 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:53 GMT /qskrps-12/ 2023-05-24T21:09:53.738 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:7ulOuSFyzfND0cttgSz4vnt94vc= 2023-05-24T21:09:53.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:7ulOuSFyzfND0cttgSz4vnt94vc=', 'Content-Length': '0'} 2023-05-24T21:09:53.741 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ca74bb324197de6-00646e7d21-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.741 DEBUG:boto:b'qskrps-12obj-a11000falseobj-a12023-05-24T21:08:15.067Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:53.742 DEBUG:boto:path=/qskrps-12/ 2023-05-24T21:09:53.743 DEBUG:boto:auth_path=/qskrps-12/ 2023-05-24T21:09:53.743 DEBUG:boto:path=/qskrps-12/?prefix=obj-a1 2023-05-24T21:09:53.743 DEBUG:boto:auth_path=/qskrps-12/?prefix=obj-a1 2023-05-24T21:09:53.744 DEBUG:boto:Method: GET 2023-05-24T21:09:53.744 DEBUG:boto:Path: /qskrps-12/?prefix=obj-a1 2023-05-24T21:09:53.745 DEBUG:boto:Data: 2023-05-24T21:09:53.745 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.745 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:53.746 DEBUG:boto:Port: 8000 2023-05-24T21:09:53.746 DEBUG:boto:Params: {} 2023-05-24T21:09:53.746 DEBUG:boto:Token: None 2023-05-24T21:09:53.747 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:53 GMT /qskrps-12/ 2023-05-24T21:09:53.747 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:7ulOuSFyzfND0cttgSz4vnt94vc= 2023-05-24T21:09:53.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:7ulOuSFyzfND0cttgSz4vnt94vc=', 'Content-Length': '0'} 2023-05-24T21:09:53.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4860b5de6937b98-00646e7d21-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.792 DEBUG:boto:b'qskrps-12obj-a11000falseobj-a12023-05-24T21:08:15.067Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:53.793 DEBUG:boto:path=/qskrps-12/obj-a1 2023-05-24T21:09:53.794 DEBUG:boto:auth_path=/qskrps-12/obj-a1 2023-05-24T21:09:53.794 DEBUG:boto:Method: HEAD 2023-05-24T21:09:53.794 DEBUG:boto:Path: /qskrps-12/obj-a1 2023-05-24T21:09:53.795 DEBUG:boto:Data: 2023-05-24T21:09:53.795 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.796 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:53.796 DEBUG:boto:Port: 8001 2023-05-24T21:09:53.796 DEBUG:boto:Params: {} 2023-05-24T21:09:53.797 DEBUG:boto:Token: None 2023-05-24T21:09:53.797 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:53 GMT /qskrps-12/obj-a1 2023-05-24T21:09:53.798 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:qB8R6ZnJdnt1adWDa3aGS/7C+U0= 2023-05-24T21:09:53.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qB8R6ZnJdnt1adWDa3aGS/7C+U0=', 'Content-Length': '0'} 2023-05-24T21:09:53.801 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:12 GMT'), ('x-amz-version-id', 'OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c6f6d27951101e8c-00646e7d21-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.801 DEBUG:boto:path=/qskrps-12/obj-a1 2023-05-24T21:09:53.802 DEBUG:boto:auth_path=/qskrps-12/obj-a1 2023-05-24T21:09:53.802 DEBUG:boto:Method: HEAD 2023-05-24T21:09:53.802 DEBUG:boto:Path: /qskrps-12/obj-a1 2023-05-24T21:09:53.803 DEBUG:boto:Data: 2023-05-24T21:09:53.803 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.804 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:53.804 DEBUG:boto:Port: 8000 2023-05-24T21:09:53.804 DEBUG:boto:Params: {} 2023-05-24T21:09:53.805 DEBUG:boto:Token: None 2023-05-24T21:09:53.805 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:53 GMT /qskrps-12/obj-a1 2023-05-24T21:09:53.806 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:qB8R6ZnJdnt1adWDa3aGS/7C+U0= 2023-05-24T21:09:53.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qB8R6ZnJdnt1adWDa3aGS/7C+U0=', 'Content-Length': '0'} 2023-05-24T21:09:53.808 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:12 GMT'), ('x-amz-version-id', 'OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f36f7322663b45c-00646e7d21-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-05-24T21:09:53.809 DEBUG:boto:path=/qskrps-12/obj-a1 2023-05-24T21:09:53.809 DEBUG:boto:auth_path=/qskrps-12/obj-a1 2023-05-24T21:09:53.810 DEBUG:boto:path=/qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:53.810 DEBUG:boto:auth_path=/qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:53.811 DEBUG:boto:Method: GET 2023-05-24T21:09:53.811 DEBUG:boto:Path: /qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:53.811 DEBUG:boto:Data: 2023-05-24T21:09:53.812 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.812 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:53.813 DEBUG:boto:Port: 8001 2023-05-24T21:09:53.813 DEBUG:boto:Params: {} 2023-05-24T21:09:53.813 DEBUG:boto:Token: None 2023-05-24T21:09:53.814 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:53 GMT /qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:53.815 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:AKrTGpKgJj/cwLkcWcLKiJLlixI= 2023-05-24T21:09:53.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:AKrTGpKgJj/cwLkcWcLKiJLlixI=', 'Content-Length': '0'} 2023-05-24T21:09:53.817 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:12 GMT'), ('x-amz-version-id', 'OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002c0bbc12bca28552-00646e7d21-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.817 DEBUG:boto:path=/qskrps-12/obj-a1 2023-05-24T21:09:53.818 DEBUG:boto:auth_path=/qskrps-12/obj-a1 2023-05-24T21:09:53.818 DEBUG:boto:path=/qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:53.819 DEBUG:boto:auth_path=/qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:53.819 DEBUG:boto:Method: GET 2023-05-24T21:09:53.820 DEBUG:boto:Path: /qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:53.820 DEBUG:boto:Data: 2023-05-24T21:09:53.820 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.821 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:53.821 DEBUG:boto:Port: 8000 2023-05-24T21:09:53.821 DEBUG:boto:Params: {} 2023-05-24T21:09:53.822 DEBUG:boto:Token: None 2023-05-24T21:09:53.822 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:53 GMT /qskrps-12/obj-a1?versionId=OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV 2023-05-24T21:09:53.823 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:AKrTGpKgJj/cwLkcWcLKiJLlixI= 2023-05-24T21:09:53.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:AKrTGpKgJj/cwLkcWcLKiJLlixI=', 'Content-Length': '0'} 2023-05-24T21:09:53.826 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:12 GMT'), ('x-amz-version-id', 'OGCpJI.tGUTZ0Nr4IKrf-55mu.IxEiV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eb7ac1a9e299bcd1-00646e7d21-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.827 DEBUG:boto:path=/qskrps-12/ 2023-05-24T21:09:53.828 DEBUG:boto:auth_path=/qskrps-12/ 2023-05-24T21:09:53.828 DEBUG:boto:path=/qskrps-12/?prefix=obj-a1 2023-05-24T21:09:53.829 DEBUG:boto:auth_path=/qskrps-12/?prefix=obj-a1 2023-05-24T21:09:53.830 DEBUG:boto:Method: GET 2023-05-24T21:09:53.830 DEBUG:boto:Path: /qskrps-12/?prefix=obj-a1 2023-05-24T21:09:53.831 DEBUG:boto:Data: 2023-05-24T21:09:53.831 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.832 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:53.832 DEBUG:boto:Port: 8001 2023-05-24T21:09:53.832 DEBUG:boto:Params: {} 2023-05-24T21:09:53.833 DEBUG:boto:Token: None 2023-05-24T21:09:53.833 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:53 GMT /qskrps-12/ 2023-05-24T21:09:53.834 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:7ulOuSFyzfND0cttgSz4vnt94vc= 2023-05-24T21:09:53.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:7ulOuSFyzfND0cttgSz4vnt94vc=', 'Content-Length': '0'} 2023-05-24T21:09:53.836 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011ab4bd743538691-00646e7d21-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.876 DEBUG:boto:b'qskrps-12obj-a11000falseobj-a12023-05-24T21:08:15.067Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:53.877 DEBUG:boto:path=/qskrps-12/ 2023-05-24T21:09:53.877 DEBUG:boto:auth_path=/qskrps-12/ 2023-05-24T21:09:53.878 DEBUG:boto:path=/qskrps-12/?prefix=obj-a1 2023-05-24T21:09:53.878 DEBUG:boto:auth_path=/qskrps-12/?prefix=obj-a1 2023-05-24T21:09:53.879 DEBUG:boto:Method: GET 2023-05-24T21:09:53.879 DEBUG:boto:Path: /qskrps-12/?prefix=obj-a1 2023-05-24T21:09:53.879 DEBUG:boto:Data: 2023-05-24T21:09:53.880 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.880 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:53.880 DEBUG:boto:Port: 8000 2023-05-24T21:09:53.881 DEBUG:boto:Params: {} 2023-05-24T21:09:53.881 DEBUG:boto:Token: None 2023-05-24T21:09:53.882 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:53 GMT /qskrps-12/ 2023-05-24T21:09:53.882 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:7ulOuSFyzfND0cttgSz4vnt94vc= 2023-05-24T21:09:53.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:7ulOuSFyzfND0cttgSz4vnt94vc=', 'Content-Length': '0'} 2023-05-24T21:09:53.885 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b77099d7a548c9ca-00646e7d21-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.886 DEBUG:boto:b'qskrps-12obj-a11000falseobj-a12023-05-24T21:08:15.067Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:53.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-05-24T21:09:53.888 DEBUG:boto:path=/qskrps-12/obj-a2 2023-05-24T21:09:53.889 DEBUG:boto:auth_path=/qskrps-12/obj-a2 2023-05-24T21:09:53.890 DEBUG:boto:path=/qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:53.891 DEBUG:boto:auth_path=/qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:53.892 DEBUG:boto:Method: GET 2023-05-24T21:09:53.893 DEBUG:boto:Path: /qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:53.893 DEBUG:boto:Data: 2023-05-24T21:09:53.894 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.895 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:53.895 DEBUG:boto:Port: 8001 2023-05-24T21:09:53.896 DEBUG:boto:Params: {} 2023-05-24T21:09:53.897 DEBUG:boto:Token: None 2023-05-24T21:09:53.898 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:53 GMT /qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:53.898 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:gDZgIceyjA2veE3zTa/ZjqTRV4k= 2023-05-24T21:09:53.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:gDZgIceyjA2veE3zTa/ZjqTRV4k=', 'Content-Length': '0'} 2023-05-24T21:09:53.901 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:12 GMT'), ('x-amz-version-id', 'PcWX5rYNtc410b9LwKxno3LHHZeAmhm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000106383d8a93c388a-00646e7d21-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.902 DEBUG:boto:path=/qskrps-12/obj-a2 2023-05-24T21:09:53.902 DEBUG:boto:auth_path=/qskrps-12/obj-a2 2023-05-24T21:09:53.902 DEBUG:boto:path=/qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:53.903 DEBUG:boto:auth_path=/qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:53.904 DEBUG:boto:Method: GET 2023-05-24T21:09:53.904 DEBUG:boto:Path: /qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:53.905 DEBUG:boto:Data: 2023-05-24T21:09:53.905 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.905 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:53.906 DEBUG:boto:Port: 8000 2023-05-24T21:09:53.906 DEBUG:boto:Params: {} 2023-05-24T21:09:53.907 DEBUG:boto:Token: None 2023-05-24T21:09:53.907 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:53 GMT /qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:53.908 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:gDZgIceyjA2veE3zTa/ZjqTRV4k= 2023-05-24T21:09:53.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:gDZgIceyjA2veE3zTa/ZjqTRV4k=', 'Content-Length': '0'} 2023-05-24T21:09:53.910 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:12 GMT'), ('x-amz-version-id', 'PcWX5rYNtc410b9LwKxno3LHHZeAmhm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005b0165a8056dbe81-00646e7d21-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.911 DEBUG:boto:path=/qskrps-12/obj-a2 2023-05-24T21:09:53.911 DEBUG:boto:auth_path=/qskrps-12/obj-a2 2023-05-24T21:09:53.912 DEBUG:boto:path=/qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:53.912 DEBUG:boto:auth_path=/qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:53.912 DEBUG:boto:Method: HEAD 2023-05-24T21:09:53.913 DEBUG:boto:Path: /qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:53.913 DEBUG:boto:Data: 2023-05-24T21:09:53.914 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.914 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:53.915 DEBUG:boto:Port: 8001 2023-05-24T21:09:53.915 DEBUG:boto:Params: {} 2023-05-24T21:09:53.916 DEBUG:boto:Token: None 2023-05-24T21:09:53.916 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:53 GMT /qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:53.916 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:cIGun210SbIPUosZpFnbrQ27d/s= 2023-05-24T21:09:53.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:cIGun210SbIPUosZpFnbrQ27d/s=', 'Content-Length': '0'} 2023-05-24T21:09:53.919 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:12 GMT'), ('x-amz-version-id', 'PcWX5rYNtc410b9LwKxno3LHHZeAmhm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000963e9995f6b28d03-00646e7d21-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.919 DEBUG:boto:path=/qskrps-12/obj-a2 2023-05-24T21:09:53.920 DEBUG:boto:auth_path=/qskrps-12/obj-a2 2023-05-24T21:09:53.920 DEBUG:boto:path=/qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:53.921 DEBUG:boto:auth_path=/qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:53.921 DEBUG:boto:Method: HEAD 2023-05-24T21:09:53.922 DEBUG:boto:Path: /qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:53.922 DEBUG:boto:Data: 2023-05-24T21:09:53.923 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.923 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:53.923 DEBUG:boto:Port: 8000 2023-05-24T21:09:53.924 DEBUG:boto:Params: {} 2023-05-24T21:09:53.924 DEBUG:boto:Token: None 2023-05-24T21:09:53.924 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:53 GMT /qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:53.925 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:cIGun210SbIPUosZpFnbrQ27d/s= 2023-05-24T21:09:53.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:cIGun210SbIPUosZpFnbrQ27d/s=', 'Content-Length': '0'} 2023-05-24T21:09:53.927 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:12 GMT'), ('x-amz-version-id', 'PcWX5rYNtc410b9LwKxno3LHHZeAmhm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001c5e9e1c1e42a87d-00646e7d21-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-05-24T21:09:53.929 DEBUG:boto:path=/qskrps-12/obj-a2 2023-05-24T21:09:53.929 DEBUG:boto:auth_path=/qskrps-12/obj-a2 2023-05-24T21:09:53.930 DEBUG:boto:path=/qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:53.931 DEBUG:boto:auth_path=/qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:53.931 DEBUG:boto:Method: GET 2023-05-24T21:09:53.932 DEBUG:boto:Path: /qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:53.933 DEBUG:boto:Data: 2023-05-24T21:09:53.933 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.934 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:53.935 DEBUG:boto:Port: 8001 2023-05-24T21:09:53.936 DEBUG:boto:Params: {} 2023-05-24T21:09:53.936 DEBUG:boto:Token: None 2023-05-24T21:09:53.937 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:53 GMT /qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:53.938 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:gDZgIceyjA2veE3zTa/ZjqTRV4k= 2023-05-24T21:09:53.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:gDZgIceyjA2veE3zTa/ZjqTRV4k=', 'Content-Length': '0'} 2023-05-24T21:09:53.941 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:12 GMT'), ('x-amz-version-id', 'PcWX5rYNtc410b9LwKxno3LHHZeAmhm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006521d9e18e490fa9-00646e7d21-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.941 DEBUG:boto:path=/qskrps-12/obj-a2 2023-05-24T21:09:53.942 DEBUG:boto:auth_path=/qskrps-12/obj-a2 2023-05-24T21:09:53.943 DEBUG:boto:path=/qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:53.943 DEBUG:boto:auth_path=/qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:53.944 DEBUG:boto:Method: GET 2023-05-24T21:09:53.945 DEBUG:boto:Path: /qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:53.946 DEBUG:boto:Data: 2023-05-24T21:09:53.946 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.947 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:53.948 DEBUG:boto:Port: 8000 2023-05-24T21:09:53.948 DEBUG:boto:Params: {} 2023-05-24T21:09:53.949 DEBUG:boto:Token: None 2023-05-24T21:09:53.950 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:53 GMT /qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:53.950 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:gDZgIceyjA2veE3zTa/ZjqTRV4k= 2023-05-24T21:09:53.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:gDZgIceyjA2veE3zTa/ZjqTRV4k=', 'Content-Length': '0'} 2023-05-24T21:09:53.953 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:12 GMT'), ('x-amz-version-id', 'PcWX5rYNtc410b9LwKxno3LHHZeAmhm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002d03cf35f6233914-00646e7d21-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:53.954 DEBUG:boto:path=/qskrps-12/ 2023-05-24T21:09:53.954 DEBUG:boto:auth_path=/qskrps-12/ 2023-05-24T21:09:53.955 DEBUG:boto:path=/qskrps-12/?prefix=obj-a2 2023-05-24T21:09:53.955 DEBUG:boto:auth_path=/qskrps-12/?prefix=obj-a2 2023-05-24T21:09:53.955 DEBUG:boto:Method: GET 2023-05-24T21:09:53.956 DEBUG:boto:Path: /qskrps-12/?prefix=obj-a2 2023-05-24T21:09:53.956 DEBUG:boto:Data: 2023-05-24T21:09:53.957 DEBUG:boto:Headers: {} 2023-05-24T21:09:53.957 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:53.957 DEBUG:boto:Port: 8001 2023-05-24T21:09:53.958 DEBUG:boto:Params: {} 2023-05-24T21:09:53.958 DEBUG:boto:Token: None 2023-05-24T21:09:53.958 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:53 GMT /qskrps-12/ 2023-05-24T21:09:53.959 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:7ulOuSFyzfND0cttgSz4vnt94vc= 2023-05-24T21:09:53.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:7ulOuSFyzfND0cttgSz4vnt94vc=', 'Content-Length': '0'} 2023-05-24T21:09:53.961 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000928965edaae26100-00646e7d21-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:54.004 DEBUG:boto:b'qskrps-12obj-a21000falseobj-a22023-05-24T21:08:15.193Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:54.005 DEBUG:boto:path=/qskrps-12/ 2023-05-24T21:09:54.005 DEBUG:boto:auth_path=/qskrps-12/ 2023-05-24T21:09:54.006 DEBUG:boto:path=/qskrps-12/?prefix=obj-a2 2023-05-24T21:09:54.006 DEBUG:boto:auth_path=/qskrps-12/?prefix=obj-a2 2023-05-24T21:09:54.007 DEBUG:boto:Method: GET 2023-05-24T21:09:54.007 DEBUG:boto:Path: /qskrps-12/?prefix=obj-a2 2023-05-24T21:09:54.007 DEBUG:boto:Data: 2023-05-24T21:09:54.008 DEBUG:boto:Headers: {} 2023-05-24T21:09:54.008 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:54.008 DEBUG:boto:Port: 8000 2023-05-24T21:09:54.009 DEBUG:boto:Params: {} 2023-05-24T21:09:54.009 DEBUG:boto:Token: None 2023-05-24T21:09:54.010 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:54 GMT /qskrps-12/ 2023-05-24T21:09:54.010 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:VNbiq3M3qkr/Zv5n/ipHy3hN1sI= 2023-05-24T21:09:54.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:VNbiq3M3qkr/Zv5n/ipHy3hN1sI=', 'Content-Length': '0'} 2023-05-24T21:09:54.012 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd272813d6d734b4-00646e7d22-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:54.013 DEBUG:boto:b'qskrps-12obj-a21000falseobj-a22023-05-24T21:08:15.193Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:54.013 DEBUG:boto:path=/qskrps-12/obj-a2 2023-05-24T21:09:54.014 DEBUG:boto:auth_path=/qskrps-12/obj-a2 2023-05-24T21:09:54.014 DEBUG:boto:Method: HEAD 2023-05-24T21:09:54.014 DEBUG:boto:Path: /qskrps-12/obj-a2 2023-05-24T21:09:54.015 DEBUG:boto:Data: 2023-05-24T21:09:54.015 DEBUG:boto:Headers: {} 2023-05-24T21:09:54.015 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:54.016 DEBUG:boto:Port: 8001 2023-05-24T21:09:54.016 DEBUG:boto:Params: {} 2023-05-24T21:09:54.017 DEBUG:boto:Token: None 2023-05-24T21:09:54.017 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:54 GMT /qskrps-12/obj-a2 2023-05-24T21:09:54.017 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:1hV3p6j35WG9tCQP47HOEYLRhF0= 2023-05-24T21:09:54.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:1hV3p6j35WG9tCQP47HOEYLRhF0=', 'Content-Length': '0'} 2023-05-24T21:09:54.020 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:12 GMT'), ('x-amz-version-id', 'PcWX5rYNtc410b9LwKxno3LHHZeAmhm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006faa5b0973d5e807-00646e7d22-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:54.020 DEBUG:boto:path=/qskrps-12/obj-a2 2023-05-24T21:09:54.021 DEBUG:boto:auth_path=/qskrps-12/obj-a2 2023-05-24T21:09:54.021 DEBUG:boto:Method: HEAD 2023-05-24T21:09:54.021 DEBUG:boto:Path: /qskrps-12/obj-a2 2023-05-24T21:09:54.022 DEBUG:boto:Data: 2023-05-24T21:09:54.022 DEBUG:boto:Headers: {} 2023-05-24T21:09:54.022 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:54.023 DEBUG:boto:Port: 8000 2023-05-24T21:09:54.023 DEBUG:boto:Params: {} 2023-05-24T21:09:54.023 DEBUG:boto:Token: None 2023-05-24T21:09:54.024 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:54 GMT /qskrps-12/obj-a2 2023-05-24T21:09:54.024 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:1hV3p6j35WG9tCQP47HOEYLRhF0= 2023-05-24T21:09:54.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:1hV3p6j35WG9tCQP47HOEYLRhF0=', 'Content-Length': '0'} 2023-05-24T21:09:54.026 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:12 GMT'), ('x-amz-version-id', 'PcWX5rYNtc410b9LwKxno3LHHZeAmhm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bdee4e4a58f5910b-00646e7d22-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:54.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-05-24T21:09:54.027 DEBUG:boto:path=/qskrps-12/obj-a2 2023-05-24T21:09:54.027 DEBUG:boto:auth_path=/qskrps-12/obj-a2 2023-05-24T21:09:54.028 DEBUG:boto:path=/qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:54.028 DEBUG:boto:auth_path=/qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:54.029 DEBUG:boto:Method: GET 2023-05-24T21:09:54.029 DEBUG:boto:Path: /qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:54.029 DEBUG:boto:Data: 2023-05-24T21:09:54.030 DEBUG:boto:Headers: {} 2023-05-24T21:09:54.030 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:54.030 DEBUG:boto:Port: 8001 2023-05-24T21:09:54.031 DEBUG:boto:Params: {} 2023-05-24T21:09:54.031 DEBUG:boto:Token: None 2023-05-24T21:09:54.031 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:54 GMT /qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:54.032 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:LDbTSrMfY1NLGamfZ17LYXG2dVQ= 2023-05-24T21:09:54.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:LDbTSrMfY1NLGamfZ17LYXG2dVQ=', 'Content-Length': '0'} 2023-05-24T21:09:54.034 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:12 GMT'), ('x-amz-version-id', 'PcWX5rYNtc410b9LwKxno3LHHZeAmhm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fc843e632a2ac86a-00646e7d22-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:54.035 DEBUG:boto:path=/qskrps-12/obj-a2 2023-05-24T21:09:54.035 DEBUG:boto:auth_path=/qskrps-12/obj-a2 2023-05-24T21:09:54.035 DEBUG:boto:path=/qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:54.036 DEBUG:boto:auth_path=/qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:54.036 DEBUG:boto:Method: GET 2023-05-24T21:09:54.036 DEBUG:boto:Path: /qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:54.037 DEBUG:boto:Data: 2023-05-24T21:09:54.037 DEBUG:boto:Headers: {} 2023-05-24T21:09:54.037 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:54.038 DEBUG:boto:Port: 8000 2023-05-24T21:09:54.038 DEBUG:boto:Params: {} 2023-05-24T21:09:54.039 DEBUG:boto:Token: None 2023-05-24T21:09:54.039 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:54 GMT /qskrps-12/obj-a2?versionId=PcWX5rYNtc410b9LwKxno3LHHZeAmhm 2023-05-24T21:09:54.039 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:LDbTSrMfY1NLGamfZ17LYXG2dVQ= 2023-05-24T21:09:54.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:LDbTSrMfY1NLGamfZ17LYXG2dVQ=', 'Content-Length': '0'} 2023-05-24T21:09:54.041 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:12 GMT'), ('x-amz-version-id', 'PcWX5rYNtc410b9LwKxno3LHHZeAmhm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aff266c466ab03f7-00646e7d22-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:54.084 DEBUG:boto:path=/qskrps-12/ 2023-05-24T21:09:54.085 DEBUG:boto:auth_path=/qskrps-12/ 2023-05-24T21:09:54.085 DEBUG:boto:path=/qskrps-12/?prefix=obj-a2 2023-05-24T21:09:54.085 DEBUG:boto:auth_path=/qskrps-12/?prefix=obj-a2 2023-05-24T21:09:54.086 DEBUG:boto:Method: GET 2023-05-24T21:09:54.086 DEBUG:boto:Path: /qskrps-12/?prefix=obj-a2 2023-05-24T21:09:54.087 DEBUG:boto:Data: 2023-05-24T21:09:54.087 DEBUG:boto:Headers: {} 2023-05-24T21:09:54.087 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:09:54.088 DEBUG:boto:Port: 8001 2023-05-24T21:09:54.088 DEBUG:boto:Params: {} 2023-05-24T21:09:54.088 DEBUG:boto:Token: None 2023-05-24T21:09:54.089 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:54 GMT /qskrps-12/ 2023-05-24T21:09:54.089 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:VNbiq3M3qkr/Zv5n/ipHy3hN1sI= 2023-05-24T21:09:54.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:VNbiq3M3qkr/Zv5n/ipHy3hN1sI=', 'Content-Length': '0'} 2023-05-24T21:09:54.092 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f59f453e807ccda5-00646e7d22-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:54.092 DEBUG:boto:b'qskrps-12obj-a21000falseobj-a22023-05-24T21:08:15.193Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:54.093 DEBUG:boto:path=/qskrps-12/ 2023-05-24T21:09:54.094 DEBUG:boto:auth_path=/qskrps-12/ 2023-05-24T21:09:54.094 DEBUG:boto:path=/qskrps-12/?prefix=obj-a2 2023-05-24T21:09:54.094 DEBUG:boto:auth_path=/qskrps-12/?prefix=obj-a2 2023-05-24T21:09:54.095 DEBUG:boto:Method: GET 2023-05-24T21:09:54.095 DEBUG:boto:Path: /qskrps-12/?prefix=obj-a2 2023-05-24T21:09:54.095 DEBUG:boto:Data: 2023-05-24T21:09:54.096 DEBUG:boto:Headers: {} 2023-05-24T21:09:54.096 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:54.096 DEBUG:boto:Port: 8000 2023-05-24T21:09:54.097 DEBUG:boto:Params: {} 2023-05-24T21:09:54.097 DEBUG:boto:Token: None 2023-05-24T21:09:54.098 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:09:54 GMT /qskrps-12/ 2023-05-24T21:09:54.098 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:VNbiq3M3qkr/Zv5n/ipHy3hN1sI= 2023-05-24T21:09:54.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:VNbiq3M3qkr/Zv5n/ipHy3hN1sI=', 'Content-Length': '0'} 2023-05-24T21:09:54.100 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4a0de2b88344822-00646e7d22-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:54.101 DEBUG:boto:b'qskrps-12obj-a21000falseobj-a22023-05-24T21:08:15.193Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:09:54.102 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qskrps-12 zones={a2, a1} 2023-05-24T21:09:54.102 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2023-05-24T21:09:54.104 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:09:54.104 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:09:54.104 DEBUG:boto:Method: PUT 2023-05-24T21:09:54.105 DEBUG:boto:Path: /qskrps-13/ 2023-05-24T21:09:54.105 DEBUG:boto:Data: 2023-05-24T21:09:54.105 DEBUG:boto:Headers: {} 2023-05-24T21:09:54.106 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:54.106 DEBUG:boto:Port: 8000 2023-05-24T21:09:54.106 DEBUG:boto:Params: {} 2023-05-24T21:09:54.107 DEBUG:boto:Token: None 2023-05-24T21:09:54.107 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:09:54 GMT /qskrps-13/ 2023-05-24T21:09:54.107 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5uK8WyYhcus5qHQ2M61s63IXcw8= 2023-05-24T21:09:54.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5uK8WyYhcus5qHQ2M61s63IXcw8=', 'Content-Length': '0'} 2023-05-24T21:09:54.121 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e98f3b014e9c52fe-00646e7d22-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:54.121 DEBUG:rgw_multi.tests:created bucket=qskrps-13 2023-05-24T21:09:54.121 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:09:54.122 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:09:54.122 DEBUG:boto:path=/qskrps-13/?versioning 2023-05-24T21:09:54.122 DEBUG:boto:auth_path=/qskrps-13/?versioning 2023-05-24T21:09:54.123 DEBUG:boto:Method: PUT 2023-05-24T21:09:54.123 DEBUG:boto:Path: /qskrps-13/?versioning 2023-05-24T21:09:54.123 DEBUG:boto:Data: Enabled Disabled 2023-05-24T21:09:54.124 DEBUG:boto:Headers: {} 2023-05-24T21:09:54.124 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:54.124 DEBUG:boto:Port: 8000 2023-05-24T21:09:54.125 DEBUG:boto:Params: {} 2023-05-24T21:09:54.125 DEBUG:boto:Token: None 2023-05-24T21:09:54.125 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:09:54 GMT /qskrps-13/?versioning 2023-05-24T21:09:54.126 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:HcFrwk0jV4KY7BBOhPLHSoN6y/w= 2023-05-24T21:09:54.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:HcFrwk0jV4KY7BBOhPLHSoN6y/w=', 'Content-Length': '228'} 2023-05-24T21:09:54.131 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f8a8d4eec5c7ab48-00646e7d22-110d-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:54.132 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:09:54.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:09:54.133 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:09:54.181 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:09:54.181 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:09:54.221 INFO:teuthology.orchestra.run.smithi071.stderr:No --period given, using current period=5c04936b-70ea-4415-b4eb-3aad3dc612e5 2023-05-24T21:09:54.240 INFO:teuthology.orchestra.run.smithi071.stdout:[ 2023-05-24T21:09:54.240 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:09:54.240 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962594.125917_123.1", 2023-05-24T21:09:54.241 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:09:54.125917Z" 2023-05-24T21:09:54.241 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:09:54.241 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:09:54.241 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:09:54.241 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:06:32.333844Z" 2023-05-24T21:09:54.241 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:09:54.241 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:09:54.242 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:09:54.242 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:09:54.242 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:09:54.242 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:09:54.242 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:09:54.242 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:09:54.243 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:09:54.243 INFO:teuthology.orchestra.run.smithi071.stdout:] 2023-05-24T21:09:54.244 DEBUG:rgw_multi.tests:master meta markers={0: '1_1684962594.125917_123.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:09:54.244 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-24T21:09:54.245 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:09:54.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:09:54.245 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:09:54.296 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:09:54.297 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:09:54.349 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:09:54.342+0000 7f63ebd99780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:09:54.349 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:09:54.343+0000 7f63ebd99780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:09:54.370 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:09:54.370 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:09:54.370 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:09:54.370 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:09:54.370 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:09:54.371 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:09:54.371 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:09:54.371 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:09:54.371 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:09:54.371 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:09:54.371 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:09:54.371 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:09:54.372 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:09:54.372 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962454.049579_106.1", 2023-05-24T21:09:54.372 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:09:54.372 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:09:54.372 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:09:54.372 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:07:34.049579Z", 2023-05-24T21:09:54.373 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:09:54.373 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:09:54.373 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:09:54.373 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:09:54.373 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:09:54.373 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:09:54.373 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:09:54.374 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:09:54.374 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:09:54.374 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:09:54.374 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:09:54.374 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:09:54.374 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:09:54.374 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:09:54.375 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:09:54.375 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:09:54.375 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:09:54.375 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:09:54.375 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:09:54.375 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:09:54.375 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:09:54.376 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:09:54.376 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:09:54.376 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:09:54.376 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:09:54.376 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:09:54.376 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:09:54.376 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:09:54.377 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:09:54.377 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:09:54.377 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:09:54.377 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:09:54.377 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:09:54.377 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:09:54.377 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:09:54.378 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:09:54.378 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:09:54.378 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:09:54.378 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:09:54.378 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:09:54.378 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:09:54.378 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:09:54.379 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:09:54.379 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:09:54.379 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:09:54.379 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:09:54Z" 2023-05-24T21:09:54.379 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:09:54.380 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962454.049579_106.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:07:34.049579Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:09:54Z" } 2023-05-24T21:09:54.381 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962454.049579_106.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:07:34.049579Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:09:54.381 DEBUG:rgw_multi.tests:log_status={0: '1_1684962594.125917_123.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:09:54.381 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962454.049579_106.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:09:54.382 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684962594.125917_123.1 target=1_1684962454.049579_106.1 2023-05-24T21:09:59.383 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:09:59.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:09:59.384 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:09:59.442 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:09:59.442 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:09:59.478 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:09:59.471+0000 7ff357277780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:09:59.479 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:09:59.472+0000 7ff357277780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:09:59.499 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:09:59.499 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:09:59.500 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:09:59.500 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:09:59.500 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:09:59.500 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:09:59.500 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:09:59.501 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:09:59.501 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:09:59.501 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:09:59.501 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:09:59.501 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:09:59.501 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:09:59.501 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962594.125917_123.1", 2023-05-24T21:09:59.502 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:09:59.502 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:09:59.502 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:09:59.502 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:09:54.125917Z", 2023-05-24T21:09:59.502 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:09:59.502 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:09:59.502 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:09:59.503 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:09:59.503 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:09:59.503 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:09:59.503 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:09:59.503 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:09:59.503 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:09:59.503 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:09:59.504 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:09:59.504 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:09:59.504 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:09:59.504 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:09:59.504 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:09:59.504 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:09:59.505 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:09:59.505 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:09:59.505 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:09:59.505 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:09:59.505 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:09:59.505 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:09:59.505 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:09:59.506 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:09:59.506 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:09:59.506 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:09:59.506 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:09:59.506 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:09:59.506 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:09:59.506 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:09:59.507 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:09:59.507 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:09:59.507 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:09:59.507 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:09:59.507 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:09:59.507 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:09:59.507 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:09:59.508 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:09:59.508 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:09:59.508 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:09:59.508 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:09:59.508 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:09:59.508 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:09:59.508 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:09:59.509 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:09:59.509 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:09:59Z" 2023-05-24T21:09:59.509 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:09:59.510 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962594.125917_123.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:09:54.125917Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:09:59Z" } 2023-05-24T21:09:59.510 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962594.125917_123.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:09:54.125917Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:09:59.511 DEBUG:rgw_multi.tests:log_status={0: '1_1684962594.125917_123.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:09:59.511 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962594.125917_123.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:09:59.511 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-24T21:09:59.512 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:09:59.512 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:09:59.513 DEBUG:boto:Method: HEAD 2023-05-24T21:09:59.513 DEBUG:boto:Path: /qskrps-13/ 2023-05-24T21:09:59.513 DEBUG:boto:Data: 2023-05-24T21:09:59.514 DEBUG:boto:Headers: {} 2023-05-24T21:09:59.514 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:59.514 DEBUG:boto:Port: 8000 2023-05-24T21:09:59.515 DEBUG:boto:Params: {} 2023-05-24T21:09:59.515 DEBUG:boto:Token: None 2023-05-24T21:09:59.515 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:09:59 GMT /qskrps-13/ 2023-05-24T21:09:59.516 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Iq7tZKbpqDO+TKjKjQTSswc6/ug= 2023-05-24T21:09:59.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:09:59 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Iq7tZKbpqDO+TKjKjQTSswc6/ug=', 'Content-Length': '0'} 2023-05-24T21:09:59.518 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003698c2abf3c70504-00646e7d27-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:09:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:59.519 DEBUG:boto:path=/qskrps-13/dummy 2023-05-24T21:09:59.520 DEBUG:boto:auth_path=/qskrps-13/dummy 2023-05-24T21:09:59.520 DEBUG:boto:Method: PUT 2023-05-24T21:09:59.520 DEBUG:boto:Path: /qskrps-13/dummy 2023-05-24T21:09:59.521 DEBUG:boto:Data: 2023-05-24T21:09:59.521 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-05-24T21:09:59.521 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:09:59.522 DEBUG:boto:Port: 8000 2023-05-24T21:09:59.522 DEBUG:boto:Params: {} 2023-05-24T21:09:59.522 DEBUG:boto:Token: None 2023-05-24T21:09:59.523 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Wed, 24 May 2023 21:09:59 GMT /qskrps-13/dummy 2023-05-24T21:09:59.523 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:xxhRbJcOTgCXqVAR0919shP/jps= 2023-05-24T21:09:59.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:09:59 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:xxhRbJcOTgCXqVAR0919shP/jps='} 2023-05-24T21:09:59.547 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX'), ('x-amz-request-id', 'tx000007b7ec80693032df1-00646e7d27-110d-a1'), ('Date', 'Wed, 24 May 2023 21:09:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:09:59.547 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=qskrps-13 2023-05-24T21:09:59.548 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:09:59.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:09:59.549 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-13 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:09:59.608 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:09:59.608 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:09:59.656 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:09:59.649+0000 7f31ca71b780 1 waiting to reach incremental sync.. 2023-05-24T21:10:04.657 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:10:04.651+0000 7f31ca71b780 1 waiting to reach incremental sync.. 2023-05-24T21:10:09.657 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:10:09.651+0000 7f31ca71b780 1 waiting to reach incremental sync.. 2023-05-24T21:10:14.659 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:10:14.653+0000 7f31ca71b780 1 bucket sync caught up with source: 2023-05-24T21:10:14.659 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000002.6.11, , , , , ] 2023-05-24T21:10:14.660 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000002.6.11, , , , , ] 2023-05-24T21:10:14.660 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:10:14.653+0000 7f31ca71b780 0 bucket checkpoint complete 2023-05-24T21:10:14.681 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=qskrps-13 2023-05-24T21:10:14.681 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:10:14.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:10:14.682 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-13 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:10:14.745 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:10:14.745 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:10:14.798 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:10:14.793+0000 7fb856240780 1 waiting to reach incremental sync.. 2023-05-24T21:10:19.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:10:19.793+0000 7fb856240780 1 waiting to reach incremental sync.. 2023-05-24T21:10:24.800 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:10:24.794+0000 7fb856240780 1 waiting to reach incremental sync.. 2023-05-24T21:10:29.800 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:10:29.794+0000 7fb856240780 1 waiting to reach incremental sync.. 2023-05-24T21:10:34.802 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:10:34.796+0000 7fb856240780 1 bucket sync caught up with source: 2023-05-24T21:10:34.802 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000002.15.11, , , , , ] 2023-05-24T21:10:34.803 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000002.15.11, , , , , ] 2023-05-24T21:10:34.803 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:10:34.796+0000 7fb856240780 0 bucket checkpoint complete 2023-05-24T21:10:34.818 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qskrps-13 zones={a2, a1} 2023-05-24T21:10:34.818 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:10:34.819 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:10:34.819 DEBUG:boto:Method: HEAD 2023-05-24T21:10:34.819 DEBUG:boto:Path: /qskrps-13/ 2023-05-24T21:10:34.820 DEBUG:boto:Data: 2023-05-24T21:10:34.820 DEBUG:boto:Headers: {} 2023-05-24T21:10:34.821 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:10:34.821 DEBUG:boto:Port: 8001 2023-05-24T21:10:34.822 DEBUG:boto:Params: {} 2023-05-24T21:10:34.823 DEBUG:boto:Token: None 2023-05-24T21:10:34.823 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:10:34 GMT /qskrps-13/ 2023-05-24T21:10:34.824 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Uviib8zAgSJTLr/cWcUmxA7thbM= 2023-05-24T21:10:34.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:10:34 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Uviib8zAgSJTLr/cWcUmxA7thbM=', 'Content-Length': '0'} 2023-05-24T21:10:34.828 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a8a05d524e1ea9db-00646e7d4a-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:10:34 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:34.829 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:10:34.829 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:10:34.830 DEBUG:boto:Method: HEAD 2023-05-24T21:10:34.831 DEBUG:boto:Path: /qskrps-13/ 2023-05-24T21:10:34.831 DEBUG:boto:Data: 2023-05-24T21:10:34.832 DEBUG:boto:Headers: {} 2023-05-24T21:10:34.833 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:10:34.833 DEBUG:boto:Port: 8000 2023-05-24T21:10:34.834 DEBUG:boto:Params: {} 2023-05-24T21:10:34.835 DEBUG:boto:Token: None 2023-05-24T21:10:34.835 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:10:34 GMT /qskrps-13/ 2023-05-24T21:10:34.836 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Uviib8zAgSJTLr/cWcUmxA7thbM= 2023-05-24T21:10:34.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:10:34 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Uviib8zAgSJTLr/cWcUmxA7thbM=', 'Content-Length': '0'} 2023-05-24T21:10:34.840 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000503ab5febc1b2ff4-00646e7d4a-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:10:34 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:34.841 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-24T21:10:34.841 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:10:34.842 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:10:34.842 DEBUG:boto:path=/qskrps-13/?versions&max-keys=999 2023-05-24T21:10:34.842 DEBUG:boto:auth_path=/qskrps-13/?versions&max-keys=999 2023-05-24T21:10:34.843 DEBUG:boto:Method: GET 2023-05-24T21:10:34.843 DEBUG:boto:Path: /qskrps-13/?versions&max-keys=999 2023-05-24T21:10:34.843 DEBUG:boto:Data: 2023-05-24T21:10:34.844 DEBUG:boto:Headers: {} 2023-05-24T21:10:34.844 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:10:34.845 DEBUG:boto:Port: 8001 2023-05-24T21:10:34.845 DEBUG:boto:Params: {} 2023-05-24T21:10:34.845 DEBUG:boto:Token: None 2023-05-24T21:10:34.846 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:10:34 GMT /qskrps-13/?versions 2023-05-24T21:10:34.846 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:mQgwTs2bh7cb56AH/f3WZ/M9t0I= 2023-05-24T21:10:34.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:10:34 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:mQgwTs2bh7cb56AH/f3WZ/M9t0I=', 'Content-Length': '0'} 2023-05-24T21:10:34.849 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009cef87c3efb938d-00646e7d4a-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:10:34 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:34.850 DEBUG:boto:b'qskrps-13999falsedummyg04CLEvxE5VlkW2A9PqVzMmjG-oLrNXtrue2023-05-24T21:09:59.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:10:34.851 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-05-24T21:10:34.851 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-24T21:10:34.851 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:10:34.852 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:10:34.852 DEBUG:boto:path=/qskrps-13/?versions&max-keys=999 2023-05-24T21:10:34.852 DEBUG:boto:auth_path=/qskrps-13/?versions&max-keys=999 2023-05-24T21:10:34.853 DEBUG:boto:Method: GET 2023-05-24T21:10:34.853 DEBUG:boto:Path: /qskrps-13/?versions&max-keys=999 2023-05-24T21:10:34.854 DEBUG:boto:Data: 2023-05-24T21:10:34.854 DEBUG:boto:Headers: {} 2023-05-24T21:10:34.854 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:10:34.855 DEBUG:boto:Port: 8000 2023-05-24T21:10:34.855 DEBUG:boto:Params: {} 2023-05-24T21:10:34.855 DEBUG:boto:Token: None 2023-05-24T21:10:34.856 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:10:34 GMT /qskrps-13/?versions 2023-05-24T21:10:34.856 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:mQgwTs2bh7cb56AH/f3WZ/M9t0I= 2023-05-24T21:10:34.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:10:34 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:mQgwTs2bh7cb56AH/f3WZ/M9t0I=', 'Content-Length': '0'} 2023-05-24T21:10:34.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000371d29a3dabf3b2-00646e7d4a-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:10:34 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:34.861 DEBUG:boto:b'qskrps-13999falsedummyg04CLEvxE5VlkW2A9PqVzMmjG-oLrNXtrue2023-05-24T21:09:59.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:10:34.861 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-05-24T21:10:34.862 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:10:34.862 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:10:34.862 DEBUG:boto:path=/qskrps-13/?versions&max-keys=999 2023-05-24T21:10:34.863 DEBUG:boto:auth_path=/qskrps-13/?versions&max-keys=999 2023-05-24T21:10:34.863 DEBUG:boto:Method: GET 2023-05-24T21:10:34.863 DEBUG:boto:Path: /qskrps-13/?versions&max-keys=999 2023-05-24T21:10:34.864 DEBUG:boto:Data: 2023-05-24T21:10:34.864 DEBUG:boto:Headers: {} 2023-05-24T21:10:34.865 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:10:34.865 DEBUG:boto:Port: 8001 2023-05-24T21:10:34.865 DEBUG:boto:Params: {} 2023-05-24T21:10:34.866 DEBUG:boto:Token: None 2023-05-24T21:10:34.866 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:10:34 GMT /qskrps-13/?versions 2023-05-24T21:10:34.867 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:mQgwTs2bh7cb56AH/f3WZ/M9t0I= 2023-05-24T21:10:34.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:10:34 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:mQgwTs2bh7cb56AH/f3WZ/M9t0I=', 'Content-Length': '0'} 2023-05-24T21:10:34.869 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6b11d97f15b251d-00646e7d4a-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:10:34 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:34.870 DEBUG:boto:b'qskrps-13999falsedummyg04CLEvxE5VlkW2A9PqVzMmjG-oLrNXtrue2023-05-24T21:09:59.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:10:34.871 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:10:34.871 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:10:34.872 DEBUG:boto:path=/qskrps-13/?versions&max-keys=999 2023-05-24T21:10:34.872 DEBUG:boto:auth_path=/qskrps-13/?versions&max-keys=999 2023-05-24T21:10:34.873 DEBUG:boto:Method: GET 2023-05-24T21:10:34.874 DEBUG:boto:Path: /qskrps-13/?versions&max-keys=999 2023-05-24T21:10:34.874 DEBUG:boto:Data: 2023-05-24T21:10:34.875 DEBUG:boto:Headers: {} 2023-05-24T21:10:34.875 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:10:34.876 DEBUG:boto:Port: 8000 2023-05-24T21:10:34.876 DEBUG:boto:Params: {} 2023-05-24T21:10:34.877 DEBUG:boto:Token: None 2023-05-24T21:10:34.878 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:10:34 GMT /qskrps-13/?versions 2023-05-24T21:10:34.878 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:mQgwTs2bh7cb56AH/f3WZ/M9t0I= 2023-05-24T21:10:34.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:10:34 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:mQgwTs2bh7cb56AH/f3WZ/M9t0I=', 'Content-Length': '0'} 2023-05-24T21:10:34.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046b27096bf32de1d-00646e7d4a-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:10:34 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:34.882 DEBUG:boto:b'qskrps-13999falsedummyg04CLEvxE5VlkW2A9PqVzMmjG-oLrNXtrue2023-05-24T21:09:59.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:10:34.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-05-24T21:10:34.884 DEBUG:boto:path=/qskrps-13/dummy 2023-05-24T21:10:34.884 DEBUG:boto:auth_path=/qskrps-13/dummy 2023-05-24T21:10:34.885 DEBUG:boto:path=/qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:10:34.885 DEBUG:boto:auth_path=/qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:10:34.886 DEBUG:boto:Method: GET 2023-05-24T21:10:34.887 DEBUG:boto:Path: /qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:10:34.888 DEBUG:boto:Data: 2023-05-24T21:10:34.889 DEBUG:boto:Headers: {} 2023-05-24T21:10:34.889 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:10:34.890 DEBUG:boto:Port: 8001 2023-05-24T21:10:34.891 DEBUG:boto:Params: {} 2023-05-24T21:10:34.891 DEBUG:boto:Token: None 2023-05-24T21:10:34.892 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:10:34 GMT /qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:10:34.892 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4R1JUWCmWZ4UlrYIuePv0EZ3hFg= 2023-05-24T21:10:34.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:10:34 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4R1JUWCmWZ4UlrYIuePv0EZ3hFg=', 'Content-Length': '0'} 2023-05-24T21:10:34.894 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:59 GMT'), ('x-amz-version-id', 'g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000517065b08e8810a6-00646e7d4a-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:10:34 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:34.895 DEBUG:boto:path=/qskrps-13/dummy 2023-05-24T21:10:34.895 DEBUG:boto:auth_path=/qskrps-13/dummy 2023-05-24T21:10:34.896 DEBUG:boto:path=/qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:10:34.896 DEBUG:boto:auth_path=/qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:10:34.897 DEBUG:boto:Method: GET 2023-05-24T21:10:34.897 DEBUG:boto:Path: /qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:10:34.898 DEBUG:boto:Data: 2023-05-24T21:10:34.898 DEBUG:boto:Headers: {} 2023-05-24T21:10:34.898 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:10:34.899 DEBUG:boto:Port: 8000 2023-05-24T21:10:34.899 DEBUG:boto:Params: {} 2023-05-24T21:10:34.900 DEBUG:boto:Token: None 2023-05-24T21:10:34.900 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:10:34 GMT /qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:10:34.901 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4R1JUWCmWZ4UlrYIuePv0EZ3hFg= 2023-05-24T21:10:34.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:10:34 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4R1JUWCmWZ4UlrYIuePv0EZ3hFg=', 'Content-Length': '0'} 2023-05-24T21:10:34.904 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:59 GMT'), ('x-amz-version-id', 'g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000009f2e8bb2d4f42e8-00646e7d4a-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:10:34 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:34.905 DEBUG:boto:path=/qskrps-13/dummy 2023-05-24T21:10:34.906 DEBUG:boto:auth_path=/qskrps-13/dummy 2023-05-24T21:10:34.906 DEBUG:boto:path=/qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:10:34.907 DEBUG:boto:auth_path=/qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:10:34.907 DEBUG:boto:Method: HEAD 2023-05-24T21:10:34.907 DEBUG:boto:Path: /qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:10:34.908 DEBUG:boto:Data: 2023-05-24T21:10:34.908 DEBUG:boto:Headers: {} 2023-05-24T21:10:34.908 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:10:34.909 DEBUG:boto:Port: 8001 2023-05-24T21:10:34.909 DEBUG:boto:Params: {} 2023-05-24T21:10:34.909 DEBUG:boto:Token: None 2023-05-24T21:10:34.910 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:10:34 GMT /qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:10:34.910 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:WumPT+FJRoLTETblI4E8uni1fb0= 2023-05-24T21:10:34.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:10:34 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WumPT+FJRoLTETblI4E8uni1fb0=', 'Content-Length': '0'} 2023-05-24T21:10:34.912 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:59 GMT'), ('x-amz-version-id', 'g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000316f87545c7e11ee-00646e7d4a-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:10:34 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:34.913 DEBUG:boto:path=/qskrps-13/dummy 2023-05-24T21:10:34.913 DEBUG:boto:auth_path=/qskrps-13/dummy 2023-05-24T21:10:34.914 DEBUG:boto:path=/qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:10:34.914 DEBUG:boto:auth_path=/qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:10:34.914 DEBUG:boto:Method: HEAD 2023-05-24T21:10:34.915 DEBUG:boto:Path: /qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:10:34.915 DEBUG:boto:Data: 2023-05-24T21:10:34.915 DEBUG:boto:Headers: {} 2023-05-24T21:10:34.916 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:10:34.916 DEBUG:boto:Port: 8000 2023-05-24T21:10:34.916 DEBUG:boto:Params: {} 2023-05-24T21:10:34.917 DEBUG:boto:Token: None 2023-05-24T21:10:34.917 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:10:34 GMT /qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:10:34.918 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:WumPT+FJRoLTETblI4E8uni1fb0= 2023-05-24T21:10:34.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:10:34 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WumPT+FJRoLTETblI4E8uni1fb0=', 'Content-Length': '0'} 2023-05-24T21:10:34.920 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:59 GMT'), ('x-amz-version-id', 'g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000096b1f4141f592fe5-00646e7d4a-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:10:34 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:34.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-05-24T21:10:34.921 DEBUG:boto:path=/qskrps-13/dummy 2023-05-24T21:10:34.921 DEBUG:boto:auth_path=/qskrps-13/dummy 2023-05-24T21:10:34.922 DEBUG:boto:path=/qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:10:34.922 DEBUG:boto:auth_path=/qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:10:34.922 DEBUG:boto:Method: GET 2023-05-24T21:10:34.923 DEBUG:boto:Path: /qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:10:34.923 DEBUG:boto:Data: 2023-05-24T21:10:34.924 DEBUG:boto:Headers: {} 2023-05-24T21:10:34.924 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:10:34.924 DEBUG:boto:Port: 8001 2023-05-24T21:10:34.925 DEBUG:boto:Params: {} 2023-05-24T21:10:34.925 DEBUG:boto:Token: None 2023-05-24T21:10:34.926 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:10:34 GMT /qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:10:34.926 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4R1JUWCmWZ4UlrYIuePv0EZ3hFg= 2023-05-24T21:10:34.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:10:34 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4R1JUWCmWZ4UlrYIuePv0EZ3hFg=', 'Content-Length': '0'} 2023-05-24T21:10:34.928 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:59 GMT'), ('x-amz-version-id', 'g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007aacd73168164ee2-00646e7d4a-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:10:34 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:34.929 DEBUG:boto:path=/qskrps-13/dummy 2023-05-24T21:10:34.929 DEBUG:boto:auth_path=/qskrps-13/dummy 2023-05-24T21:10:34.929 DEBUG:boto:path=/qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:10:34.930 DEBUG:boto:auth_path=/qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:10:34.930 DEBUG:boto:Method: GET 2023-05-24T21:10:34.931 DEBUG:boto:Path: /qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:10:34.931 DEBUG:boto:Data: 2023-05-24T21:10:34.932 DEBUG:boto:Headers: {} 2023-05-24T21:10:34.932 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:10:34.933 DEBUG:boto:Port: 8000 2023-05-24T21:10:34.933 DEBUG:boto:Params: {} 2023-05-24T21:10:34.933 DEBUG:boto:Token: None 2023-05-24T21:10:34.934 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:10:34 GMT /qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:10:34.934 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4R1JUWCmWZ4UlrYIuePv0EZ3hFg= 2023-05-24T21:10:34.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:10:34 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4R1JUWCmWZ4UlrYIuePv0EZ3hFg=', 'Content-Length': '0'} 2023-05-24T21:10:34.936 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:59 GMT'), ('x-amz-version-id', 'g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000098be24ab73bcc44a-00646e7d4a-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:10:34 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:34.937 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:10:34.938 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:10:34.939 DEBUG:boto:path=/qskrps-13/?prefix=dummy 2023-05-24T21:10:34.939 DEBUG:boto:auth_path=/qskrps-13/?prefix=dummy 2023-05-24T21:10:34.940 DEBUG:boto:Method: GET 2023-05-24T21:10:34.941 DEBUG:boto:Path: /qskrps-13/?prefix=dummy 2023-05-24T21:10:34.942 DEBUG:boto:Data: 2023-05-24T21:10:34.943 DEBUG:boto:Headers: {} 2023-05-24T21:10:34.944 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:10:34.945 DEBUG:boto:Port: 8001 2023-05-24T21:10:34.945 DEBUG:boto:Params: {} 2023-05-24T21:10:34.946 DEBUG:boto:Token: None 2023-05-24T21:10:34.947 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:10:34 GMT /qskrps-13/ 2023-05-24T21:10:34.948 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:hdwJUcaZorgnPpHKTDJGNpoHV5g= 2023-05-24T21:10:34.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:10:34 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:hdwJUcaZorgnPpHKTDJGNpoHV5g=', 'Content-Length': '0'} 2023-05-24T21:10:34.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000febd9920a82dfdf9-00646e7d4a-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:10:34 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:34.992 DEBUG:boto:b'qskrps-13dummy1000falsedummy2023-05-24T21:09:59.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:10:34.993 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:10:34.994 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:10:34.995 DEBUG:boto:path=/qskrps-13/?prefix=dummy 2023-05-24T21:10:34.995 DEBUG:boto:auth_path=/qskrps-13/?prefix=dummy 2023-05-24T21:10:34.995 DEBUG:boto:Method: GET 2023-05-24T21:10:34.996 DEBUG:boto:Path: /qskrps-13/?prefix=dummy 2023-05-24T21:10:34.996 DEBUG:boto:Data: 2023-05-24T21:10:34.997 DEBUG:boto:Headers: {} 2023-05-24T21:10:34.997 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:10:34.997 DEBUG:boto:Port: 8000 2023-05-24T21:10:34.998 DEBUG:boto:Params: {} 2023-05-24T21:10:34.998 DEBUG:boto:Token: None 2023-05-24T21:10:34.999 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:10:34 GMT /qskrps-13/ 2023-05-24T21:10:34.999 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:hdwJUcaZorgnPpHKTDJGNpoHV5g= 2023-05-24T21:10:34.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:10:34 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:hdwJUcaZorgnPpHKTDJGNpoHV5g=', 'Content-Length': '0'} 2023-05-24T21:10:35.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aeb87310e435b652-00646e7d4a-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:10:34 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.003 DEBUG:boto:b'qskrps-13dummy1000falsedummy2023-05-24T21:09:59.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:10:35.003 DEBUG:boto:path=/qskrps-13/dummy 2023-05-24T21:10:35.004 DEBUG:boto:auth_path=/qskrps-13/dummy 2023-05-24T21:10:35.005 DEBUG:boto:Method: HEAD 2023-05-24T21:10:35.005 DEBUG:boto:Path: /qskrps-13/dummy 2023-05-24T21:10:35.005 DEBUG:boto:Data: 2023-05-24T21:10:35.006 DEBUG:boto:Headers: {} 2023-05-24T21:10:35.006 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:10:35.006 DEBUG:boto:Port: 8001 2023-05-24T21:10:35.007 DEBUG:boto:Params: {} 2023-05-24T21:10:35.007 DEBUG:boto:Token: None 2023-05-24T21:10:35.007 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:10:35 GMT /qskrps-13/dummy 2023-05-24T21:10:35.008 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:kVyY+rLe947MNCqkBhICwaHF82A= 2023-05-24T21:10:35.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:10:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:kVyY+rLe947MNCqkBhICwaHF82A=', 'Content-Length': '0'} 2023-05-24T21:10:35.010 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:59 GMT'), ('x-amz-version-id', 'g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003d00d4310ed36cae-00646e7d4b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:10:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.011 DEBUG:boto:path=/qskrps-13/dummy 2023-05-24T21:10:35.011 DEBUG:boto:auth_path=/qskrps-13/dummy 2023-05-24T21:10:35.012 DEBUG:boto:Method: HEAD 2023-05-24T21:10:35.012 DEBUG:boto:Path: /qskrps-13/dummy 2023-05-24T21:10:35.012 DEBUG:boto:Data: 2023-05-24T21:10:35.013 DEBUG:boto:Headers: {} 2023-05-24T21:10:35.013 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:10:35.013 DEBUG:boto:Port: 8000 2023-05-24T21:10:35.014 DEBUG:boto:Params: {} 2023-05-24T21:10:35.014 DEBUG:boto:Token: None 2023-05-24T21:10:35.014 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:10:35 GMT /qskrps-13/dummy 2023-05-24T21:10:35.015 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:kVyY+rLe947MNCqkBhICwaHF82A= 2023-05-24T21:10:35.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:10:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:kVyY+rLe947MNCqkBhICwaHF82A=', 'Content-Length': '0'} 2023-05-24T21:10:35.017 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:59 GMT'), ('x-amz-version-id', 'g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003c90767fee66826b-00646e7d4b-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:10:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-05-24T21:10:35.019 DEBUG:boto:path=/qskrps-13/dummy 2023-05-24T21:10:35.020 DEBUG:boto:auth_path=/qskrps-13/dummy 2023-05-24T21:10:35.020 DEBUG:boto:path=/qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:10:35.021 DEBUG:boto:auth_path=/qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:10:35.022 DEBUG:boto:Method: GET 2023-05-24T21:10:35.023 DEBUG:boto:Path: /qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:10:35.023 DEBUG:boto:Data: 2023-05-24T21:10:35.024 DEBUG:boto:Headers: {} 2023-05-24T21:10:35.024 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:10:35.025 DEBUG:boto:Port: 8001 2023-05-24T21:10:35.025 DEBUG:boto:Params: {} 2023-05-24T21:10:35.025 DEBUG:boto:Token: None 2023-05-24T21:10:35.026 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:10:35 GMT /qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:10:35.026 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:OFfyi/J5Bu54wIpOrJEopTE+qN4= 2023-05-24T21:10:35.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:10:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:OFfyi/J5Bu54wIpOrJEopTE+qN4=', 'Content-Length': '0'} 2023-05-24T21:10:35.029 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:59 GMT'), ('x-amz-version-id', 'g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006055e23a4b98bdd2-00646e7d4b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:10:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.029 DEBUG:boto:path=/qskrps-13/dummy 2023-05-24T21:10:35.030 DEBUG:boto:auth_path=/qskrps-13/dummy 2023-05-24T21:10:35.030 DEBUG:boto:path=/qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:10:35.031 DEBUG:boto:auth_path=/qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:10:35.031 DEBUG:boto:Method: GET 2023-05-24T21:10:35.031 DEBUG:boto:Path: /qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:10:35.032 DEBUG:boto:Data: 2023-05-24T21:10:35.032 DEBUG:boto:Headers: {} 2023-05-24T21:10:35.032 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:10:35.033 DEBUG:boto:Port: 8000 2023-05-24T21:10:35.033 DEBUG:boto:Params: {} 2023-05-24T21:10:35.034 DEBUG:boto:Token: None 2023-05-24T21:10:35.034 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:10:35 GMT /qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:10:35.034 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:OFfyi/J5Bu54wIpOrJEopTE+qN4= 2023-05-24T21:10:35.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:10:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:OFfyi/J5Bu54wIpOrJEopTE+qN4=', 'Content-Length': '0'} 2023-05-24T21:10:35.037 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:59 GMT'), ('x-amz-version-id', 'g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000014f4ee7964d572c4-00646e7d4b-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:10:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.038 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:10:35.038 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:10:35.038 DEBUG:boto:path=/qskrps-13/?prefix=dummy 2023-05-24T21:10:35.039 DEBUG:boto:auth_path=/qskrps-13/?prefix=dummy 2023-05-24T21:10:35.039 DEBUG:boto:Method: GET 2023-05-24T21:10:35.039 DEBUG:boto:Path: /qskrps-13/?prefix=dummy 2023-05-24T21:10:35.040 DEBUG:boto:Data: 2023-05-24T21:10:35.040 DEBUG:boto:Headers: {} 2023-05-24T21:10:35.041 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:10:35.041 DEBUG:boto:Port: 8001 2023-05-24T21:10:35.041 DEBUG:boto:Params: {} 2023-05-24T21:10:35.042 DEBUG:boto:Token: None 2023-05-24T21:10:35.042 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:10:35 GMT /qskrps-13/ 2023-05-24T21:10:35.043 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:R05UAx71NG6BiCHkYKlElPIqBsk= 2023-05-24T21:10:35.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:10:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:R05UAx71NG6BiCHkYKlElPIqBsk=', 'Content-Length': '0'} 2023-05-24T21:10:35.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d8a0f58cfc4b354-00646e7d4b-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:10:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.046 DEBUG:boto:b'qskrps-13dummy1000falsedummy2023-05-24T21:09:59.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:10:35.046 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:10:35.047 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:10:35.047 DEBUG:boto:path=/qskrps-13/?prefix=dummy 2023-05-24T21:10:35.048 DEBUG:boto:auth_path=/qskrps-13/?prefix=dummy 2023-05-24T21:10:35.048 DEBUG:boto:Method: GET 2023-05-24T21:10:35.048 DEBUG:boto:Path: /qskrps-13/?prefix=dummy 2023-05-24T21:10:35.049 DEBUG:boto:Data: 2023-05-24T21:10:35.049 DEBUG:boto:Headers: {} 2023-05-24T21:10:35.050 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:10:35.050 DEBUG:boto:Port: 8000 2023-05-24T21:10:35.051 DEBUG:boto:Params: {} 2023-05-24T21:10:35.051 DEBUG:boto:Token: None 2023-05-24T21:10:35.052 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:10:35 GMT /qskrps-13/ 2023-05-24T21:10:35.052 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:R05UAx71NG6BiCHkYKlElPIqBsk= 2023-05-24T21:10:35.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:10:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:R05UAx71NG6BiCHkYKlElPIqBsk=', 'Content-Length': '0'} 2023-05-24T21:10:35.055 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009854e5bc7a1c45c8-00646e7d4b-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:10:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.096 DEBUG:boto:b'qskrps-13dummy1000falsedummy2023-05-24T21:09:59.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:10:35.097 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qskrps-13 zones={a2, a1} 2023-05-24T21:10:35.098 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:10:35.098 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:10:35.098 DEBUG:boto:Method: HEAD 2023-05-24T21:10:35.099 DEBUG:boto:Path: /qskrps-13/ 2023-05-24T21:10:35.099 DEBUG:boto:Data: 2023-05-24T21:10:35.099 DEBUG:boto:Headers: {} 2023-05-24T21:10:35.100 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:10:35.100 DEBUG:boto:Port: 8000 2023-05-24T21:10:35.101 DEBUG:boto:Params: {} 2023-05-24T21:10:35.101 DEBUG:boto:Token: None 2023-05-24T21:10:35.101 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:10:35 GMT /qskrps-13/ 2023-05-24T21:10:35.102 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Zm3PPjG10IqP0JC2UXVF+3wgkh4= 2023-05-24T21:10:35.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:10:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Zm3PPjG10IqP0JC2UXVF+3wgkh4=', 'Content-Length': '0'} 2023-05-24T21:10:35.105 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ad3692f5fefd1bf8-00646e7d4b-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:10:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.105 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:10:35.106 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:10:35.106 DEBUG:boto:Method: PUT 2023-05-24T21:10:35.106 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:10:35.107 DEBUG:boto:Data: 2023-05-24T21:10:35.107 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-24T21:10:35.107 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:10:35.108 DEBUG:boto:Port: 8000 2023-05-24T21:10:35.108 DEBUG:boto:Params: {} 2023-05-24T21:10:35.108 DEBUG:boto:Token: None 2023-05-24T21:10:35.109 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 24 May 2023 21:10:35 GMT /qskrps-13/obj 2023-05-24T21:10:35.109 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:FcupaljGb7eQLYzF9J0UIQRU+S0= 2023-05-24T21:10:35.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:10:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:FcupaljGb7eQLYzF9J0UIQRU+S0='} 2023-05-24T21:10:35.122 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'agpDexVocguP.SXhiu6FyO5RbOuqAnw'), ('x-amz-request-id', 'tx00000d6f0fefaf4279284-00646e7d4b-110d-a1'), ('Date', 'Wed, 24 May 2023 21:10:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.123 DEBUG:rgw_multi.tests:zone=a1 version=agpDexVocguP.SXhiu6FyO5RbOuqAnw 2023-05-24T21:10:35.124 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:10:35.125 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:10:35.126 DEBUG:boto:Method: HEAD 2023-05-24T21:10:35.126 DEBUG:boto:Path: /qskrps-13/ 2023-05-24T21:10:35.127 DEBUG:boto:Data: 2023-05-24T21:10:35.127 DEBUG:boto:Headers: {} 2023-05-24T21:10:35.127 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:10:35.128 DEBUG:boto:Port: 8001 2023-05-24T21:10:35.128 DEBUG:boto:Params: {} 2023-05-24T21:10:35.129 DEBUG:boto:Token: None 2023-05-24T21:10:35.129 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:10:35 GMT /qskrps-13/ 2023-05-24T21:10:35.130 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Zm3PPjG10IqP0JC2UXVF+3wgkh4= 2023-05-24T21:10:35.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:10:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Zm3PPjG10IqP0JC2UXVF+3wgkh4=', 'Content-Length': '0'} 2023-05-24T21:10:35.133 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e6bde4285ea82a60-00646e7d4b-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:10:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.133 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:10:35.134 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:10:35.134 DEBUG:boto:Method: PUT 2023-05-24T21:10:35.134 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:10:35.135 DEBUG:boto:Data: 2023-05-24T21:10:35.135 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-24T21:10:35.135 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:10:35.136 DEBUG:boto:Port: 8001 2023-05-24T21:10:35.136 DEBUG:boto:Params: {} 2023-05-24T21:10:35.136 DEBUG:boto:Token: None 2023-05-24T21:10:35.137 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 24 May 2023 21:10:35 GMT /qskrps-13/obj 2023-05-24T21:10:35.137 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:FcupaljGb7eQLYzF9J0UIQRU+S0= 2023-05-24T21:10:35.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:10:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:FcupaljGb7eQLYzF9J0UIQRU+S0='} 2023-05-24T21:10:35.180 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'BdBswNggliM4dGAQ3Dgxcgleuyd0Qxo'), ('x-amz-request-id', 'tx00000346a01ae65818100-00646e7d4b-1111-a2'), ('Date', 'Wed, 24 May 2023 21:10:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.181 DEBUG:rgw_multi.tests:zone=a2 version=BdBswNggliM4dGAQ3Dgxcgleuyd0Qxo 2023-05-24T21:10:35.181 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:10:35.182 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:10:35.182 DEBUG:boto:Method: HEAD 2023-05-24T21:10:35.182 DEBUG:boto:Path: /qskrps-13/ 2023-05-24T21:10:35.183 DEBUG:boto:Data: 2023-05-24T21:10:35.183 DEBUG:boto:Headers: {} 2023-05-24T21:10:35.183 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:10:35.184 DEBUG:boto:Port: 8000 2023-05-24T21:10:35.184 DEBUG:boto:Params: {} 2023-05-24T21:10:35.185 DEBUG:boto:Token: None 2023-05-24T21:10:35.185 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:10:35 GMT /qskrps-13/ 2023-05-24T21:10:35.185 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Zm3PPjG10IqP0JC2UXVF+3wgkh4= 2023-05-24T21:10:35.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:10:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Zm3PPjG10IqP0JC2UXVF+3wgkh4=', 'Content-Length': '0'} 2023-05-24T21:10:35.188 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c7ed63e444bfe977-00646e7d4b-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:10:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.189 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:10:35.190 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:10:35.190 DEBUG:boto:Method: PUT 2023-05-24T21:10:35.190 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:10:35.191 DEBUG:boto:Data: 2023-05-24T21:10:35.191 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-24T21:10:35.191 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:10:35.192 DEBUG:boto:Port: 8000 2023-05-24T21:10:35.192 DEBUG:boto:Params: {} 2023-05-24T21:10:35.192 DEBUG:boto:Token: None 2023-05-24T21:10:35.193 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 24 May 2023 21:10:35 GMT /qskrps-13/obj 2023-05-24T21:10:35.193 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:FcupaljGb7eQLYzF9J0UIQRU+S0= 2023-05-24T21:10:35.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:10:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:FcupaljGb7eQLYzF9J0UIQRU+S0='} 2023-05-24T21:10:35.237 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'vVL.Oc-NasmiTK3cSFNuRAIlXlfwbAC'), ('x-amz-request-id', 'tx00000b1b3308c8fa6b193-00646e7d4b-110d-a1'), ('Date', 'Wed, 24 May 2023 21:10:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.238 DEBUG:rgw_multi.tests:zone=a1 version=vVL.Oc-NasmiTK3cSFNuRAIlXlfwbAC 2023-05-24T21:10:35.238 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:10:35.239 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:10:35.239 DEBUG:boto:Method: HEAD 2023-05-24T21:10:35.240 DEBUG:boto:Path: /qskrps-13/ 2023-05-24T21:10:35.241 DEBUG:boto:Data: 2023-05-24T21:10:35.241 DEBUG:boto:Headers: {} 2023-05-24T21:10:35.242 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:10:35.243 DEBUG:boto:Port: 8001 2023-05-24T21:10:35.243 DEBUG:boto:Params: {} 2023-05-24T21:10:35.244 DEBUG:boto:Token: None 2023-05-24T21:10:35.245 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:10:35 GMT /qskrps-13/ 2023-05-24T21:10:35.245 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Zm3PPjG10IqP0JC2UXVF+3wgkh4= 2023-05-24T21:10:35.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:10:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Zm3PPjG10IqP0JC2UXVF+3wgkh4=', 'Content-Length': '0'} 2023-05-24T21:10:35.249 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000161c0ff040450559-00646e7d4b-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:10:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.250 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:10:35.251 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:10:35.251 DEBUG:boto:Method: PUT 2023-05-24T21:10:35.251 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:10:35.252 DEBUG:boto:Data: 2023-05-24T21:10:35.252 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-24T21:10:35.253 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:10:35.253 DEBUG:boto:Port: 8001 2023-05-24T21:10:35.253 DEBUG:boto:Params: {} 2023-05-24T21:10:35.254 DEBUG:boto:Token: None 2023-05-24T21:10:35.254 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 24 May 2023 21:10:35 GMT /qskrps-13/obj 2023-05-24T21:10:35.255 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:FcupaljGb7eQLYzF9J0UIQRU+S0= 2023-05-24T21:10:35.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:10:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:FcupaljGb7eQLYzF9J0UIQRU+S0='} 2023-05-24T21:10:35.272 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '3wYQ13EDow0kiwE6ITIMyTWEPt0A0Hx'), ('x-amz-request-id', 'tx00000fcf6cf136b52abbf-00646e7d4b-1111-a2'), ('Date', 'Wed, 24 May 2023 21:10:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.273 DEBUG:rgw_multi.tests:zone=a2 version=3wYQ13EDow0kiwE6ITIMyTWEPt0A0Hx 2023-05-24T21:10:35.273 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:10:35.273 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:10:35.274 DEBUG:boto:Method: HEAD 2023-05-24T21:10:35.274 DEBUG:boto:Path: /qskrps-13/ 2023-05-24T21:10:35.274 DEBUG:boto:Data: 2023-05-24T21:10:35.275 DEBUG:boto:Headers: {} 2023-05-24T21:10:35.275 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:10:35.275 DEBUG:boto:Port: 8000 2023-05-24T21:10:35.276 DEBUG:boto:Params: {} 2023-05-24T21:10:35.276 DEBUG:boto:Token: None 2023-05-24T21:10:35.277 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:10:35 GMT /qskrps-13/ 2023-05-24T21:10:35.277 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Zm3PPjG10IqP0JC2UXVF+3wgkh4= 2023-05-24T21:10:35.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:10:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Zm3PPjG10IqP0JC2UXVF+3wgkh4=', 'Content-Length': '0'} 2023-05-24T21:10:35.281 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000085c7083904bfe03d-00646e7d4b-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:10:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.282 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:10:35.283 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:10:35.284 DEBUG:boto:Method: PUT 2023-05-24T21:10:35.284 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:10:35.284 DEBUG:boto:Data: 2023-05-24T21:10:35.285 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-24T21:10:35.285 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:10:35.285 DEBUG:boto:Port: 8000 2023-05-24T21:10:35.286 DEBUG:boto:Params: {} 2023-05-24T21:10:35.286 DEBUG:boto:Token: None 2023-05-24T21:10:35.287 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 24 May 2023 21:10:35 GMT /qskrps-13/obj 2023-05-24T21:10:35.287 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:FcupaljGb7eQLYzF9J0UIQRU+S0= 2023-05-24T21:10:35.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:10:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:FcupaljGb7eQLYzF9J0UIQRU+S0='} 2023-05-24T21:10:35.302 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'XBH9.ADrbaEY1JrgZ6O0QvO0TBVsG2v'), ('x-amz-request-id', 'tx00000d9c80d3d529dbc27-00646e7d4b-110d-a1'), ('Date', 'Wed, 24 May 2023 21:10:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.303 DEBUG:rgw_multi.tests:zone=a1 version=XBH9.ADrbaEY1JrgZ6O0QvO0TBVsG2v 2023-05-24T21:10:35.303 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:10:35.304 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:10:35.304 DEBUG:boto:Method: HEAD 2023-05-24T21:10:35.305 DEBUG:boto:Path: /qskrps-13/ 2023-05-24T21:10:35.306 DEBUG:boto:Data: 2023-05-24T21:10:35.306 DEBUG:boto:Headers: {} 2023-05-24T21:10:35.307 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:10:35.307 DEBUG:boto:Port: 8001 2023-05-24T21:10:35.308 DEBUG:boto:Params: {} 2023-05-24T21:10:35.308 DEBUG:boto:Token: None 2023-05-24T21:10:35.309 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:10:35 GMT /qskrps-13/ 2023-05-24T21:10:35.309 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Zm3PPjG10IqP0JC2UXVF+3wgkh4= 2023-05-24T21:10:35.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:10:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Zm3PPjG10IqP0JC2UXVF+3wgkh4=', 'Content-Length': '0'} 2023-05-24T21:10:35.312 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001523ed76fee6e0bb-00646e7d4b-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:10:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.313 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:10:35.314 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:10:35.315 DEBUG:boto:Method: PUT 2023-05-24T21:10:35.316 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:10:35.316 DEBUG:boto:Data: 2023-05-24T21:10:35.316 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-24T21:10:35.317 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:10:35.317 DEBUG:boto:Port: 8001 2023-05-24T21:10:35.317 DEBUG:boto:Params: {} 2023-05-24T21:10:35.318 DEBUG:boto:Token: None 2023-05-24T21:10:35.318 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 24 May 2023 21:10:35 GMT /qskrps-13/obj 2023-05-24T21:10:35.319 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:FcupaljGb7eQLYzF9J0UIQRU+S0= 2023-05-24T21:10:35.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:10:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:FcupaljGb7eQLYzF9J0UIQRU+S0='} 2023-05-24T21:10:35.332 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '1BSwhaY-MjwOOLy73m-3KiPjCrRBsod'), ('x-amz-request-id', 'tx00000db39b857e807c2cb-00646e7d4b-1111-a2'), ('Date', 'Wed, 24 May 2023 21:10:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.333 DEBUG:rgw_multi.tests:zone=a2 version=1BSwhaY-MjwOOLy73m-3KiPjCrRBsod 2023-05-24T21:10:35.334 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:10:35.335 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:10:35.335 DEBUG:boto:Method: HEAD 2023-05-24T21:10:35.336 DEBUG:boto:Path: /qskrps-13/ 2023-05-24T21:10:35.336 DEBUG:boto:Data: 2023-05-24T21:10:35.337 DEBUG:boto:Headers: {} 2023-05-24T21:10:35.337 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:10:35.337 DEBUG:boto:Port: 8000 2023-05-24T21:10:35.338 DEBUG:boto:Params: {} 2023-05-24T21:10:35.338 DEBUG:boto:Token: None 2023-05-24T21:10:35.339 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:10:35 GMT /qskrps-13/ 2023-05-24T21:10:35.339 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Zm3PPjG10IqP0JC2UXVF+3wgkh4= 2023-05-24T21:10:35.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:10:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Zm3PPjG10IqP0JC2UXVF+3wgkh4=', 'Content-Length': '0'} 2023-05-24T21:10:35.342 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000599c7bc1fa916e97-00646e7d4b-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:10:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.342 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:10:35.343 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:10:35.343 DEBUG:boto:Method: PUT 2023-05-24T21:10:35.343 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:10:35.344 DEBUG:boto:Data: 2023-05-24T21:10:35.344 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-24T21:10:35.344 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:10:35.345 DEBUG:boto:Port: 8000 2023-05-24T21:10:35.345 DEBUG:boto:Params: {} 2023-05-24T21:10:35.345 DEBUG:boto:Token: None 2023-05-24T21:10:35.346 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 24 May 2023 21:10:35 GMT /qskrps-13/obj 2023-05-24T21:10:35.346 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:FcupaljGb7eQLYzF9J0UIQRU+S0= 2023-05-24T21:10:35.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:10:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:FcupaljGb7eQLYzF9J0UIQRU+S0='} 2023-05-24T21:10:35.366 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'DkBjLdO6nXireb7JdIhAsx00H76MZGD'), ('x-amz-request-id', 'tx00000b137a75134b79361-00646e7d4b-110d-a1'), ('Date', 'Wed, 24 May 2023 21:10:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.366 DEBUG:rgw_multi.tests:zone=a1 version=DkBjLdO6nXireb7JdIhAsx00H76MZGD 2023-05-24T21:10:35.367 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:10:35.367 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:10:35.367 DEBUG:boto:Method: HEAD 2023-05-24T21:10:35.368 DEBUG:boto:Path: /qskrps-13/ 2023-05-24T21:10:35.368 DEBUG:boto:Data: 2023-05-24T21:10:35.368 DEBUG:boto:Headers: {} 2023-05-24T21:10:35.369 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:10:35.369 DEBUG:boto:Port: 8001 2023-05-24T21:10:35.369 DEBUG:boto:Params: {} 2023-05-24T21:10:35.370 DEBUG:boto:Token: None 2023-05-24T21:10:35.370 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:10:35 GMT /qskrps-13/ 2023-05-24T21:10:35.371 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Zm3PPjG10IqP0JC2UXVF+3wgkh4= 2023-05-24T21:10:35.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:10:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Zm3PPjG10IqP0JC2UXVF+3wgkh4=', 'Content-Length': '0'} 2023-05-24T21:10:35.373 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003c8f73dd69c297f1-00646e7d4b-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:10:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.374 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:10:35.374 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:10:35.375 DEBUG:boto:Method: PUT 2023-05-24T21:10:35.375 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:10:35.375 DEBUG:boto:Data: 2023-05-24T21:10:35.376 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-24T21:10:35.376 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:10:35.376 DEBUG:boto:Port: 8001 2023-05-24T21:10:35.377 DEBUG:boto:Params: {} 2023-05-24T21:10:35.377 DEBUG:boto:Token: None 2023-05-24T21:10:35.377 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 24 May 2023 21:10:35 GMT /qskrps-13/obj 2023-05-24T21:10:35.378 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:FcupaljGb7eQLYzF9J0UIQRU+S0= 2023-05-24T21:10:35.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:10:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:FcupaljGb7eQLYzF9J0UIQRU+S0='} 2023-05-24T21:10:35.387 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'U4zUwdtKVyScPebZNZT8NHXc8rQ.8f-'), ('x-amz-request-id', 'tx00000bbec98ae64883be4-00646e7d4b-1111-a2'), ('Date', 'Wed, 24 May 2023 21:10:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.388 DEBUG:rgw_multi.tests:zone=a2 version=U4zUwdtKVyScPebZNZT8NHXc8rQ.8f- 2023-05-24T21:10:35.388 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:10:35.389 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:10:35.389 DEBUG:boto:Method: HEAD 2023-05-24T21:10:35.389 DEBUG:boto:Path: /qskrps-13/ 2023-05-24T21:10:35.390 DEBUG:boto:Data: 2023-05-24T21:10:35.390 DEBUG:boto:Headers: {} 2023-05-24T21:10:35.390 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:10:35.391 DEBUG:boto:Port: 8000 2023-05-24T21:10:35.391 DEBUG:boto:Params: {} 2023-05-24T21:10:35.391 DEBUG:boto:Token: None 2023-05-24T21:10:35.392 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:10:35 GMT /qskrps-13/ 2023-05-24T21:10:35.392 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Zm3PPjG10IqP0JC2UXVF+3wgkh4= 2023-05-24T21:10:35.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:10:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Zm3PPjG10IqP0JC2UXVF+3wgkh4=', 'Content-Length': '0'} 2023-05-24T21:10:35.395 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000153de473cfcbbcf8-00646e7d4b-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:10:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.395 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:10:35.396 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:10:35.396 DEBUG:boto:Method: PUT 2023-05-24T21:10:35.397 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:10:35.397 DEBUG:boto:Data: 2023-05-24T21:10:35.397 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-24T21:10:35.398 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:10:35.398 DEBUG:boto:Port: 8000 2023-05-24T21:10:35.398 DEBUG:boto:Params: {} 2023-05-24T21:10:35.399 DEBUG:boto:Token: None 2023-05-24T21:10:35.399 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 24 May 2023 21:10:35 GMT /qskrps-13/obj 2023-05-24T21:10:35.399 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:FcupaljGb7eQLYzF9J0UIQRU+S0= 2023-05-24T21:10:35.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:10:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:FcupaljGb7eQLYzF9J0UIQRU+S0='} 2023-05-24T21:10:35.410 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'c.ug7Ld36X8.TaQXWIjev99m-BKd5qJ'), ('x-amz-request-id', 'tx000006c05c3212c860195-00646e7d4b-110d-a1'), ('Date', 'Wed, 24 May 2023 21:10:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.410 DEBUG:rgw_multi.tests:zone=a1 version=c.ug7Ld36X8.TaQXWIjev99m-BKd5qJ 2023-05-24T21:10:35.410 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:10:35.411 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:10:35.411 DEBUG:boto:Method: HEAD 2023-05-24T21:10:35.411 DEBUG:boto:Path: /qskrps-13/ 2023-05-24T21:10:35.412 DEBUG:boto:Data: 2023-05-24T21:10:35.412 DEBUG:boto:Headers: {} 2023-05-24T21:10:35.412 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:10:35.413 DEBUG:boto:Port: 8001 2023-05-24T21:10:35.413 DEBUG:boto:Params: {} 2023-05-24T21:10:35.413 DEBUG:boto:Token: None 2023-05-24T21:10:35.414 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:10:35 GMT /qskrps-13/ 2023-05-24T21:10:35.414 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Zm3PPjG10IqP0JC2UXVF+3wgkh4= 2023-05-24T21:10:35.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:10:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Zm3PPjG10IqP0JC2UXVF+3wgkh4=', 'Content-Length': '0'} 2023-05-24T21:10:35.417 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004737c639208fc71f-00646e7d4b-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:10:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.417 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:10:35.418 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:10:35.418 DEBUG:boto:Method: PUT 2023-05-24T21:10:35.419 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:10:35.419 DEBUG:boto:Data: 2023-05-24T21:10:35.419 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-24T21:10:35.420 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:10:35.420 DEBUG:boto:Port: 8001 2023-05-24T21:10:35.421 DEBUG:boto:Params: {} 2023-05-24T21:10:35.421 DEBUG:boto:Token: None 2023-05-24T21:10:35.421 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 24 May 2023 21:10:35 GMT /qskrps-13/obj 2023-05-24T21:10:35.422 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:FcupaljGb7eQLYzF9J0UIQRU+S0= 2023-05-24T21:10:35.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:10:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:FcupaljGb7eQLYzF9J0UIQRU+S0='} 2023-05-24T21:10:35.468 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'LsSugEEr8NPamOV4ZLrgX.pixmT7G5y'), ('x-amz-request-id', 'tx0000067625cbf37e63e7f-00646e7d4b-1111-a2'), ('Date', 'Wed, 24 May 2023 21:10:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.469 DEBUG:rgw_multi.tests:zone=a2 version=LsSugEEr8NPamOV4ZLrgX.pixmT7G5y 2023-05-24T21:10:35.469 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:10:35.470 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:10:35.470 DEBUG:boto:Method: HEAD 2023-05-24T21:10:35.470 DEBUG:boto:Path: /qskrps-13/ 2023-05-24T21:10:35.471 DEBUG:boto:Data: 2023-05-24T21:10:35.471 DEBUG:boto:Headers: {} 2023-05-24T21:10:35.472 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:10:35.472 DEBUG:boto:Port: 8000 2023-05-24T21:10:35.472 DEBUG:boto:Params: {} 2023-05-24T21:10:35.473 DEBUG:boto:Token: None 2023-05-24T21:10:35.473 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:10:35 GMT /qskrps-13/ 2023-05-24T21:10:35.473 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Zm3PPjG10IqP0JC2UXVF+3wgkh4= 2023-05-24T21:10:35.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:10:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Zm3PPjG10IqP0JC2UXVF+3wgkh4=', 'Content-Length': '0'} 2023-05-24T21:10:35.476 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000377debcefa11d30d-00646e7d4b-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:10:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.477 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:10:35.477 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:10:35.478 DEBUG:boto:Method: PUT 2023-05-24T21:10:35.478 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:10:35.478 DEBUG:boto:Data: 2023-05-24T21:10:35.479 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-24T21:10:35.479 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:10:35.479 DEBUG:boto:Port: 8000 2023-05-24T21:10:35.480 DEBUG:boto:Params: {} 2023-05-24T21:10:35.480 DEBUG:boto:Token: None 2023-05-24T21:10:35.481 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 24 May 2023 21:10:35 GMT /qskrps-13/obj 2023-05-24T21:10:35.481 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:FcupaljGb7eQLYzF9J0UIQRU+S0= 2023-05-24T21:10:35.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:10:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:FcupaljGb7eQLYzF9J0UIQRU+S0='} 2023-05-24T21:10:35.497 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'hcen3y8E05rcJbSPsSSwDQfkEHk1NI7'), ('x-amz-request-id', 'tx0000061c88733e1281bed-00646e7d4b-110d-a1'), ('Date', 'Wed, 24 May 2023 21:10:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.498 DEBUG:rgw_multi.tests:zone=a1 version=hcen3y8E05rcJbSPsSSwDQfkEHk1NI7 2023-05-24T21:10:35.498 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:10:35.499 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:10:35.499 DEBUG:boto:Method: HEAD 2023-05-24T21:10:35.499 DEBUG:boto:Path: /qskrps-13/ 2023-05-24T21:10:35.500 DEBUG:boto:Data: 2023-05-24T21:10:35.500 DEBUG:boto:Headers: {} 2023-05-24T21:10:35.500 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:10:35.501 DEBUG:boto:Port: 8001 2023-05-24T21:10:35.501 DEBUG:boto:Params: {} 2023-05-24T21:10:35.502 DEBUG:boto:Token: None 2023-05-24T21:10:35.502 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:10:35 GMT /qskrps-13/ 2023-05-24T21:10:35.502 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Zm3PPjG10IqP0JC2UXVF+3wgkh4= 2023-05-24T21:10:35.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:10:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Zm3PPjG10IqP0JC2UXVF+3wgkh4=', 'Content-Length': '0'} 2023-05-24T21:10:35.505 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c9190be8f2590b1d-00646e7d4b-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:10:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.506 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:10:35.506 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:10:35.507 DEBUG:boto:Method: PUT 2023-05-24T21:10:35.507 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:10:35.507 DEBUG:boto:Data: 2023-05-24T21:10:35.508 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-24T21:10:35.508 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:10:35.509 DEBUG:boto:Port: 8001 2023-05-24T21:10:35.509 DEBUG:boto:Params: {} 2023-05-24T21:10:35.510 DEBUG:boto:Token: None 2023-05-24T21:10:35.510 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 24 May 2023 21:10:35 GMT /qskrps-13/obj 2023-05-24T21:10:35.510 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:FcupaljGb7eQLYzF9J0UIQRU+S0= 2023-05-24T21:10:35.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:10:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:FcupaljGb7eQLYzF9J0UIQRU+S0='} 2023-05-24T21:10:35.521 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'U-3Niwfe1WdmuD.qX1nCRvbymdYC9Ns'), ('x-amz-request-id', 'tx00000a857c49512f9cbef-00646e7d4b-1111-a2'), ('Date', 'Wed, 24 May 2023 21:10:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.521 DEBUG:rgw_multi.tests:zone=a2 version=U-3Niwfe1WdmuD.qX1nCRvbymdYC9Ns 2023-05-24T21:10:35.522 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:10:35.522 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:10:35.522 DEBUG:boto:Method: HEAD 2023-05-24T21:10:35.523 DEBUG:boto:Path: /qskrps-13/ 2023-05-24T21:10:35.523 DEBUG:boto:Data: 2023-05-24T21:10:35.523 DEBUG:boto:Headers: {} 2023-05-24T21:10:35.524 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:10:35.524 DEBUG:boto:Port: 8000 2023-05-24T21:10:35.524 DEBUG:boto:Params: {} 2023-05-24T21:10:35.525 DEBUG:boto:Token: None 2023-05-24T21:10:35.525 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:10:35 GMT /qskrps-13/ 2023-05-24T21:10:35.525 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Zm3PPjG10IqP0JC2UXVF+3wgkh4= 2023-05-24T21:10:35.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:10:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Zm3PPjG10IqP0JC2UXVF+3wgkh4=', 'Content-Length': '0'} 2023-05-24T21:10:35.528 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cedb0f486427e920-00646e7d4b-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:10:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.529 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:10:35.529 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:10:35.529 DEBUG:boto:Method: PUT 2023-05-24T21:10:35.530 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:10:35.530 DEBUG:boto:Data: 2023-05-24T21:10:35.530 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-24T21:10:35.531 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:10:35.531 DEBUG:boto:Port: 8000 2023-05-24T21:10:35.531 DEBUG:boto:Params: {} 2023-05-24T21:10:35.532 DEBUG:boto:Token: None 2023-05-24T21:10:35.532 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 24 May 2023 21:10:35 GMT /qskrps-13/obj 2023-05-24T21:10:35.533 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:FcupaljGb7eQLYzF9J0UIQRU+S0= 2023-05-24T21:10:35.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:10:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:FcupaljGb7eQLYzF9J0UIQRU+S0='} 2023-05-24T21:10:35.580 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'QzHAWShKdleKYlXrNH1akYPxtOhaV6z'), ('x-amz-request-id', 'tx00000682b7aa2dc544d90-00646e7d4b-110d-a1'), ('Date', 'Wed, 24 May 2023 21:10:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.583 DEBUG:rgw_multi.tests:zone=a1 version=QzHAWShKdleKYlXrNH1akYPxtOhaV6z 2023-05-24T21:10:35.583 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:10:35.583 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:10:35.584 DEBUG:boto:Method: HEAD 2023-05-24T21:10:35.584 DEBUG:boto:Path: /qskrps-13/ 2023-05-24T21:10:35.584 DEBUG:boto:Data: 2023-05-24T21:10:35.585 DEBUG:boto:Headers: {} 2023-05-24T21:10:35.585 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:10:35.585 DEBUG:boto:Port: 8001 2023-05-24T21:10:35.586 DEBUG:boto:Params: {} 2023-05-24T21:10:35.586 DEBUG:boto:Token: None 2023-05-24T21:10:35.586 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:10:35 GMT /qskrps-13/ 2023-05-24T21:10:35.587 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Zm3PPjG10IqP0JC2UXVF+3wgkh4= 2023-05-24T21:10:35.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:10:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Zm3PPjG10IqP0JC2UXVF+3wgkh4=', 'Content-Length': '0'} 2023-05-24T21:10:35.589 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e467d0e625ebd0e7-00646e7d4b-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:10:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.590 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:10:35.590 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:10:35.591 DEBUG:boto:Method: PUT 2023-05-24T21:10:35.592 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:10:35.592 DEBUG:boto:Data: 2023-05-24T21:10:35.592 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-24T21:10:35.593 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:10:35.593 DEBUG:boto:Port: 8001 2023-05-24T21:10:35.593 DEBUG:boto:Params: {} 2023-05-24T21:10:35.594 DEBUG:boto:Token: None 2023-05-24T21:10:35.594 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 24 May 2023 21:10:35 GMT /qskrps-13/obj 2023-05-24T21:10:35.594 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:FcupaljGb7eQLYzF9J0UIQRU+S0= 2023-05-24T21:10:35.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:10:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:FcupaljGb7eQLYzF9J0UIQRU+S0='} 2023-05-24T21:10:35.636 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'b9BUNjISbr2Ry0Y4wv-brHNozZY03Qi'), ('x-amz-request-id', 'tx00000f5f259b09bf70cee-00646e7d4b-1111-a2'), ('Date', 'Wed, 24 May 2023 21:10:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.637 DEBUG:rgw_multi.tests:zone=a2 version=b9BUNjISbr2Ry0Y4wv-brHNozZY03Qi 2023-05-24T21:10:35.638 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:10:35.639 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:10:35.640 DEBUG:boto:Method: HEAD 2023-05-24T21:10:35.640 DEBUG:boto:Path: /qskrps-13/ 2023-05-24T21:10:35.641 DEBUG:boto:Data: 2023-05-24T21:10:35.642 DEBUG:boto:Headers: {} 2023-05-24T21:10:35.643 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:10:35.643 DEBUG:boto:Port: 8000 2023-05-24T21:10:35.644 DEBUG:boto:Params: {} 2023-05-24T21:10:35.644 DEBUG:boto:Token: None 2023-05-24T21:10:35.644 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:10:35 GMT /qskrps-13/ 2023-05-24T21:10:35.645 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Zm3PPjG10IqP0JC2UXVF+3wgkh4= 2023-05-24T21:10:35.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:10:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Zm3PPjG10IqP0JC2UXVF+3wgkh4=', 'Content-Length': '0'} 2023-05-24T21:10:35.647 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bbf85f7a6eaa16e8-00646e7d4b-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:10:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.648 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:10:35.649 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:10:35.649 DEBUG:boto:Method: PUT 2023-05-24T21:10:35.649 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:10:35.650 DEBUG:boto:Data: 2023-05-24T21:10:35.650 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-24T21:10:35.650 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:10:35.651 DEBUG:boto:Port: 8000 2023-05-24T21:10:35.651 DEBUG:boto:Params: {} 2023-05-24T21:10:35.651 DEBUG:boto:Token: None 2023-05-24T21:10:35.652 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 24 May 2023 21:10:35 GMT /qskrps-13/obj 2023-05-24T21:10:35.652 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:FcupaljGb7eQLYzF9J0UIQRU+S0= 2023-05-24T21:10:35.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:10:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:FcupaljGb7eQLYzF9J0UIQRU+S0='} 2023-05-24T21:10:35.671 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '-H.jz834T6ck7fPleRHP.k9CJzkLgis'), ('x-amz-request-id', 'tx0000046d674d1b7c64850-00646e7d4b-110d-a1'), ('Date', 'Wed, 24 May 2023 21:10:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.672 DEBUG:rgw_multi.tests:zone=a1 version=-H.jz834T6ck7fPleRHP.k9CJzkLgis 2023-05-24T21:10:35.673 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:10:35.674 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:10:35.675 DEBUG:boto:Method: HEAD 2023-05-24T21:10:35.675 DEBUG:boto:Path: /qskrps-13/ 2023-05-24T21:10:35.675 DEBUG:boto:Data: 2023-05-24T21:10:35.676 DEBUG:boto:Headers: {} 2023-05-24T21:10:35.676 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:10:35.676 DEBUG:boto:Port: 8001 2023-05-24T21:10:35.677 DEBUG:boto:Params: {} 2023-05-24T21:10:35.677 DEBUG:boto:Token: None 2023-05-24T21:10:35.678 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:10:35 GMT /qskrps-13/ 2023-05-24T21:10:35.678 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Zm3PPjG10IqP0JC2UXVF+3wgkh4= 2023-05-24T21:10:35.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:10:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Zm3PPjG10IqP0JC2UXVF+3wgkh4=', 'Content-Length': '0'} 2023-05-24T21:10:35.681 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d440e1c78e4dc45f-00646e7d4b-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:10:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.682 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:10:35.683 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:10:35.683 DEBUG:boto:Method: PUT 2023-05-24T21:10:35.684 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:10:35.685 DEBUG:boto:Data: 2023-05-24T21:10:35.685 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-24T21:10:35.686 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:10:35.687 DEBUG:boto:Port: 8001 2023-05-24T21:10:35.687 DEBUG:boto:Params: {} 2023-05-24T21:10:35.688 DEBUG:boto:Token: None 2023-05-24T21:10:35.688 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 24 May 2023 21:10:35 GMT /qskrps-13/obj 2023-05-24T21:10:35.689 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:FcupaljGb7eQLYzF9J0UIQRU+S0= 2023-05-24T21:10:35.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:10:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:FcupaljGb7eQLYzF9J0UIQRU+S0='} 2023-05-24T21:10:35.703 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ORI9tDTIyFkUt3B9j46aV8ZBUYqr2Kp'), ('x-amz-request-id', 'tx000001c9cbb22f911601d-00646e7d4b-1111-a2'), ('Date', 'Wed, 24 May 2023 21:10:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.703 DEBUG:rgw_multi.tests:zone=a2 version=ORI9tDTIyFkUt3B9j46aV8ZBUYqr2Kp 2023-05-24T21:10:35.704 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:10:35.704 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:10:35.704 DEBUG:boto:Method: HEAD 2023-05-24T21:10:35.705 DEBUG:boto:Path: /qskrps-13/ 2023-05-24T21:10:35.705 DEBUG:boto:Data: 2023-05-24T21:10:35.705 DEBUG:boto:Headers: {} 2023-05-24T21:10:35.706 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:10:35.706 DEBUG:boto:Port: 8000 2023-05-24T21:10:35.706 DEBUG:boto:Params: {} 2023-05-24T21:10:35.707 DEBUG:boto:Token: None 2023-05-24T21:10:35.707 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:10:35 GMT /qskrps-13/ 2023-05-24T21:10:35.708 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Zm3PPjG10IqP0JC2UXVF+3wgkh4= 2023-05-24T21:10:35.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:10:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Zm3PPjG10IqP0JC2UXVF+3wgkh4=', 'Content-Length': '0'} 2023-05-24T21:10:35.710 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000067656997485a7cf2-00646e7d4b-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:10:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.711 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:10:35.711 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:10:35.712 DEBUG:boto:Method: PUT 2023-05-24T21:10:35.712 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:10:35.712 DEBUG:boto:Data: 2023-05-24T21:10:35.713 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-24T21:10:35.713 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:10:35.713 DEBUG:boto:Port: 8000 2023-05-24T21:10:35.714 DEBUG:boto:Params: {} 2023-05-24T21:10:35.714 DEBUG:boto:Token: None 2023-05-24T21:10:35.714 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 24 May 2023 21:10:35 GMT /qskrps-13/obj 2023-05-24T21:10:35.715 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:FcupaljGb7eQLYzF9J0UIQRU+S0= 2023-05-24T21:10:35.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:10:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:FcupaljGb7eQLYzF9J0UIQRU+S0='} 2023-05-24T21:10:35.731 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'T6O7lunS-ugcpId8P5cmF9RHD1HJqI8'), ('x-amz-request-id', 'tx000004168936aebab65f8-00646e7d4b-110d-a1'), ('Date', 'Wed, 24 May 2023 21:10:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.731 DEBUG:rgw_multi.tests:zone=a1 version=T6O7lunS-ugcpId8P5cmF9RHD1HJqI8 2023-05-24T21:10:35.732 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:10:35.732 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:10:35.732 DEBUG:boto:Method: HEAD 2023-05-24T21:10:35.733 DEBUG:boto:Path: /qskrps-13/ 2023-05-24T21:10:35.733 DEBUG:boto:Data: 2023-05-24T21:10:35.733 DEBUG:boto:Headers: {} 2023-05-24T21:10:35.734 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:10:35.734 DEBUG:boto:Port: 8001 2023-05-24T21:10:35.734 DEBUG:boto:Params: {} 2023-05-24T21:10:35.735 DEBUG:boto:Token: None 2023-05-24T21:10:35.735 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:10:35 GMT /qskrps-13/ 2023-05-24T21:10:35.736 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Zm3PPjG10IqP0JC2UXVF+3wgkh4= 2023-05-24T21:10:35.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:10:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Zm3PPjG10IqP0JC2UXVF+3wgkh4=', 'Content-Length': '0'} 2023-05-24T21:10:35.738 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b3ba2d1238e44464-00646e7d4b-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:10:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.739 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:10:35.739 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:10:35.740 DEBUG:boto:Method: PUT 2023-05-24T21:10:35.740 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:10:35.741 DEBUG:boto:Data: 2023-05-24T21:10:35.741 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-24T21:10:35.741 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:10:35.742 DEBUG:boto:Port: 8001 2023-05-24T21:10:35.742 DEBUG:boto:Params: {} 2023-05-24T21:10:35.742 DEBUG:boto:Token: None 2023-05-24T21:10:35.743 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 24 May 2023 21:10:35 GMT /qskrps-13/obj 2023-05-24T21:10:35.743 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:FcupaljGb7eQLYzF9J0UIQRU+S0= 2023-05-24T21:10:35.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:10:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:FcupaljGb7eQLYzF9J0UIQRU+S0='} 2023-05-24T21:10:35.754 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'RsY7UgV2wkSvgE1I0nN3nCJ0pjKqhZX'), ('x-amz-request-id', 'tx00000d1064f6226757f38-00646e7d4b-1111-a2'), ('Date', 'Wed, 24 May 2023 21:10:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.755 DEBUG:rgw_multi.tests:zone=a2 version=RsY7UgV2wkSvgE1I0nN3nCJ0pjKqhZX 2023-05-24T21:10:35.756 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:10:35.756 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:10:35.757 DEBUG:boto:Method: HEAD 2023-05-24T21:10:35.757 DEBUG:boto:Path: /qskrps-13/ 2023-05-24T21:10:35.758 DEBUG:boto:Data: 2023-05-24T21:10:35.758 DEBUG:boto:Headers: {} 2023-05-24T21:10:35.759 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:10:35.760 DEBUG:boto:Port: 8000 2023-05-24T21:10:35.760 DEBUG:boto:Params: {} 2023-05-24T21:10:35.761 DEBUG:boto:Token: None 2023-05-24T21:10:35.761 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:10:35 GMT /qskrps-13/ 2023-05-24T21:10:35.762 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Zm3PPjG10IqP0JC2UXVF+3wgkh4= 2023-05-24T21:10:35.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:10:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Zm3PPjG10IqP0JC2UXVF+3wgkh4=', 'Content-Length': '0'} 2023-05-24T21:10:35.765 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000569393e40b57cb4e-00646e7d4b-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:10:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.766 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:10:35.766 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:10:35.767 DEBUG:boto:Method: PUT 2023-05-24T21:10:35.767 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:10:35.767 DEBUG:boto:Data: 2023-05-24T21:10:35.768 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-24T21:10:35.768 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:10:35.768 DEBUG:boto:Port: 8000 2023-05-24T21:10:35.769 DEBUG:boto:Params: {} 2023-05-24T21:10:35.769 DEBUG:boto:Token: None 2023-05-24T21:10:35.770 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 24 May 2023 21:10:35 GMT /qskrps-13/obj 2023-05-24T21:10:35.770 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:FcupaljGb7eQLYzF9J0UIQRU+S0= 2023-05-24T21:10:35.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:10:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:FcupaljGb7eQLYzF9J0UIQRU+S0='} 2023-05-24T21:10:35.784 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-amz-request-id', 'tx00000c54d78fb2d50c46c-00646e7d4b-110d-a1'), ('Date', 'Wed, 24 May 2023 21:10:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.785 DEBUG:rgw_multi.tests:zone=a1 version=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:10:35.785 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:10:35.786 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:10:35.786 DEBUG:boto:Method: HEAD 2023-05-24T21:10:35.786 DEBUG:boto:Path: /qskrps-13/ 2023-05-24T21:10:35.787 DEBUG:boto:Data: 2023-05-24T21:10:35.787 DEBUG:boto:Headers: {} 2023-05-24T21:10:35.787 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:10:35.788 DEBUG:boto:Port: 8001 2023-05-24T21:10:35.788 DEBUG:boto:Params: {} 2023-05-24T21:10:35.788 DEBUG:boto:Token: None 2023-05-24T21:10:35.789 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:10:35 GMT /qskrps-13/ 2023-05-24T21:10:35.789 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Zm3PPjG10IqP0JC2UXVF+3wgkh4= 2023-05-24T21:10:35.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:10:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Zm3PPjG10IqP0JC2UXVF+3wgkh4=', 'Content-Length': '0'} 2023-05-24T21:10:35.792 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000e8e676c141679cc-00646e7d4b-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:10:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.792 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:10:35.793 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:10:35.793 DEBUG:boto:Method: PUT 2023-05-24T21:10:35.793 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:10:35.794 DEBUG:boto:Data: 2023-05-24T21:10:35.794 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-24T21:10:35.794 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:10:35.795 DEBUG:boto:Port: 8001 2023-05-24T21:10:35.795 DEBUG:boto:Params: {} 2023-05-24T21:10:35.796 DEBUG:boto:Token: None 2023-05-24T21:10:35.796 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 24 May 2023 21:10:35 GMT /qskrps-13/obj 2023-05-24T21:10:35.797 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:FcupaljGb7eQLYzF9J0UIQRU+S0= 2023-05-24T21:10:35.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:10:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:FcupaljGb7eQLYzF9J0UIQRU+S0='} 2023-05-24T21:10:35.840 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'jqoKobaDdC.ssJkHV2IlgFRFcAWxTE9'), ('x-amz-request-id', 'tx00000bbc255bc380f0bc5-00646e7d4b-1111-a2'), ('Date', 'Wed, 24 May 2023 21:10:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:10:35.841 DEBUG:rgw_multi.tests:zone=a2 version=jqoKobaDdC.ssJkHV2IlgFRFcAWxTE9 2023-05-24T21:10:35.841 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=qskrps-13 2023-05-24T21:10:35.842 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:10:35.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:10:35.843 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-13 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:10:35.900 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:10:35.900 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:10:35.946 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:10:35.940+0000 7f3376689780 1 waiting for incremental sync to catch up: 2023-05-24T21:10:35.946 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000002.6.11, , , , , ] 2023-05-24T21:10:35.946 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000003.8.10, , , 00000000020.45.12, , ] 2023-05-24T21:10:40.947 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:10:40.940+0000 7f3376689780 1 waiting for incremental sync to catch up: 2023-05-24T21:10:40.948 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000002.6.11, , , , , ] 2023-05-24T21:10:40.948 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000003.8.10, , , 00000000020.45.12, , ] 2023-05-24T21:10:45.949 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:10:45.942+0000 7f3376689780 1 waiting for incremental sync to catch up: 2023-05-24T21:10:45.949 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000002.6.11, , , , , ] 2023-05-24T21:10:45.949 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000003.8.10, , , 00000000020.45.12, , ] 2023-05-24T21:10:50.950 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:10:50.943+0000 7f3376689780 1 waiting for incremental sync to catch up: 2023-05-24T21:10:50.951 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000002.6.11, , , , , ] 2023-05-24T21:10:50.951 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000003.8.10, , , 00000000020.45.12, , ] 2023-05-24T21:10:55.951 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:10:55.945+0000 7f3376689780 1 waiting for incremental sync to catch up: 2023-05-24T21:10:55.952 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000003.8.10, , , 00000000009.24.6, , ] 2023-05-24T21:10:55.952 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000003.8.10, , , 00000000020.45.12, , ] 2023-05-24T21:11:00.953 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:11:00.947+0000 7f3376689780 1 waiting for incremental sync to catch up: 2023-05-24T21:11:00.953 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000003.8.10, , , 00000000009.24.6, , ] 2023-05-24T21:11:00.953 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000003.8.10, , , 00000000020.45.12, , ] 2023-05-24T21:11:05.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:11:05.948+0000 7f3376689780 1 waiting for incremental sync to catch up: 2023-05-24T21:11:05.954 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000003.8.10, , , 00000000009.24.6, , ] 2023-05-24T21:11:05.954 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000003.8.10, , , 00000000020.45.12, , ] 2023-05-24T21:11:10.955 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:11:10.949+0000 7f3376689780 1 waiting for incremental sync to catch up: 2023-05-24T21:11:10.956 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000003.8.10, , , 00000000009.24.6, , ] 2023-05-24T21:11:10.956 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000003.8.10, , , 00000000020.45.12, , ] 2023-05-24T21:11:15.957 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:11:15.951+0000 7f3376689780 1 bucket sync caught up with source: 2023-05-24T21:11:15.957 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000003.8.10, , , 00000000031.76.9, , ] 2023-05-24T21:11:15.957 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000003.8.10, , , 00000000020.45.12, , ] 2023-05-24T21:11:15.957 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:11:15.951+0000 7f3376689780 0 bucket checkpoint complete 2023-05-24T21:11:15.975 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=qskrps-13 2023-05-24T21:11:15.976 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:11:15.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:11:15.977 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-13 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:11:16.040 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:11:16.041 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:11:16.102 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:11:16.097+0000 7fad870a3780 1 waiting for incremental sync to catch up: 2023-05-24T21:11:16.103 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000002.15.11, , , 00000000024.68.6, , ] 2023-05-24T21:11:16.103 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000002.15.11, , , 00000000031.87.12, , ] 2023-05-24T21:11:21.104 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:11:21.098+0000 7fad870a3780 1 waiting for incremental sync to catch up: 2023-05-24T21:11:21.104 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000002.15.11, , , 00000000024.68.6, , ] 2023-05-24T21:11:21.104 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000002.15.11, , , 00000000031.87.12, , ] 2023-05-24T21:11:26.105 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:11:26.099+0000 7fad870a3780 1 waiting for incremental sync to catch up: 2023-05-24T21:11:26.106 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000002.15.11, , , 00000000024.68.6, , ] 2023-05-24T21:11:26.106 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000002.15.11, , , 00000000031.87.12, , ] 2023-05-24T21:11:31.106 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:11:31.101+0000 7fad870a3780 1 waiting for incremental sync to catch up: 2023-05-24T21:11:31.107 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000002.15.11, , , 00000000024.68.6, , ] 2023-05-24T21:11:31.107 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000002.15.11, , , 00000000031.87.12, , ] 2023-05-24T21:11:36.108 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:11:36.101+0000 7fad870a3780 1 bucket sync caught up with source: 2023-05-24T21:11:36.108 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000002.15.11, , , 00000000031.87.12, , ] 2023-05-24T21:11:36.108 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000002.15.11, , , 00000000031.87.12, , ] 2023-05-24T21:11:36.109 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:11:36.101+0000 7fad870a3780 0 bucket checkpoint complete 2023-05-24T21:11:36.130 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qskrps-13 zones={a2, a1} 2023-05-24T21:11:36.130 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:36.131 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:36.131 DEBUG:boto:Method: HEAD 2023-05-24T21:11:36.131 DEBUG:boto:Path: /qskrps-13/ 2023-05-24T21:11:36.132 DEBUG:boto:Data: 2023-05-24T21:11:36.132 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.132 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:36.133 DEBUG:boto:Port: 8001 2023-05-24T21:11:36.133 DEBUG:boto:Params: {} 2023-05-24T21:11:36.134 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-24T21:11:36.134 DEBUG:boto:Token: None 2023-05-24T21:11:36.135 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:36 GMT /qskrps-13/ 2023-05-24T21:11:36.135 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:vGB+y9SylFkTNJRU/WjkKTCCoCE= 2023-05-24T21:11:36.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:vGB+y9SylFkTNJRU/WjkKTCCoCE=', 'Content-Length': '0'} 2023-05-24T21:11:36.139 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '160'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000240181c1add62ea9-00646e7d88-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.140 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:36.140 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:36.140 DEBUG:boto:Method: HEAD 2023-05-24T21:11:36.141 DEBUG:boto:Path: /qskrps-13/ 2023-05-24T21:11:36.141 DEBUG:boto:Data: 2023-05-24T21:11:36.141 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.142 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:36.142 DEBUG:boto:Port: 8000 2023-05-24T21:11:36.142 DEBUG:boto:Params: {} 2023-05-24T21:11:36.143 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T21:11:36.143 DEBUG:boto:Token: None 2023-05-24T21:11:36.144 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:36 GMT /qskrps-13/ 2023-05-24T21:11:36.144 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:vGB+y9SylFkTNJRU/WjkKTCCoCE= 2023-05-24T21:11:36.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:vGB+y9SylFkTNJRU/WjkKTCCoCE=', 'Content-Length': '0'} 2023-05-24T21:11:36.148 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '160'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fca146370d9087d0-00646e7d88-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.148 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-24T21:11:36.149 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:36.149 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:36.149 DEBUG:boto:path=/qskrps-13/?versions&max-keys=999 2023-05-24T21:11:36.150 DEBUG:boto:auth_path=/qskrps-13/?versions&max-keys=999 2023-05-24T21:11:36.150 DEBUG:boto:Method: GET 2023-05-24T21:11:36.150 DEBUG:boto:Path: /qskrps-13/?versions&max-keys=999 2023-05-24T21:11:36.151 DEBUG:boto:Data: 2023-05-24T21:11:36.151 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.152 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:36.152 DEBUG:boto:Port: 8001 2023-05-24T21:11:36.152 DEBUG:boto:Params: {} 2023-05-24T21:11:36.153 DEBUG:boto:Token: None 2023-05-24T21:11:36.153 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:36 GMT /qskrps-13/?versions 2023-05-24T21:11:36.154 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:XRde6RQQ/fKjaOR9B7swlwXLUOk= 2023-05-24T21:11:36.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:XRde6RQQ/fKjaOR9B7swlwXLUOk=', 'Content-Length': '0'} 2023-05-24T21:11:36.157 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090234c1d3a2936c5-00646e7d88-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.157 DEBUG:boto:b'qskrps-13999falsedummyg04CLEvxE5VlkW2A9PqVzMmjG-oLrNXtrue2023-05-24T21:09:59.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjG.4hkCINEGMtwc87D.MHv0zLLwQSDS7true2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjjqoKobaDdC.ssJkHV2IlgFRFcAWxTE9false2023-05-24T21:10:35.794Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRsY7UgV2wkSvgE1I0nN3nCJ0pjKqhZXfalse2023-05-24T21:10:35.741Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjT6O7lunS-ugcpId8P5cmF9RHD1HJqI8false2023-05-24T21:10:35.712Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-H.jz834T6ck7fPleRHP.k9CJzkLgisfalse2023-05-24T21:10:35.649Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjORI9tDTIyFkUt3B9j46aV8ZBUYqr2Kpfalse2023-05-24T21:10:35.687Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQzHAWShKdleKYlXrNH1akYPxtOhaV6zfalse2023-05-24T21:10:35.530Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjb9BUNjISbr2Ry0Y4wv-brHNozZY03Qifalse2023-05-24T21:10:35.591Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjU-3Niwfe1WdmuD.qX1nCRvbymdYC9Nsfalse2023-05-24T21:10:35.507Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhcen3y8E05rcJbSPsSSwDQfkEHk1NI7false2023-05-24T21:10:35.478Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLsSugEEr8NPamOV4ZLrgX.pixmT7G5yfalse2023-05-24T21:10:35.419Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjc.ug7Ld36X8.TaQXWIjev99m-BKd5qJfalse2023-05-24T21:10:35.397Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjDkBjLdO6nXireb7JdIhAsx00H76MZGDfalse2023-05-24T21:10:35.343Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjU4zUwdtKVyScPebZNZT8NHXc8rQ.8f-false2023-05-24T21:10:35.375Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1BSwhaY-MjwOOLy73m-3KiPjCrRBsodfalse2023-05-24T21:10:35.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXBH9.ADrbaEY1JrgZ6O0QvO0TBVsG2vfalse2023-05-24T21:10:35.285Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3wYQ13EDow0kiwE6ITIMyTWEPt0A0Hxfalse2023-05-24T21:10:35.252Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjvVL.Oc-NasmiTK3cSFNuRAIlXlfwbACfalse2023-05-24T21:10:35.191Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBdBswNggliM4dGAQ3Dgxcgleuyd0Qxofalse2023-05-24T21:10:35.134Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjagpDexVocguP.SXhiu6FyO5RbOuqAnwfalse2023-05-24T21:10:35.107Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:36.159 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-05-24T21:11:36.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:11:36.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:11:36.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:11:36.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:11:36.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:11:36.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:11:36.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:11:36.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:11:36.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:11:36.162 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:11:36.162 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:11:36.162 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:11:36.163 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:11:36.163 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:11:36.163 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:11:36.163 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:11:36.164 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:11:36.164 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:11:36.164 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:11:36.165 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:11:36.165 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-24T21:11:36.165 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:36.166 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:36.166 DEBUG:boto:path=/qskrps-13/?versions&max-keys=999 2023-05-24T21:11:36.166 DEBUG:boto:auth_path=/qskrps-13/?versions&max-keys=999 2023-05-24T21:11:36.167 DEBUG:boto:Method: GET 2023-05-24T21:11:36.167 DEBUG:boto:Path: /qskrps-13/?versions&max-keys=999 2023-05-24T21:11:36.167 DEBUG:boto:Data: 2023-05-24T21:11:36.168 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.168 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:36.168 DEBUG:boto:Port: 8000 2023-05-24T21:11:36.169 DEBUG:boto:Params: {} 2023-05-24T21:11:36.169 DEBUG:boto:Token: None 2023-05-24T21:11:36.170 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:36 GMT /qskrps-13/?versions 2023-05-24T21:11:36.170 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:XRde6RQQ/fKjaOR9B7swlwXLUOk= 2023-05-24T21:11:36.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:XRde6RQQ/fKjaOR9B7swlwXLUOk=', 'Content-Length': '0'} 2023-05-24T21:11:36.173 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7d728ce1e15eee5-00646e7d88-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.174 DEBUG:boto:b'qskrps-13999falsedummyg04CLEvxE5VlkW2A9PqVzMmjG-oLrNXtrue2023-05-24T21:09:59.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjG.4hkCINEGMtwc87D.MHv0zLLwQSDS7true2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjjqoKobaDdC.ssJkHV2IlgFRFcAWxTE9false2023-05-24T21:10:35.794Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRsY7UgV2wkSvgE1I0nN3nCJ0pjKqhZXfalse2023-05-24T21:10:35.741Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjT6O7lunS-ugcpId8P5cmF9RHD1HJqI8false2023-05-24T21:10:35.712Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-H.jz834T6ck7fPleRHP.k9CJzkLgisfalse2023-05-24T21:10:35.649Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjORI9tDTIyFkUt3B9j46aV8ZBUYqr2Kpfalse2023-05-24T21:10:35.687Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQzHAWShKdleKYlXrNH1akYPxtOhaV6zfalse2023-05-24T21:10:35.530Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjb9BUNjISbr2Ry0Y4wv-brHNozZY03Qifalse2023-05-24T21:10:35.591Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjU-3Niwfe1WdmuD.qX1nCRvbymdYC9Nsfalse2023-05-24T21:10:35.507Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhcen3y8E05rcJbSPsSSwDQfkEHk1NI7false2023-05-24T21:10:35.478Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLsSugEEr8NPamOV4ZLrgX.pixmT7G5yfalse2023-05-24T21:10:35.419Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjc.ug7Ld36X8.TaQXWIjev99m-BKd5qJfalse2023-05-24T21:10:35.397Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjDkBjLdO6nXireb7JdIhAsx00H76MZGDfalse2023-05-24T21:10:35.343Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjU4zUwdtKVyScPebZNZT8NHXc8rQ.8f-false2023-05-24T21:10:35.375Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1BSwhaY-MjwOOLy73m-3KiPjCrRBsodfalse2023-05-24T21:10:35.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXBH9.ADrbaEY1JrgZ6O0QvO0TBVsG2vfalse2023-05-24T21:10:35.285Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3wYQ13EDow0kiwE6ITIMyTWEPt0A0Hxfalse2023-05-24T21:10:35.252Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjvVL.Oc-NasmiTK3cSFNuRAIlXlfwbACfalse2023-05-24T21:10:35.191Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBdBswNggliM4dGAQ3Dgxcgleuyd0Qxofalse2023-05-24T21:10:35.134Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjagpDexVocguP.SXhiu6FyO5RbOuqAnwfalse2023-05-24T21:10:35.107Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:36.175 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-05-24T21:11:36.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:11:36.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:11:36.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:11:36.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:11:36.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:11:36.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:11:36.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:11:36.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:11:36.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:11:36.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:11:36.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:11:36.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:11:36.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:11:36.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:11:36.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:11:36.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:11:36.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:11:36.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:11:36.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:11:36.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:11:36.181 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:36.182 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:36.182 DEBUG:boto:path=/qskrps-13/?versions&max-keys=999 2023-05-24T21:11:36.182 DEBUG:boto:auth_path=/qskrps-13/?versions&max-keys=999 2023-05-24T21:11:36.183 DEBUG:boto:Method: GET 2023-05-24T21:11:36.183 DEBUG:boto:Path: /qskrps-13/?versions&max-keys=999 2023-05-24T21:11:36.183 DEBUG:boto:Data: 2023-05-24T21:11:36.184 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.184 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:36.184 DEBUG:boto:Port: 8001 2023-05-24T21:11:36.185 DEBUG:boto:Params: {} 2023-05-24T21:11:36.185 DEBUG:boto:Token: None 2023-05-24T21:11:36.186 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:36 GMT /qskrps-13/?versions 2023-05-24T21:11:36.186 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:XRde6RQQ/fKjaOR9B7swlwXLUOk= 2023-05-24T21:11:36.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:XRde6RQQ/fKjaOR9B7swlwXLUOk=', 'Content-Length': '0'} 2023-05-24T21:11:36.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008265c5d1c5a7b45a-00646e7d88-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.190 DEBUG:boto:b'qskrps-13999falsedummyg04CLEvxE5VlkW2A9PqVzMmjG-oLrNXtrue2023-05-24T21:09:59.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjG.4hkCINEGMtwc87D.MHv0zLLwQSDS7true2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjjqoKobaDdC.ssJkHV2IlgFRFcAWxTE9false2023-05-24T21:10:35.794Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRsY7UgV2wkSvgE1I0nN3nCJ0pjKqhZXfalse2023-05-24T21:10:35.741Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjT6O7lunS-ugcpId8P5cmF9RHD1HJqI8false2023-05-24T21:10:35.712Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-H.jz834T6ck7fPleRHP.k9CJzkLgisfalse2023-05-24T21:10:35.649Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjORI9tDTIyFkUt3B9j46aV8ZBUYqr2Kpfalse2023-05-24T21:10:35.687Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQzHAWShKdleKYlXrNH1akYPxtOhaV6zfalse2023-05-24T21:10:35.530Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjb9BUNjISbr2Ry0Y4wv-brHNozZY03Qifalse2023-05-24T21:10:35.591Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjU-3Niwfe1WdmuD.qX1nCRvbymdYC9Nsfalse2023-05-24T21:10:35.507Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhcen3y8E05rcJbSPsSSwDQfkEHk1NI7false2023-05-24T21:10:35.478Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLsSugEEr8NPamOV4ZLrgX.pixmT7G5yfalse2023-05-24T21:10:35.419Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjc.ug7Ld36X8.TaQXWIjev99m-BKd5qJfalse2023-05-24T21:10:35.397Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjDkBjLdO6nXireb7JdIhAsx00H76MZGDfalse2023-05-24T21:10:35.343Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjU4zUwdtKVyScPebZNZT8NHXc8rQ.8f-false2023-05-24T21:10:35.375Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1BSwhaY-MjwOOLy73m-3KiPjCrRBsodfalse2023-05-24T21:10:35.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXBH9.ADrbaEY1JrgZ6O0QvO0TBVsG2vfalse2023-05-24T21:10:35.285Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3wYQ13EDow0kiwE6ITIMyTWEPt0A0Hxfalse2023-05-24T21:10:35.252Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjvVL.Oc-NasmiTK3cSFNuRAIlXlfwbACfalse2023-05-24T21:10:35.191Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBdBswNggliM4dGAQ3Dgxcgleuyd0Qxofalse2023-05-24T21:10:35.134Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjagpDexVocguP.SXhiu6FyO5RbOuqAnwfalse2023-05-24T21:10:35.107Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:36.192 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:36.192 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:36.193 DEBUG:boto:path=/qskrps-13/?versions&max-keys=999 2023-05-24T21:11:36.193 DEBUG:boto:auth_path=/qskrps-13/?versions&max-keys=999 2023-05-24T21:11:36.193 DEBUG:boto:Method: GET 2023-05-24T21:11:36.194 DEBUG:boto:Path: /qskrps-13/?versions&max-keys=999 2023-05-24T21:11:36.194 DEBUG:boto:Data: 2023-05-24T21:11:36.194 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.195 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:36.195 DEBUG:boto:Port: 8000 2023-05-24T21:11:36.195 DEBUG:boto:Params: {} 2023-05-24T21:11:36.196 DEBUG:boto:Token: None 2023-05-24T21:11:36.196 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:36 GMT /qskrps-13/?versions 2023-05-24T21:11:36.197 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:XRde6RQQ/fKjaOR9B7swlwXLUOk= 2023-05-24T21:11:36.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:XRde6RQQ/fKjaOR9B7swlwXLUOk=', 'Content-Length': '0'} 2023-05-24T21:11:36.200 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cad690e3ee2dee74-00646e7d88-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.201 DEBUG:boto:b'qskrps-13999falsedummyg04CLEvxE5VlkW2A9PqVzMmjG-oLrNXtrue2023-05-24T21:09:59.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjG.4hkCINEGMtwc87D.MHv0zLLwQSDS7true2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjjqoKobaDdC.ssJkHV2IlgFRFcAWxTE9false2023-05-24T21:10:35.794Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRsY7UgV2wkSvgE1I0nN3nCJ0pjKqhZXfalse2023-05-24T21:10:35.741Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjT6O7lunS-ugcpId8P5cmF9RHD1HJqI8false2023-05-24T21:10:35.712Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-H.jz834T6ck7fPleRHP.k9CJzkLgisfalse2023-05-24T21:10:35.649Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjORI9tDTIyFkUt3B9j46aV8ZBUYqr2Kpfalse2023-05-24T21:10:35.687Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQzHAWShKdleKYlXrNH1akYPxtOhaV6zfalse2023-05-24T21:10:35.530Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjb9BUNjISbr2Ry0Y4wv-brHNozZY03Qifalse2023-05-24T21:10:35.591Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjU-3Niwfe1WdmuD.qX1nCRvbymdYC9Nsfalse2023-05-24T21:10:35.507Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhcen3y8E05rcJbSPsSSwDQfkEHk1NI7false2023-05-24T21:10:35.478Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLsSugEEr8NPamOV4ZLrgX.pixmT7G5yfalse2023-05-24T21:10:35.419Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjc.ug7Ld36X8.TaQXWIjev99m-BKd5qJfalse2023-05-24T21:10:35.397Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjDkBjLdO6nXireb7JdIhAsx00H76MZGDfalse2023-05-24T21:10:35.343Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjU4zUwdtKVyScPebZNZT8NHXc8rQ.8f-false2023-05-24T21:10:35.375Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1BSwhaY-MjwOOLy73m-3KiPjCrRBsodfalse2023-05-24T21:10:35.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXBH9.ADrbaEY1JrgZ6O0QvO0TBVsG2vfalse2023-05-24T21:10:35.285Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3wYQ13EDow0kiwE6ITIMyTWEPt0A0Hxfalse2023-05-24T21:10:35.252Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjvVL.Oc-NasmiTK3cSFNuRAIlXlfwbACfalse2023-05-24T21:10:35.191Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBdBswNggliM4dGAQ3Dgxcgleuyd0Qxofalse2023-05-24T21:10:35.134Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjagpDexVocguP.SXhiu6FyO5RbOuqAnwfalse2023-05-24T21:10:35.107Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:36.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-05-24T21:11:36.202 DEBUG:boto:path=/qskrps-13/dummy 2023-05-24T21:11:36.203 DEBUG:boto:auth_path=/qskrps-13/dummy 2023-05-24T21:11:36.203 DEBUG:boto:path=/qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:11:36.204 DEBUG:boto:auth_path=/qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:11:36.204 DEBUG:boto:Method: GET 2023-05-24T21:11:36.204 DEBUG:boto:Path: /qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:11:36.205 DEBUG:boto:Data: 2023-05-24T21:11:36.205 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.205 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:36.206 DEBUG:boto:Port: 8001 2023-05-24T21:11:36.206 DEBUG:boto:Params: {} 2023-05-24T21:11:36.207 DEBUG:boto:Token: None 2023-05-24T21:11:36.207 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:36 GMT /qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:11:36.207 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:K1pk4SMSlUV69uBXdFZ3HwInFXg= 2023-05-24T21:11:36.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:K1pk4SMSlUV69uBXdFZ3HwInFXg=', 'Content-Length': '0'} 2023-05-24T21:11:36.210 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:59 GMT'), ('x-amz-version-id', 'g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000085cd39ceb1e9b5fa-00646e7d88-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.210 DEBUG:boto:path=/qskrps-13/dummy 2023-05-24T21:11:36.211 DEBUG:boto:auth_path=/qskrps-13/dummy 2023-05-24T21:11:36.211 DEBUG:boto:path=/qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:11:36.212 DEBUG:boto:auth_path=/qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:11:36.212 DEBUG:boto:Method: GET 2023-05-24T21:11:36.212 DEBUG:boto:Path: /qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:11:36.213 DEBUG:boto:Data: 2023-05-24T21:11:36.213 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.213 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:36.214 DEBUG:boto:Port: 8000 2023-05-24T21:11:36.214 DEBUG:boto:Params: {} 2023-05-24T21:11:36.215 DEBUG:boto:Token: None 2023-05-24T21:11:36.215 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:36 GMT /qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:11:36.215 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:K1pk4SMSlUV69uBXdFZ3HwInFXg= 2023-05-24T21:11:36.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:K1pk4SMSlUV69uBXdFZ3HwInFXg=', 'Content-Length': '0'} 2023-05-24T21:11:36.217 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:59 GMT'), ('x-amz-version-id', 'g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007777a6e7c300b9d9-00646e7d88-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.218 DEBUG:boto:path=/qskrps-13/dummy 2023-05-24T21:11:36.218 DEBUG:boto:auth_path=/qskrps-13/dummy 2023-05-24T21:11:36.219 DEBUG:boto:path=/qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:11:36.219 DEBUG:boto:auth_path=/qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:11:36.220 DEBUG:boto:Method: HEAD 2023-05-24T21:11:36.220 DEBUG:boto:Path: /qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:11:36.220 DEBUG:boto:Data: 2023-05-24T21:11:36.221 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.221 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:36.221 DEBUG:boto:Port: 8001 2023-05-24T21:11:36.222 DEBUG:boto:Params: {} 2023-05-24T21:11:36.222 DEBUG:boto:Token: None 2023-05-24T21:11:36.222 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:36 GMT /qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:11:36.223 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:+rw23cnZTPX597ReSzmlI+XbaUk= 2023-05-24T21:11:36.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:+rw23cnZTPX597ReSzmlI+XbaUk=', 'Content-Length': '0'} 2023-05-24T21:11:36.225 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:59 GMT'), ('x-amz-version-id', 'g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b65ec34bb5572039-00646e7d88-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.225 DEBUG:boto:path=/qskrps-13/dummy 2023-05-24T21:11:36.226 DEBUG:boto:auth_path=/qskrps-13/dummy 2023-05-24T21:11:36.226 DEBUG:boto:path=/qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:11:36.226 DEBUG:boto:auth_path=/qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:11:36.227 DEBUG:boto:Method: HEAD 2023-05-24T21:11:36.227 DEBUG:boto:Path: /qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:11:36.227 DEBUG:boto:Data: 2023-05-24T21:11:36.228 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.228 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:36.229 DEBUG:boto:Port: 8000 2023-05-24T21:11:36.229 DEBUG:boto:Params: {} 2023-05-24T21:11:36.229 DEBUG:boto:Token: None 2023-05-24T21:11:36.230 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:36 GMT /qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:11:36.230 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:+rw23cnZTPX597ReSzmlI+XbaUk= 2023-05-24T21:11:36.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:+rw23cnZTPX597ReSzmlI+XbaUk=', 'Content-Length': '0'} 2023-05-24T21:11:36.232 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:59 GMT'), ('x-amz-version-id', 'g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b01dd674ad605d03-00646e7d88-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-05-24T21:11:36.233 DEBUG:boto:path=/qskrps-13/dummy 2023-05-24T21:11:36.233 DEBUG:boto:auth_path=/qskrps-13/dummy 2023-05-24T21:11:36.234 DEBUG:boto:path=/qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:11:36.234 DEBUG:boto:auth_path=/qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:11:36.234 DEBUG:boto:Method: GET 2023-05-24T21:11:36.235 DEBUG:boto:Path: /qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:11:36.235 DEBUG:boto:Data: 2023-05-24T21:11:36.236 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.236 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:36.236 DEBUG:boto:Port: 8001 2023-05-24T21:11:36.237 DEBUG:boto:Params: {} 2023-05-24T21:11:36.237 DEBUG:boto:Token: None 2023-05-24T21:11:36.238 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:36 GMT /qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:11:36.238 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:K1pk4SMSlUV69uBXdFZ3HwInFXg= 2023-05-24T21:11:36.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:K1pk4SMSlUV69uBXdFZ3HwInFXg=', 'Content-Length': '0'} 2023-05-24T21:11:36.240 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:59 GMT'), ('x-amz-version-id', 'g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009619a2fb01d1f30f-00646e7d88-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.241 DEBUG:boto:path=/qskrps-13/dummy 2023-05-24T21:11:36.241 DEBUG:boto:auth_path=/qskrps-13/dummy 2023-05-24T21:11:36.241 DEBUG:boto:path=/qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:11:36.242 DEBUG:boto:auth_path=/qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:11:36.242 DEBUG:boto:Method: GET 2023-05-24T21:11:36.242 DEBUG:boto:Path: /qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:11:36.243 DEBUG:boto:Data: 2023-05-24T21:11:36.243 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.243 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:36.244 DEBUG:boto:Port: 8000 2023-05-24T21:11:36.244 DEBUG:boto:Params: {} 2023-05-24T21:11:36.245 DEBUG:boto:Token: None 2023-05-24T21:11:36.245 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:36 GMT /qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:11:36.245 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:K1pk4SMSlUV69uBXdFZ3HwInFXg= 2023-05-24T21:11:36.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:K1pk4SMSlUV69uBXdFZ3HwInFXg=', 'Content-Length': '0'} 2023-05-24T21:11:36.247 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:59 GMT'), ('x-amz-version-id', 'g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d354f610107cb9e2-00646e7d88-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.248 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:36.249 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:36.249 DEBUG:boto:path=/qskrps-13/?prefix=dummy 2023-05-24T21:11:36.249 DEBUG:boto:auth_path=/qskrps-13/?prefix=dummy 2023-05-24T21:11:36.250 DEBUG:boto:Method: GET 2023-05-24T21:11:36.250 DEBUG:boto:Path: /qskrps-13/?prefix=dummy 2023-05-24T21:11:36.250 DEBUG:boto:Data: 2023-05-24T21:11:36.251 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.251 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:36.251 DEBUG:boto:Port: 8001 2023-05-24T21:11:36.252 DEBUG:boto:Params: {} 2023-05-24T21:11:36.252 DEBUG:boto:Token: None 2023-05-24T21:11:36.253 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:36 GMT /qskrps-13/ 2023-05-24T21:11:36.253 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:9f6nRfR1der32Nrg/FglWlhTnE8= 2023-05-24T21:11:36.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:9f6nRfR1der32Nrg/FglWlhTnE8=', 'Content-Length': '0'} 2023-05-24T21:11:36.256 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040807f6fe65c2d7c-00646e7d88-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.296 DEBUG:boto:b'qskrps-13dummy1000falsedummy2023-05-24T21:09:59.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:36.297 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:36.297 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:36.298 DEBUG:boto:path=/qskrps-13/?prefix=dummy 2023-05-24T21:11:36.298 DEBUG:boto:auth_path=/qskrps-13/?prefix=dummy 2023-05-24T21:11:36.298 DEBUG:boto:Method: GET 2023-05-24T21:11:36.299 DEBUG:boto:Path: /qskrps-13/?prefix=dummy 2023-05-24T21:11:36.299 DEBUG:boto:Data: 2023-05-24T21:11:36.299 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.300 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:36.300 DEBUG:boto:Port: 8000 2023-05-24T21:11:36.301 DEBUG:boto:Params: {} 2023-05-24T21:11:36.301 DEBUG:boto:Token: None 2023-05-24T21:11:36.301 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:36 GMT /qskrps-13/ 2023-05-24T21:11:36.302 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:9f6nRfR1der32Nrg/FglWlhTnE8= 2023-05-24T21:11:36.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:9f6nRfR1der32Nrg/FglWlhTnE8=', 'Content-Length': '0'} 2023-05-24T21:11:36.304 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025714455c8b62aea-00646e7d88-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.305 DEBUG:boto:b'qskrps-13dummy1000falsedummy2023-05-24T21:09:59.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:36.306 DEBUG:boto:path=/qskrps-13/dummy 2023-05-24T21:11:36.306 DEBUG:boto:auth_path=/qskrps-13/dummy 2023-05-24T21:11:36.306 DEBUG:boto:Method: HEAD 2023-05-24T21:11:36.307 DEBUG:boto:Path: /qskrps-13/dummy 2023-05-24T21:11:36.307 DEBUG:boto:Data: 2023-05-24T21:11:36.307 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.308 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:36.308 DEBUG:boto:Port: 8001 2023-05-24T21:11:36.308 DEBUG:boto:Params: {} 2023-05-24T21:11:36.309 DEBUG:boto:Token: None 2023-05-24T21:11:36.309 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:36 GMT /qskrps-13/dummy 2023-05-24T21:11:36.310 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:EZ05ag8x5gUFDzPFjHTCSDbN6EY= 2023-05-24T21:11:36.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:EZ05ag8x5gUFDzPFjHTCSDbN6EY=', 'Content-Length': '0'} 2023-05-24T21:11:36.312 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:59 GMT'), ('x-amz-version-id', 'g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009d9f40390de86a58-00646e7d88-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.313 DEBUG:boto:path=/qskrps-13/dummy 2023-05-24T21:11:36.313 DEBUG:boto:auth_path=/qskrps-13/dummy 2023-05-24T21:11:36.313 DEBUG:boto:Method: HEAD 2023-05-24T21:11:36.314 DEBUG:boto:Path: /qskrps-13/dummy 2023-05-24T21:11:36.314 DEBUG:boto:Data: 2023-05-24T21:11:36.314 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.315 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:36.315 DEBUG:boto:Port: 8000 2023-05-24T21:11:36.315 DEBUG:boto:Params: {} 2023-05-24T21:11:36.316 DEBUG:boto:Token: None 2023-05-24T21:11:36.316 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:36 GMT /qskrps-13/dummy 2023-05-24T21:11:36.317 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:EZ05ag8x5gUFDzPFjHTCSDbN6EY= 2023-05-24T21:11:36.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:EZ05ag8x5gUFDzPFjHTCSDbN6EY=', 'Content-Length': '0'} 2023-05-24T21:11:36.319 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:59 GMT'), ('x-amz-version-id', 'g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000049b37b76a610a5f2-00646e7d88-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-05-24T21:11:36.320 DEBUG:boto:path=/qskrps-13/dummy 2023-05-24T21:11:36.320 DEBUG:boto:auth_path=/qskrps-13/dummy 2023-05-24T21:11:36.321 DEBUG:boto:path=/qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:11:36.321 DEBUG:boto:auth_path=/qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:11:36.321 DEBUG:boto:Method: GET 2023-05-24T21:11:36.322 DEBUG:boto:Path: /qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:11:36.322 DEBUG:boto:Data: 2023-05-24T21:11:36.323 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.323 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:36.323 DEBUG:boto:Port: 8001 2023-05-24T21:11:36.324 DEBUG:boto:Params: {} 2023-05-24T21:11:36.324 DEBUG:boto:Token: None 2023-05-24T21:11:36.324 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:36 GMT /qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:11:36.325 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:K1pk4SMSlUV69uBXdFZ3HwInFXg= 2023-05-24T21:11:36.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:K1pk4SMSlUV69uBXdFZ3HwInFXg=', 'Content-Length': '0'} 2023-05-24T21:11:36.327 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:59 GMT'), ('x-amz-version-id', 'g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ca6c8ba98edb7b9e-00646e7d88-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.328 DEBUG:boto:path=/qskrps-13/dummy 2023-05-24T21:11:36.328 DEBUG:boto:auth_path=/qskrps-13/dummy 2023-05-24T21:11:36.328 DEBUG:boto:path=/qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:11:36.329 DEBUG:boto:auth_path=/qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:11:36.329 DEBUG:boto:Method: GET 2023-05-24T21:11:36.330 DEBUG:boto:Path: /qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:11:36.330 DEBUG:boto:Data: 2023-05-24T21:11:36.330 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.331 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:36.331 DEBUG:boto:Port: 8000 2023-05-24T21:11:36.331 DEBUG:boto:Params: {} 2023-05-24T21:11:36.332 DEBUG:boto:Token: None 2023-05-24T21:11:36.332 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:36 GMT /qskrps-13/dummy?versionId=g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX 2023-05-24T21:11:36.333 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:K1pk4SMSlUV69uBXdFZ3HwInFXg= 2023-05-24T21:11:36.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:K1pk4SMSlUV69uBXdFZ3HwInFXg=', 'Content-Length': '0'} 2023-05-24T21:11:36.335 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:09:59 GMT'), ('x-amz-version-id', 'g04CLEvxE5VlkW2A9PqVzMmjG-oLrNX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ef6dfe5b7ef820ad-00646e7d88-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.335 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:36.336 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:36.336 DEBUG:boto:path=/qskrps-13/?prefix=dummy 2023-05-24T21:11:36.337 DEBUG:boto:auth_path=/qskrps-13/?prefix=dummy 2023-05-24T21:11:36.337 DEBUG:boto:Method: GET 2023-05-24T21:11:36.337 DEBUG:boto:Path: /qskrps-13/?prefix=dummy 2023-05-24T21:11:36.338 DEBUG:boto:Data: 2023-05-24T21:11:36.338 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.338 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:36.339 DEBUG:boto:Port: 8001 2023-05-24T21:11:36.339 DEBUG:boto:Params: {} 2023-05-24T21:11:36.339 DEBUG:boto:Token: None 2023-05-24T21:11:36.340 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:36 GMT /qskrps-13/ 2023-05-24T21:11:36.340 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:9f6nRfR1der32Nrg/FglWlhTnE8= 2023-05-24T21:11:36.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:9f6nRfR1der32Nrg/FglWlhTnE8=', 'Content-Length': '0'} 2023-05-24T21:11:36.343 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000118aa96d86c855cb-00646e7d88-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.344 DEBUG:boto:b'qskrps-13dummy1000falsedummy2023-05-24T21:09:59.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:36.344 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:36.345 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:36.345 DEBUG:boto:path=/qskrps-13/?prefix=dummy 2023-05-24T21:11:36.345 DEBUG:boto:auth_path=/qskrps-13/?prefix=dummy 2023-05-24T21:11:36.346 DEBUG:boto:Method: GET 2023-05-24T21:11:36.346 DEBUG:boto:Path: /qskrps-13/?prefix=dummy 2023-05-24T21:11:36.347 DEBUG:boto:Data: 2023-05-24T21:11:36.347 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.347 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:36.348 DEBUG:boto:Port: 8000 2023-05-24T21:11:36.348 DEBUG:boto:Params: {} 2023-05-24T21:11:36.349 DEBUG:boto:Token: None 2023-05-24T21:11:36.349 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:36 GMT /qskrps-13/ 2023-05-24T21:11:36.349 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:9f6nRfR1der32Nrg/FglWlhTnE8= 2023-05-24T21:11:36.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:9f6nRfR1der32Nrg/FglWlhTnE8=', 'Content-Length': '0'} 2023-05-24T21:11:36.352 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce537f845e670ded-00646e7d88-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.396 DEBUG:boto:b'qskrps-13dummy1000falsedummy2023-05-24T21:09:59.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:36.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:36.398 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:36.398 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:36.398 DEBUG:boto:path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.399 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.399 DEBUG:boto:Method: GET 2023-05-24T21:11:36.400 DEBUG:boto:Path: /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.400 DEBUG:boto:Data: 2023-05-24T21:11:36.400 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.401 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:36.401 DEBUG:boto:Port: 8001 2023-05-24T21:11:36.401 DEBUG:boto:Params: {} 2023-05-24T21:11:36.402 DEBUG:boto:Token: None 2023-05-24T21:11:36.402 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:36 GMT /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.403 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:sJUzw+r5yxlszGWfjSDqoyLBzPM= 2023-05-24T21:11:36.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:sJUzw+r5yxlszGWfjSDqoyLBzPM=', 'Content-Length': '0'} 2023-05-24T21:11:36.406 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004c8873db72bc250c-00646e7d88-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.406 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:36.407 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:36.407 DEBUG:boto:path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.407 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.408 DEBUG:boto:Method: GET 2023-05-24T21:11:36.408 DEBUG:boto:Path: /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.409 DEBUG:boto:Data: 2023-05-24T21:11:36.409 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.409 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:36.410 DEBUG:boto:Port: 8000 2023-05-24T21:11:36.410 DEBUG:boto:Params: {} 2023-05-24T21:11:36.410 DEBUG:boto:Token: None 2023-05-24T21:11:36.411 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:36 GMT /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.411 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:sJUzw+r5yxlszGWfjSDqoyLBzPM= 2023-05-24T21:11:36.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:sJUzw+r5yxlszGWfjSDqoyLBzPM=', 'Content-Length': '0'} 2023-05-24T21:11:36.413 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000064141e1e393bbd91-00646e7d88-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.414 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:36.415 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:36.415 DEBUG:boto:path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.415 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.416 DEBUG:boto:Method: HEAD 2023-05-24T21:11:36.416 DEBUG:boto:Path: /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.416 DEBUG:boto:Data: 2023-05-24T21:11:36.417 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.417 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:36.417 DEBUG:boto:Port: 8001 2023-05-24T21:11:36.418 DEBUG:boto:Params: {} 2023-05-24T21:11:36.418 DEBUG:boto:Token: None 2023-05-24T21:11:36.418 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:36 GMT /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.419 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:njxXP7DlxRoTIFITIiN07COWMD8= 2023-05-24T21:11:36.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:njxXP7DlxRoTIFITIiN07COWMD8=', 'Content-Length': '0'} 2023-05-24T21:11:36.421 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a58781c5309608ec-00646e7d88-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.422 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:36.422 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:36.422 DEBUG:boto:path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.423 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.423 DEBUG:boto:Method: HEAD 2023-05-24T21:11:36.423 DEBUG:boto:Path: /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.424 DEBUG:boto:Data: 2023-05-24T21:11:36.424 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.424 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:36.425 DEBUG:boto:Port: 8000 2023-05-24T21:11:36.425 DEBUG:boto:Params: {} 2023-05-24T21:11:36.425 DEBUG:boto:Token: None 2023-05-24T21:11:36.426 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:36 GMT /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.426 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:njxXP7DlxRoTIFITIiN07COWMD8= 2023-05-24T21:11:36.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:njxXP7DlxRoTIFITIiN07COWMD8=', 'Content-Length': '0'} 2023-05-24T21:11:36.428 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e201f85856aa8b65-00646e7d88-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:36.429 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:36.430 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:36.430 DEBUG:boto:path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.430 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.431 DEBUG:boto:Method: GET 2023-05-24T21:11:36.431 DEBUG:boto:Path: /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.431 DEBUG:boto:Data: 2023-05-24T21:11:36.432 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.432 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:36.432 DEBUG:boto:Port: 8001 2023-05-24T21:11:36.433 DEBUG:boto:Params: {} 2023-05-24T21:11:36.433 DEBUG:boto:Token: None 2023-05-24T21:11:36.434 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:36 GMT /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.434 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:sJUzw+r5yxlszGWfjSDqoyLBzPM= 2023-05-24T21:11:36.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:sJUzw+r5yxlszGWfjSDqoyLBzPM=', 'Content-Length': '0'} 2023-05-24T21:11:36.436 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000089a2325fc8abc6d2-00646e7d88-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.480 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:36.481 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:36.481 DEBUG:boto:path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.482 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.482 DEBUG:boto:Method: GET 2023-05-24T21:11:36.482 DEBUG:boto:Path: /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.483 DEBUG:boto:Data: 2023-05-24T21:11:36.483 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.483 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:36.484 DEBUG:boto:Port: 8000 2023-05-24T21:11:36.484 DEBUG:boto:Params: {} 2023-05-24T21:11:36.485 DEBUG:boto:Token: None 2023-05-24T21:11:36.485 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:36 GMT /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.485 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:sJUzw+r5yxlszGWfjSDqoyLBzPM= 2023-05-24T21:11:36.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:sJUzw+r5yxlszGWfjSDqoyLBzPM=', 'Content-Length': '0'} 2023-05-24T21:11:36.488 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e71a67295086e20-00646e7d88-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.489 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:36.489 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:36.489 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:36.490 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:36.490 DEBUG:boto:Method: GET 2023-05-24T21:11:36.490 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:36.491 DEBUG:boto:Data: 2023-05-24T21:11:36.491 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.491 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:36.492 DEBUG:boto:Port: 8001 2023-05-24T21:11:36.492 DEBUG:boto:Params: {} 2023-05-24T21:11:36.493 DEBUG:boto:Token: None 2023-05-24T21:11:36.493 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:36 GMT /qskrps-13/ 2023-05-24T21:11:36.493 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:9f6nRfR1der32Nrg/FglWlhTnE8= 2023-05-24T21:11:36.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:9f6nRfR1der32Nrg/FglWlhTnE8=', 'Content-Length': '0'} 2023-05-24T21:11:36.497 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a569040d0d70899d-00646e7d88-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.497 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:36.498 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:36.498 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:36.499 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:36.499 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:36.499 DEBUG:boto:Method: GET 2023-05-24T21:11:36.500 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:36.500 DEBUG:boto:Data: 2023-05-24T21:11:36.500 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.501 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:36.501 DEBUG:boto:Port: 8000 2023-05-24T21:11:36.501 DEBUG:boto:Params: {} 2023-05-24T21:11:36.502 DEBUG:boto:Token: None 2023-05-24T21:11:36.502 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:36 GMT /qskrps-13/ 2023-05-24T21:11:36.503 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:9f6nRfR1der32Nrg/FglWlhTnE8= 2023-05-24T21:11:36.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:9f6nRfR1der32Nrg/FglWlhTnE8=', 'Content-Length': '0'} 2023-05-24T21:11:36.506 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bc2b8756aae330f-00646e7d88-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.506 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:36.507 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:36.507 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:36.508 DEBUG:boto:Method: HEAD 2023-05-24T21:11:36.508 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:11:36.508 DEBUG:boto:Data: 2023-05-24T21:11:36.509 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.509 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:36.509 DEBUG:boto:Port: 8001 2023-05-24T21:11:36.510 DEBUG:boto:Params: {} 2023-05-24T21:11:36.510 DEBUG:boto:Token: None 2023-05-24T21:11:36.510 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:36 GMT /qskrps-13/obj 2023-05-24T21:11:36.511 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:QpeXDiVOdYqoPMJTuV41OUkFxp0= 2023-05-24T21:11:36.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:QpeXDiVOdYqoPMJTuV41OUkFxp0=', 'Content-Length': '0'} 2023-05-24T21:11:36.514 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000023fd0d76e50a2ff6-00646e7d88-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.515 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:36.515 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:36.515 DEBUG:boto:Method: HEAD 2023-05-24T21:11:36.516 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:11:36.516 DEBUG:boto:Data: 2023-05-24T21:11:36.516 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.517 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:36.517 DEBUG:boto:Port: 8000 2023-05-24T21:11:36.517 DEBUG:boto:Params: {} 2023-05-24T21:11:36.518 DEBUG:boto:Token: None 2023-05-24T21:11:36.518 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:36 GMT /qskrps-13/obj 2023-05-24T21:11:36.518 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:QpeXDiVOdYqoPMJTuV41OUkFxp0= 2023-05-24T21:11:36.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:QpeXDiVOdYqoPMJTuV41OUkFxp0=', 'Content-Length': '0'} 2023-05-24T21:11:36.522 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e2955c916d8eaf76-00646e7d88-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:36.523 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:36.523 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:36.523 DEBUG:boto:path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.524 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.524 DEBUG:boto:Method: GET 2023-05-24T21:11:36.524 DEBUG:boto:Path: /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.525 DEBUG:boto:Data: 2023-05-24T21:11:36.525 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.526 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:36.526 DEBUG:boto:Port: 8001 2023-05-24T21:11:36.526 DEBUG:boto:Params: {} 2023-05-24T21:11:36.527 DEBUG:boto:Token: None 2023-05-24T21:11:36.527 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:36 GMT /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.528 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:sJUzw+r5yxlszGWfjSDqoyLBzPM= 2023-05-24T21:11:36.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:sJUzw+r5yxlszGWfjSDqoyLBzPM=', 'Content-Length': '0'} 2023-05-24T21:11:36.530 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002db72b57f22141a0-00646e7d88-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.572 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:36.573 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:36.573 DEBUG:boto:path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.574 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.574 DEBUG:boto:Method: GET 2023-05-24T21:11:36.574 DEBUG:boto:Path: /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.575 DEBUG:boto:Data: 2023-05-24T21:11:36.575 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.575 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:36.576 DEBUG:boto:Port: 8000 2023-05-24T21:11:36.576 DEBUG:boto:Params: {} 2023-05-24T21:11:36.576 DEBUG:boto:Token: None 2023-05-24T21:11:36.577 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:36 GMT /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.577 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:sJUzw+r5yxlszGWfjSDqoyLBzPM= 2023-05-24T21:11:36.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:sJUzw+r5yxlszGWfjSDqoyLBzPM=', 'Content-Length': '0'} 2023-05-24T21:11:36.580 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a141ec88d91aeb8d-00646e7d88-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.580 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:36.581 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:36.581 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:36.582 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:36.582 DEBUG:boto:Method: GET 2023-05-24T21:11:36.582 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:36.583 DEBUG:boto:Data: 2023-05-24T21:11:36.583 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.583 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:36.584 DEBUG:boto:Port: 8001 2023-05-24T21:11:36.584 DEBUG:boto:Params: {} 2023-05-24T21:11:36.584 DEBUG:boto:Token: None 2023-05-24T21:11:36.585 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:36 GMT /qskrps-13/ 2023-05-24T21:11:36.585 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:9f6nRfR1der32Nrg/FglWlhTnE8= 2023-05-24T21:11:36.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:9f6nRfR1der32Nrg/FglWlhTnE8=', 'Content-Length': '0'} 2023-05-24T21:11:36.588 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000259a1fed3ce42b5f-00646e7d88-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.589 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:36.590 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:36.590 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:36.590 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:36.591 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:36.591 DEBUG:boto:Method: GET 2023-05-24T21:11:36.591 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:36.592 DEBUG:boto:Data: 2023-05-24T21:11:36.592 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.593 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:36.593 DEBUG:boto:Port: 8000 2023-05-24T21:11:36.593 DEBUG:boto:Params: {} 2023-05-24T21:11:36.594 DEBUG:boto:Token: None 2023-05-24T21:11:36.594 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:36 GMT /qskrps-13/ 2023-05-24T21:11:36.595 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:9f6nRfR1der32Nrg/FglWlhTnE8= 2023-05-24T21:11:36.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:9f6nRfR1der32Nrg/FglWlhTnE8=', 'Content-Length': '0'} 2023-05-24T21:11:36.598 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8a6d84ea0ed6c5f-00646e7d88-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.598 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:36.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:36.599 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:36.600 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:36.600 DEBUG:boto:path=/qskrps-13/obj?versionId=jqoKobaDdC.ssJkHV2IlgFRFcAWxTE9 2023-05-24T21:11:36.600 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=jqoKobaDdC.ssJkHV2IlgFRFcAWxTE9 2023-05-24T21:11:36.601 DEBUG:boto:Method: GET 2023-05-24T21:11:36.601 DEBUG:boto:Path: /qskrps-13/obj?versionId=jqoKobaDdC.ssJkHV2IlgFRFcAWxTE9 2023-05-24T21:11:36.601 DEBUG:boto:Data: 2023-05-24T21:11:36.602 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.602 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:36.603 DEBUG:boto:Port: 8001 2023-05-24T21:11:36.603 DEBUG:boto:Params: {} 2023-05-24T21:11:36.603 DEBUG:boto:Token: None 2023-05-24T21:11:36.604 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:36 GMT /qskrps-13/obj?versionId=jqoKobaDdC.ssJkHV2IlgFRFcAWxTE9 2023-05-24T21:11:36.604 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:7tfZoTpdTt34HqmH5SYPirxdBK0= 2023-05-24T21:11:36.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:7tfZoTpdTt34HqmH5SYPirxdBK0=', 'Content-Length': '0'} 2023-05-24T21:11:36.606 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'jqoKobaDdC.ssJkHV2IlgFRFcAWxTE9'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000509ad65a08735d4a-00646e7d88-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.607 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:36.607 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:36.608 DEBUG:boto:path=/qskrps-13/obj?versionId=jqoKobaDdC.ssJkHV2IlgFRFcAWxTE9 2023-05-24T21:11:36.608 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=jqoKobaDdC.ssJkHV2IlgFRFcAWxTE9 2023-05-24T21:11:36.608 DEBUG:boto:Method: GET 2023-05-24T21:11:36.609 DEBUG:boto:Path: /qskrps-13/obj?versionId=jqoKobaDdC.ssJkHV2IlgFRFcAWxTE9 2023-05-24T21:11:36.609 DEBUG:boto:Data: 2023-05-24T21:11:36.610 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.610 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:36.610 DEBUG:boto:Port: 8000 2023-05-24T21:11:36.611 DEBUG:boto:Params: {} 2023-05-24T21:11:36.611 DEBUG:boto:Token: None 2023-05-24T21:11:36.611 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:36 GMT /qskrps-13/obj?versionId=jqoKobaDdC.ssJkHV2IlgFRFcAWxTE9 2023-05-24T21:11:36.612 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:7tfZoTpdTt34HqmH5SYPirxdBK0= 2023-05-24T21:11:36.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:7tfZoTpdTt34HqmH5SYPirxdBK0=', 'Content-Length': '0'} 2023-05-24T21:11:36.614 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'jqoKobaDdC.ssJkHV2IlgFRFcAWxTE9'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d68e0f6936f6dd90-00646e7d88-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.656 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:36.657 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:36.658 DEBUG:boto:path=/qskrps-13/obj?versionId=jqoKobaDdC.ssJkHV2IlgFRFcAWxTE9 2023-05-24T21:11:36.658 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=jqoKobaDdC.ssJkHV2IlgFRFcAWxTE9 2023-05-24T21:11:36.658 DEBUG:boto:Method: HEAD 2023-05-24T21:11:36.659 DEBUG:boto:Path: /qskrps-13/obj?versionId=jqoKobaDdC.ssJkHV2IlgFRFcAWxTE9 2023-05-24T21:11:36.659 DEBUG:boto:Data: 2023-05-24T21:11:36.660 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.660 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:36.660 DEBUG:boto:Port: 8001 2023-05-24T21:11:36.661 DEBUG:boto:Params: {} 2023-05-24T21:11:36.661 DEBUG:boto:Token: None 2023-05-24T21:11:36.662 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:36 GMT /qskrps-13/obj?versionId=jqoKobaDdC.ssJkHV2IlgFRFcAWxTE9 2023-05-24T21:11:36.662 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:E/EB3SMSfUtntGuM5A2bY1gP0+o= 2023-05-24T21:11:36.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:E/EB3SMSfUtntGuM5A2bY1gP0+o=', 'Content-Length': '0'} 2023-05-24T21:11:36.665 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'jqoKobaDdC.ssJkHV2IlgFRFcAWxTE9'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001c7b586759e4e348-00646e7d88-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.665 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:36.666 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:36.666 DEBUG:boto:path=/qskrps-13/obj?versionId=jqoKobaDdC.ssJkHV2IlgFRFcAWxTE9 2023-05-24T21:11:36.666 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=jqoKobaDdC.ssJkHV2IlgFRFcAWxTE9 2023-05-24T21:11:36.667 DEBUG:boto:Method: HEAD 2023-05-24T21:11:36.667 DEBUG:boto:Path: /qskrps-13/obj?versionId=jqoKobaDdC.ssJkHV2IlgFRFcAWxTE9 2023-05-24T21:11:36.667 DEBUG:boto:Data: 2023-05-24T21:11:36.668 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.668 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:36.668 DEBUG:boto:Port: 8000 2023-05-24T21:11:36.669 DEBUG:boto:Params: {} 2023-05-24T21:11:36.669 DEBUG:boto:Token: None 2023-05-24T21:11:36.669 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:36 GMT /qskrps-13/obj?versionId=jqoKobaDdC.ssJkHV2IlgFRFcAWxTE9 2023-05-24T21:11:36.670 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:E/EB3SMSfUtntGuM5A2bY1gP0+o= 2023-05-24T21:11:36.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:E/EB3SMSfUtntGuM5A2bY1gP0+o=', 'Content-Length': '0'} 2023-05-24T21:11:36.672 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'jqoKobaDdC.ssJkHV2IlgFRFcAWxTE9'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d454c9c143c3c0b8-00646e7d88-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:36.673 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:36.674 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:36.674 DEBUG:boto:path=/qskrps-13/obj?versionId=jqoKobaDdC.ssJkHV2IlgFRFcAWxTE9 2023-05-24T21:11:36.674 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=jqoKobaDdC.ssJkHV2IlgFRFcAWxTE9 2023-05-24T21:11:36.675 DEBUG:boto:Method: GET 2023-05-24T21:11:36.675 DEBUG:boto:Path: /qskrps-13/obj?versionId=jqoKobaDdC.ssJkHV2IlgFRFcAWxTE9 2023-05-24T21:11:36.675 DEBUG:boto:Data: 2023-05-24T21:11:36.676 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.676 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:36.677 DEBUG:boto:Port: 8001 2023-05-24T21:11:36.677 DEBUG:boto:Params: {} 2023-05-24T21:11:36.677 DEBUG:boto:Token: None 2023-05-24T21:11:36.678 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:36 GMT /qskrps-13/obj?versionId=jqoKobaDdC.ssJkHV2IlgFRFcAWxTE9 2023-05-24T21:11:36.678 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:7tfZoTpdTt34HqmH5SYPirxdBK0= 2023-05-24T21:11:36.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:7tfZoTpdTt34HqmH5SYPirxdBK0=', 'Content-Length': '0'} 2023-05-24T21:11:36.680 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'jqoKobaDdC.ssJkHV2IlgFRFcAWxTE9'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000048d87d5bd6bca88-00646e7d88-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.724 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:36.725 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:36.725 DEBUG:boto:path=/qskrps-13/obj?versionId=jqoKobaDdC.ssJkHV2IlgFRFcAWxTE9 2023-05-24T21:11:36.726 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=jqoKobaDdC.ssJkHV2IlgFRFcAWxTE9 2023-05-24T21:11:36.726 DEBUG:boto:Method: GET 2023-05-24T21:11:36.726 DEBUG:boto:Path: /qskrps-13/obj?versionId=jqoKobaDdC.ssJkHV2IlgFRFcAWxTE9 2023-05-24T21:11:36.727 DEBUG:boto:Data: 2023-05-24T21:11:36.727 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.728 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:36.728 DEBUG:boto:Port: 8000 2023-05-24T21:11:36.728 DEBUG:boto:Params: {} 2023-05-24T21:11:36.729 DEBUG:boto:Token: None 2023-05-24T21:11:36.729 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:36 GMT /qskrps-13/obj?versionId=jqoKobaDdC.ssJkHV2IlgFRFcAWxTE9 2023-05-24T21:11:36.730 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:7tfZoTpdTt34HqmH5SYPirxdBK0= 2023-05-24T21:11:36.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:7tfZoTpdTt34HqmH5SYPirxdBK0=', 'Content-Length': '0'} 2023-05-24T21:11:36.732 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'jqoKobaDdC.ssJkHV2IlgFRFcAWxTE9'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000088ef03d864113e15-00646e7d88-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.733 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:36.734 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:36.734 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:36.734 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:36.735 DEBUG:boto:Method: GET 2023-05-24T21:11:36.735 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:36.735 DEBUG:boto:Data: 2023-05-24T21:11:36.736 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.736 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:36.737 DEBUG:boto:Port: 8001 2023-05-24T21:11:36.737 DEBUG:boto:Params: {} 2023-05-24T21:11:36.737 DEBUG:boto:Token: None 2023-05-24T21:11:36.738 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:36 GMT /qskrps-13/ 2023-05-24T21:11:36.738 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:9f6nRfR1der32Nrg/FglWlhTnE8= 2023-05-24T21:11:36.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:9f6nRfR1der32Nrg/FglWlhTnE8=', 'Content-Length': '0'} 2023-05-24T21:11:36.741 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b705fb51775822fb-00646e7d88-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.742 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:36.742 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:36.743 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:36.743 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:36.744 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:36.744 DEBUG:boto:Method: GET 2023-05-24T21:11:36.744 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:36.745 DEBUG:boto:Data: 2023-05-24T21:11:36.745 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.746 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:36.746 DEBUG:boto:Port: 8000 2023-05-24T21:11:36.746 DEBUG:boto:Params: {} 2023-05-24T21:11:36.747 DEBUG:boto:Token: None 2023-05-24T21:11:36.747 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:36 GMT /qskrps-13/ 2023-05-24T21:11:36.748 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:9f6nRfR1der32Nrg/FglWlhTnE8= 2023-05-24T21:11:36.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:9f6nRfR1der32Nrg/FglWlhTnE8=', 'Content-Length': '0'} 2023-05-24T21:11:36.751 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000073e1bae601e535f-00646e7d88-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.751 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:36.752 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:36.752 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:36.753 DEBUG:boto:Method: HEAD 2023-05-24T21:11:36.753 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:11:36.753 DEBUG:boto:Data: 2023-05-24T21:11:36.754 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.754 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:36.755 DEBUG:boto:Port: 8001 2023-05-24T21:11:36.755 DEBUG:boto:Params: {} 2023-05-24T21:11:36.755 DEBUG:boto:Token: None 2023-05-24T21:11:36.756 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:36 GMT /qskrps-13/obj 2023-05-24T21:11:36.756 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:QpeXDiVOdYqoPMJTuV41OUkFxp0= 2023-05-24T21:11:36.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:QpeXDiVOdYqoPMJTuV41OUkFxp0=', 'Content-Length': '0'} 2023-05-24T21:11:36.759 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000153c68198cbb7b4d-00646e7d88-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.760 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:36.761 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:36.761 DEBUG:boto:Method: HEAD 2023-05-24T21:11:36.761 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:11:36.762 DEBUG:boto:Data: 2023-05-24T21:11:36.762 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.762 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:36.763 DEBUG:boto:Port: 8000 2023-05-24T21:11:36.763 DEBUG:boto:Params: {} 2023-05-24T21:11:36.763 DEBUG:boto:Token: None 2023-05-24T21:11:36.764 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:36 GMT /qskrps-13/obj 2023-05-24T21:11:36.764 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:QpeXDiVOdYqoPMJTuV41OUkFxp0= 2023-05-24T21:11:36.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:QpeXDiVOdYqoPMJTuV41OUkFxp0=', 'Content-Length': '0'} 2023-05-24T21:11:36.768 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cd0b9c77b66fc3da-00646e7d88-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:36.769 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:36.769 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:36.769 DEBUG:boto:path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.770 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.770 DEBUG:boto:Method: GET 2023-05-24T21:11:36.770 DEBUG:boto:Path: /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.771 DEBUG:boto:Data: 2023-05-24T21:11:36.771 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.772 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:36.772 DEBUG:boto:Port: 8001 2023-05-24T21:11:36.772 DEBUG:boto:Params: {} 2023-05-24T21:11:36.773 DEBUG:boto:Token: None 2023-05-24T21:11:36.773 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:36 GMT /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.774 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:sJUzw+r5yxlszGWfjSDqoyLBzPM= 2023-05-24T21:11:36.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:sJUzw+r5yxlszGWfjSDqoyLBzPM=', 'Content-Length': '0'} 2023-05-24T21:11:36.776 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000097f4b331a470c9dd-00646e7d88-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.776 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:36.777 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:36.777 DEBUG:boto:path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.778 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.778 DEBUG:boto:Method: GET 2023-05-24T21:11:36.778 DEBUG:boto:Path: /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.779 DEBUG:boto:Data: 2023-05-24T21:11:36.779 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.779 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:36.780 DEBUG:boto:Port: 8000 2023-05-24T21:11:36.780 DEBUG:boto:Params: {} 2023-05-24T21:11:36.780 DEBUG:boto:Token: None 2023-05-24T21:11:36.781 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:36 GMT /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.781 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:sJUzw+r5yxlszGWfjSDqoyLBzPM= 2023-05-24T21:11:36.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:sJUzw+r5yxlszGWfjSDqoyLBzPM=', 'Content-Length': '0'} 2023-05-24T21:11:36.783 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dac2aa7ed987b697-00646e7d88-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.784 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:36.784 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:36.785 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:36.785 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:36.786 DEBUG:boto:Method: GET 2023-05-24T21:11:36.786 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:36.786 DEBUG:boto:Data: 2023-05-24T21:11:36.787 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.787 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:36.787 DEBUG:boto:Port: 8001 2023-05-24T21:11:36.788 DEBUG:boto:Params: {} 2023-05-24T21:11:36.788 DEBUG:boto:Token: None 2023-05-24T21:11:36.788 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:36 GMT /qskrps-13/ 2023-05-24T21:11:36.789 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:9f6nRfR1der32Nrg/FglWlhTnE8= 2023-05-24T21:11:36.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:9f6nRfR1der32Nrg/FglWlhTnE8=', 'Content-Length': '0'} 2023-05-24T21:11:36.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009bdc67a6d6c5a8d-00646e7d88-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.792 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:36.793 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:36.793 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:36.794 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:36.794 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:36.794 DEBUG:boto:Method: GET 2023-05-24T21:11:36.795 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:36.795 DEBUG:boto:Data: 2023-05-24T21:11:36.795 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.796 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:36.796 DEBUG:boto:Port: 8000 2023-05-24T21:11:36.797 DEBUG:boto:Params: {} 2023-05-24T21:11:36.797 DEBUG:boto:Token: None 2023-05-24T21:11:36.797 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:36 GMT /qskrps-13/ 2023-05-24T21:11:36.798 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:9f6nRfR1der32Nrg/FglWlhTnE8= 2023-05-24T21:11:36.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:9f6nRfR1der32Nrg/FglWlhTnE8=', 'Content-Length': '0'} 2023-05-24T21:11:36.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f14f649f61747e43-00646e7d88-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.801 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:36.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:36.802 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:36.803 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:36.803 DEBUG:boto:path=/qskrps-13/obj?versionId=RsY7UgV2wkSvgE1I0nN3nCJ0pjKqhZX 2023-05-24T21:11:36.803 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=RsY7UgV2wkSvgE1I0nN3nCJ0pjKqhZX 2023-05-24T21:11:36.804 DEBUG:boto:Method: GET 2023-05-24T21:11:36.804 DEBUG:boto:Path: /qskrps-13/obj?versionId=RsY7UgV2wkSvgE1I0nN3nCJ0pjKqhZX 2023-05-24T21:11:36.804 DEBUG:boto:Data: 2023-05-24T21:11:36.805 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.805 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:36.806 DEBUG:boto:Port: 8001 2023-05-24T21:11:36.806 DEBUG:boto:Params: {} 2023-05-24T21:11:36.806 DEBUG:boto:Token: None 2023-05-24T21:11:36.807 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:36 GMT /qskrps-13/obj?versionId=RsY7UgV2wkSvgE1I0nN3nCJ0pjKqhZX 2023-05-24T21:11:36.807 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:3sJeSRipEu1eLQ9rLUYVPzg1k9o= 2023-05-24T21:11:36.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:3sJeSRipEu1eLQ9rLUYVPzg1k9o=', 'Content-Length': '0'} 2023-05-24T21:11:36.809 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'RsY7UgV2wkSvgE1I0nN3nCJ0pjKqhZX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000329167e30631e7c8-00646e7d88-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.810 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:36.810 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:36.811 DEBUG:boto:path=/qskrps-13/obj?versionId=RsY7UgV2wkSvgE1I0nN3nCJ0pjKqhZX 2023-05-24T21:11:36.811 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=RsY7UgV2wkSvgE1I0nN3nCJ0pjKqhZX 2023-05-24T21:11:36.812 DEBUG:boto:Method: GET 2023-05-24T21:11:36.812 DEBUG:boto:Path: /qskrps-13/obj?versionId=RsY7UgV2wkSvgE1I0nN3nCJ0pjKqhZX 2023-05-24T21:11:36.812 DEBUG:boto:Data: 2023-05-24T21:11:36.813 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.813 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:36.813 DEBUG:boto:Port: 8000 2023-05-24T21:11:36.814 DEBUG:boto:Params: {} 2023-05-24T21:11:36.814 DEBUG:boto:Token: None 2023-05-24T21:11:36.815 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:36 GMT /qskrps-13/obj?versionId=RsY7UgV2wkSvgE1I0nN3nCJ0pjKqhZX 2023-05-24T21:11:36.815 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:3sJeSRipEu1eLQ9rLUYVPzg1k9o= 2023-05-24T21:11:36.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:3sJeSRipEu1eLQ9rLUYVPzg1k9o=', 'Content-Length': '0'} 2023-05-24T21:11:36.817 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'RsY7UgV2wkSvgE1I0nN3nCJ0pjKqhZX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ec9661f2579b2c69-00646e7d88-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.860 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:36.861 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:36.861 DEBUG:boto:path=/qskrps-13/obj?versionId=RsY7UgV2wkSvgE1I0nN3nCJ0pjKqhZX 2023-05-24T21:11:36.862 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=RsY7UgV2wkSvgE1I0nN3nCJ0pjKqhZX 2023-05-24T21:11:36.862 DEBUG:boto:Method: HEAD 2023-05-24T21:11:36.862 DEBUG:boto:Path: /qskrps-13/obj?versionId=RsY7UgV2wkSvgE1I0nN3nCJ0pjKqhZX 2023-05-24T21:11:36.863 DEBUG:boto:Data: 2023-05-24T21:11:36.863 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.863 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:36.864 DEBUG:boto:Port: 8001 2023-05-24T21:11:36.864 DEBUG:boto:Params: {} 2023-05-24T21:11:36.865 DEBUG:boto:Token: None 2023-05-24T21:11:36.865 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:36 GMT /qskrps-13/obj?versionId=RsY7UgV2wkSvgE1I0nN3nCJ0pjKqhZX 2023-05-24T21:11:36.865 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:UZnMDkTOBEu/fdONBLc6u/DltVc= 2023-05-24T21:11:36.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:UZnMDkTOBEu/fdONBLc6u/DltVc=', 'Content-Length': '0'} 2023-05-24T21:11:36.868 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'RsY7UgV2wkSvgE1I0nN3nCJ0pjKqhZX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000038111a5e7570b79f-00646e7d88-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.868 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:36.869 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:36.869 DEBUG:boto:path=/qskrps-13/obj?versionId=RsY7UgV2wkSvgE1I0nN3nCJ0pjKqhZX 2023-05-24T21:11:36.870 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=RsY7UgV2wkSvgE1I0nN3nCJ0pjKqhZX 2023-05-24T21:11:36.870 DEBUG:boto:Method: HEAD 2023-05-24T21:11:36.870 DEBUG:boto:Path: /qskrps-13/obj?versionId=RsY7UgV2wkSvgE1I0nN3nCJ0pjKqhZX 2023-05-24T21:11:36.871 DEBUG:boto:Data: 2023-05-24T21:11:36.871 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.871 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:36.872 DEBUG:boto:Port: 8000 2023-05-24T21:11:36.872 DEBUG:boto:Params: {} 2023-05-24T21:11:36.872 DEBUG:boto:Token: None 2023-05-24T21:11:36.873 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:36 GMT /qskrps-13/obj?versionId=RsY7UgV2wkSvgE1I0nN3nCJ0pjKqhZX 2023-05-24T21:11:36.873 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:UZnMDkTOBEu/fdONBLc6u/DltVc= 2023-05-24T21:11:36.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:UZnMDkTOBEu/fdONBLc6u/DltVc=', 'Content-Length': '0'} 2023-05-24T21:11:36.875 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'RsY7UgV2wkSvgE1I0nN3nCJ0pjKqhZX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000549a5933704480c3-00646e7d88-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:36.876 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:36.877 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:36.877 DEBUG:boto:path=/qskrps-13/obj?versionId=RsY7UgV2wkSvgE1I0nN3nCJ0pjKqhZX 2023-05-24T21:11:36.877 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=RsY7UgV2wkSvgE1I0nN3nCJ0pjKqhZX 2023-05-24T21:11:36.878 DEBUG:boto:Method: GET 2023-05-24T21:11:36.878 DEBUG:boto:Path: /qskrps-13/obj?versionId=RsY7UgV2wkSvgE1I0nN3nCJ0pjKqhZX 2023-05-24T21:11:36.878 DEBUG:boto:Data: 2023-05-24T21:11:36.879 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.879 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:36.879 DEBUG:boto:Port: 8001 2023-05-24T21:11:36.880 DEBUG:boto:Params: {} 2023-05-24T21:11:36.880 DEBUG:boto:Token: None 2023-05-24T21:11:36.881 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:36 GMT /qskrps-13/obj?versionId=RsY7UgV2wkSvgE1I0nN3nCJ0pjKqhZX 2023-05-24T21:11:36.881 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:3sJeSRipEu1eLQ9rLUYVPzg1k9o= 2023-05-24T21:11:36.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:3sJeSRipEu1eLQ9rLUYVPzg1k9o=', 'Content-Length': '0'} 2023-05-24T21:11:36.883 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'RsY7UgV2wkSvgE1I0nN3nCJ0pjKqhZX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008de3172e38839a60-00646e7d88-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.884 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:36.884 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:36.885 DEBUG:boto:path=/qskrps-13/obj?versionId=RsY7UgV2wkSvgE1I0nN3nCJ0pjKqhZX 2023-05-24T21:11:36.885 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=RsY7UgV2wkSvgE1I0nN3nCJ0pjKqhZX 2023-05-24T21:11:36.886 DEBUG:boto:Method: GET 2023-05-24T21:11:36.886 DEBUG:boto:Path: /qskrps-13/obj?versionId=RsY7UgV2wkSvgE1I0nN3nCJ0pjKqhZX 2023-05-24T21:11:36.886 DEBUG:boto:Data: 2023-05-24T21:11:36.887 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.887 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:36.887 DEBUG:boto:Port: 8000 2023-05-24T21:11:36.888 DEBUG:boto:Params: {} 2023-05-24T21:11:36.888 DEBUG:boto:Token: None 2023-05-24T21:11:36.889 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:36 GMT /qskrps-13/obj?versionId=RsY7UgV2wkSvgE1I0nN3nCJ0pjKqhZX 2023-05-24T21:11:36.889 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:3sJeSRipEu1eLQ9rLUYVPzg1k9o= 2023-05-24T21:11:36.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:3sJeSRipEu1eLQ9rLUYVPzg1k9o=', 'Content-Length': '0'} 2023-05-24T21:11:36.891 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'RsY7UgV2wkSvgE1I0nN3nCJ0pjKqhZX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000025e744ec4d1f5a64-00646e7d88-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.892 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:36.892 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:36.893 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:36.893 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:36.893 DEBUG:boto:Method: GET 2023-05-24T21:11:36.894 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:36.894 DEBUG:boto:Data: 2023-05-24T21:11:36.894 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.895 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:36.895 DEBUG:boto:Port: 8001 2023-05-24T21:11:36.895 DEBUG:boto:Params: {} 2023-05-24T21:11:36.896 DEBUG:boto:Token: None 2023-05-24T21:11:36.896 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:36 GMT /qskrps-13/ 2023-05-24T21:11:36.897 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:9f6nRfR1der32Nrg/FglWlhTnE8= 2023-05-24T21:11:36.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:9f6nRfR1der32Nrg/FglWlhTnE8=', 'Content-Length': '0'} 2023-05-24T21:11:36.899 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008af262da32a87b14-00646e7d88-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.900 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:36.901 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:36.901 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:36.901 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:36.902 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:36.902 DEBUG:boto:Method: GET 2023-05-24T21:11:36.903 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:36.903 DEBUG:boto:Data: 2023-05-24T21:11:36.903 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.904 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:36.904 DEBUG:boto:Port: 8000 2023-05-24T21:11:36.904 DEBUG:boto:Params: {} 2023-05-24T21:11:36.905 DEBUG:boto:Token: None 2023-05-24T21:11:36.905 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:36 GMT /qskrps-13/ 2023-05-24T21:11:36.906 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:9f6nRfR1der32Nrg/FglWlhTnE8= 2023-05-24T21:11:36.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:9f6nRfR1der32Nrg/FglWlhTnE8=', 'Content-Length': '0'} 2023-05-24T21:11:36.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa02d370b707c43f-00646e7d88-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.909 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:36.910 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:36.910 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:36.911 DEBUG:boto:Method: HEAD 2023-05-24T21:11:36.911 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:11:36.911 DEBUG:boto:Data: 2023-05-24T21:11:36.912 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.912 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:36.912 DEBUG:boto:Port: 8001 2023-05-24T21:11:36.913 DEBUG:boto:Params: {} 2023-05-24T21:11:36.913 DEBUG:boto:Token: None 2023-05-24T21:11:36.913 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:36 GMT /qskrps-13/obj 2023-05-24T21:11:36.914 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:QpeXDiVOdYqoPMJTuV41OUkFxp0= 2023-05-24T21:11:36.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:QpeXDiVOdYqoPMJTuV41OUkFxp0=', 'Content-Length': '0'} 2023-05-24T21:11:36.917 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004ef92899d3cde21e-00646e7d88-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.918 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:36.918 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:36.919 DEBUG:boto:Method: HEAD 2023-05-24T21:11:36.919 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:11:36.919 DEBUG:boto:Data: 2023-05-24T21:11:36.920 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.920 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:36.920 DEBUG:boto:Port: 8000 2023-05-24T21:11:36.921 DEBUG:boto:Params: {} 2023-05-24T21:11:36.921 DEBUG:boto:Token: None 2023-05-24T21:11:36.922 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:36 GMT /qskrps-13/obj 2023-05-24T21:11:36.922 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:QpeXDiVOdYqoPMJTuV41OUkFxp0= 2023-05-24T21:11:36.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:QpeXDiVOdYqoPMJTuV41OUkFxp0=', 'Content-Length': '0'} 2023-05-24T21:11:36.925 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c19f6a1c0b645730-00646e7d88-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:36.926 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:36.927 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:36.927 DEBUG:boto:path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.928 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.928 DEBUG:boto:Method: GET 2023-05-24T21:11:36.928 DEBUG:boto:Path: /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.929 DEBUG:boto:Data: 2023-05-24T21:11:36.929 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.929 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:36.930 DEBUG:boto:Port: 8001 2023-05-24T21:11:36.930 DEBUG:boto:Params: {} 2023-05-24T21:11:36.930 DEBUG:boto:Token: None 2023-05-24T21:11:36.931 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:36 GMT /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.931 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:sJUzw+r5yxlszGWfjSDqoyLBzPM= 2023-05-24T21:11:36.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:sJUzw+r5yxlszGWfjSDqoyLBzPM=', 'Content-Length': '0'} 2023-05-24T21:11:36.934 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b66b405f9cf44268-00646e7d88-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.980 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:36.981 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:36.981 DEBUG:boto:path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.982 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.982 DEBUG:boto:Method: GET 2023-05-24T21:11:36.983 DEBUG:boto:Path: /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.983 DEBUG:boto:Data: 2023-05-24T21:11:36.983 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.984 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:36.984 DEBUG:boto:Port: 8000 2023-05-24T21:11:36.984 DEBUG:boto:Params: {} 2023-05-24T21:11:36.985 DEBUG:boto:Token: None 2023-05-24T21:11:36.985 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:36 GMT /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:36.986 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:sJUzw+r5yxlszGWfjSDqoyLBzPM= 2023-05-24T21:11:36.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:sJUzw+r5yxlszGWfjSDqoyLBzPM=', 'Content-Length': '0'} 2023-05-24T21:11:36.988 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000324326fc645d3157-00646e7d88-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.989 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:36.989 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:36.990 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:36.990 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:36.990 DEBUG:boto:Method: GET 2023-05-24T21:11:36.991 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:36.991 DEBUG:boto:Data: 2023-05-24T21:11:36.992 DEBUG:boto:Headers: {} 2023-05-24T21:11:36.992 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:36.992 DEBUG:boto:Port: 8001 2023-05-24T21:11:36.993 DEBUG:boto:Params: {} 2023-05-24T21:11:36.993 DEBUG:boto:Token: None 2023-05-24T21:11:36.993 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:36 GMT /qskrps-13/ 2023-05-24T21:11:36.994 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:9f6nRfR1der32Nrg/FglWlhTnE8= 2023-05-24T21:11:36.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:9f6nRfR1der32Nrg/FglWlhTnE8=', 'Content-Length': '0'} 2023-05-24T21:11:36.997 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e33116b5caf7963-00646e7d88-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:36.998 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:36.998 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:36.999 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:36.999 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:36.999 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:37.000 DEBUG:boto:Method: GET 2023-05-24T21:11:37.000 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:37.001 DEBUG:boto:Data: 2023-05-24T21:11:37.001 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.001 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:37.002 DEBUG:boto:Port: 8000 2023-05-24T21:11:37.002 DEBUG:boto:Params: {} 2023-05-24T21:11:37.002 DEBUG:boto:Token: None 2023-05-24T21:11:37.003 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/ 2023-05-24T21:11:37.003 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Yq7x3Vk2BaiZhXUVMszjtFsbHVw= 2023-05-24T21:11:37.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Yq7x3Vk2BaiZhXUVMszjtFsbHVw=', 'Content-Length': '0'} 2023-05-24T21:11:37.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001276c03b9ab25849-00646e7d88-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.052 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:37.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:37.053 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.054 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.054 DEBUG:boto:path=/qskrps-13/obj?versionId=T6O7lunS-ugcpId8P5cmF9RHD1HJqI8 2023-05-24T21:11:37.055 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=T6O7lunS-ugcpId8P5cmF9RHD1HJqI8 2023-05-24T21:11:37.055 DEBUG:boto:Method: GET 2023-05-24T21:11:37.055 DEBUG:boto:Path: /qskrps-13/obj?versionId=T6O7lunS-ugcpId8P5cmF9RHD1HJqI8 2023-05-24T21:11:37.056 DEBUG:boto:Data: 2023-05-24T21:11:37.056 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.056 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:37.057 DEBUG:boto:Port: 8001 2023-05-24T21:11:37.057 DEBUG:boto:Params: {} 2023-05-24T21:11:37.058 DEBUG:boto:Token: None 2023-05-24T21:11:37.058 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj?versionId=T6O7lunS-ugcpId8P5cmF9RHD1HJqI8 2023-05-24T21:11:37.059 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:d6x3OsSCZgPIi1Pp9eI6uPbbPxM= 2023-05-24T21:11:37.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:d6x3OsSCZgPIi1Pp9eI6uPbbPxM=', 'Content-Length': '0'} 2023-05-24T21:11:37.061 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'T6O7lunS-ugcpId8P5cmF9RHD1HJqI8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c1b84da14f032abb-00646e7d89-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.062 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.062 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.063 DEBUG:boto:path=/qskrps-13/obj?versionId=T6O7lunS-ugcpId8P5cmF9RHD1HJqI8 2023-05-24T21:11:37.063 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=T6O7lunS-ugcpId8P5cmF9RHD1HJqI8 2023-05-24T21:11:37.064 DEBUG:boto:Method: GET 2023-05-24T21:11:37.064 DEBUG:boto:Path: /qskrps-13/obj?versionId=T6O7lunS-ugcpId8P5cmF9RHD1HJqI8 2023-05-24T21:11:37.064 DEBUG:boto:Data: 2023-05-24T21:11:37.065 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.065 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:37.065 DEBUG:boto:Port: 8000 2023-05-24T21:11:37.066 DEBUG:boto:Params: {} 2023-05-24T21:11:37.066 DEBUG:boto:Token: None 2023-05-24T21:11:37.067 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj?versionId=T6O7lunS-ugcpId8P5cmF9RHD1HJqI8 2023-05-24T21:11:37.067 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:d6x3OsSCZgPIi1Pp9eI6uPbbPxM= 2023-05-24T21:11:37.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:d6x3OsSCZgPIi1Pp9eI6uPbbPxM=', 'Content-Length': '0'} 2023-05-24T21:11:37.069 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'T6O7lunS-ugcpId8P5cmF9RHD1HJqI8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a0e39502f70cd16-00646e7d89-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.070 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.070 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.071 DEBUG:boto:path=/qskrps-13/obj?versionId=T6O7lunS-ugcpId8P5cmF9RHD1HJqI8 2023-05-24T21:11:37.071 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=T6O7lunS-ugcpId8P5cmF9RHD1HJqI8 2023-05-24T21:11:37.071 DEBUG:boto:Method: HEAD 2023-05-24T21:11:37.072 DEBUG:boto:Path: /qskrps-13/obj?versionId=T6O7lunS-ugcpId8P5cmF9RHD1HJqI8 2023-05-24T21:11:37.072 DEBUG:boto:Data: 2023-05-24T21:11:37.072 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.073 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:37.074 DEBUG:boto:Port: 8001 2023-05-24T21:11:37.074 DEBUG:boto:Params: {} 2023-05-24T21:11:37.075 DEBUG:boto:Token: None 2023-05-24T21:11:37.075 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj?versionId=T6O7lunS-ugcpId8P5cmF9RHD1HJqI8 2023-05-24T21:11:37.076 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:k6jdLPD28ieUBJjQTwMF4Tlek3g= 2023-05-24T21:11:37.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:k6jdLPD28ieUBJjQTwMF4Tlek3g=', 'Content-Length': '0'} 2023-05-24T21:11:37.078 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'T6O7lunS-ugcpId8P5cmF9RHD1HJqI8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d9cab261ade32637-00646e7d89-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.079 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.079 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.079 DEBUG:boto:path=/qskrps-13/obj?versionId=T6O7lunS-ugcpId8P5cmF9RHD1HJqI8 2023-05-24T21:11:37.080 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=T6O7lunS-ugcpId8P5cmF9RHD1HJqI8 2023-05-24T21:11:37.080 DEBUG:boto:Method: HEAD 2023-05-24T21:11:37.081 DEBUG:boto:Path: /qskrps-13/obj?versionId=T6O7lunS-ugcpId8P5cmF9RHD1HJqI8 2023-05-24T21:11:37.081 DEBUG:boto:Data: 2023-05-24T21:11:37.081 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.082 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:37.082 DEBUG:boto:Port: 8000 2023-05-24T21:11:37.082 DEBUG:boto:Params: {} 2023-05-24T21:11:37.083 DEBUG:boto:Token: None 2023-05-24T21:11:37.083 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj?versionId=T6O7lunS-ugcpId8P5cmF9RHD1HJqI8 2023-05-24T21:11:37.084 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:k6jdLPD28ieUBJjQTwMF4Tlek3g= 2023-05-24T21:11:37.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:k6jdLPD28ieUBJjQTwMF4Tlek3g=', 'Content-Length': '0'} 2023-05-24T21:11:37.086 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'T6O7lunS-ugcpId8P5cmF9RHD1HJqI8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d2b07399322c1dd3-00646e7d89-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:37.087 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.088 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.088 DEBUG:boto:path=/qskrps-13/obj?versionId=T6O7lunS-ugcpId8P5cmF9RHD1HJqI8 2023-05-24T21:11:37.089 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=T6O7lunS-ugcpId8P5cmF9RHD1HJqI8 2023-05-24T21:11:37.089 DEBUG:boto:Method: GET 2023-05-24T21:11:37.089 DEBUG:boto:Path: /qskrps-13/obj?versionId=T6O7lunS-ugcpId8P5cmF9RHD1HJqI8 2023-05-24T21:11:37.090 DEBUG:boto:Data: 2023-05-24T21:11:37.090 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.091 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:37.091 DEBUG:boto:Port: 8001 2023-05-24T21:11:37.092 DEBUG:boto:Params: {} 2023-05-24T21:11:37.093 DEBUG:boto:Token: None 2023-05-24T21:11:37.093 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj?versionId=T6O7lunS-ugcpId8P5cmF9RHD1HJqI8 2023-05-24T21:11:37.094 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:d6x3OsSCZgPIi1Pp9eI6uPbbPxM= 2023-05-24T21:11:37.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:d6x3OsSCZgPIi1Pp9eI6uPbbPxM=', 'Content-Length': '0'} 2023-05-24T21:11:37.096 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'T6O7lunS-ugcpId8P5cmF9RHD1HJqI8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003ccb148e7e399e58-00646e7d89-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.097 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.098 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.098 DEBUG:boto:path=/qskrps-13/obj?versionId=T6O7lunS-ugcpId8P5cmF9RHD1HJqI8 2023-05-24T21:11:37.099 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=T6O7lunS-ugcpId8P5cmF9RHD1HJqI8 2023-05-24T21:11:37.099 DEBUG:boto:Method: GET 2023-05-24T21:11:37.100 DEBUG:boto:Path: /qskrps-13/obj?versionId=T6O7lunS-ugcpId8P5cmF9RHD1HJqI8 2023-05-24T21:11:37.100 DEBUG:boto:Data: 2023-05-24T21:11:37.101 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.101 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:37.102 DEBUG:boto:Port: 8000 2023-05-24T21:11:37.103 DEBUG:boto:Params: {} 2023-05-24T21:11:37.103 DEBUG:boto:Token: None 2023-05-24T21:11:37.104 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj?versionId=T6O7lunS-ugcpId8P5cmF9RHD1HJqI8 2023-05-24T21:11:37.104 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:d6x3OsSCZgPIi1Pp9eI6uPbbPxM= 2023-05-24T21:11:37.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:d6x3OsSCZgPIi1Pp9eI6uPbbPxM=', 'Content-Length': '0'} 2023-05-24T21:11:37.107 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'T6O7lunS-ugcpId8P5cmF9RHD1HJqI8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000090721ddca646d6f1-00646e7d89-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.108 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:37.108 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:37.109 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:37.109 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:37.110 DEBUG:boto:Method: GET 2023-05-24T21:11:37.110 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:37.111 DEBUG:boto:Data: 2023-05-24T21:11:37.111 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.111 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:37.112 DEBUG:boto:Port: 8001 2023-05-24T21:11:37.112 DEBUG:boto:Params: {} 2023-05-24T21:11:37.113 DEBUG:boto:Token: None 2023-05-24T21:11:37.113 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/ 2023-05-24T21:11:37.114 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Yq7x3Vk2BaiZhXUVMszjtFsbHVw= 2023-05-24T21:11:37.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Yq7x3Vk2BaiZhXUVMszjtFsbHVw=', 'Content-Length': '0'} 2023-05-24T21:11:37.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f4b43b88f6aa6d2-00646e7d89-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.160 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:37.161 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:37.162 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:37.162 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:37.163 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:37.163 DEBUG:boto:Method: GET 2023-05-24T21:11:37.164 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:37.164 DEBUG:boto:Data: 2023-05-24T21:11:37.165 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.165 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:37.165 DEBUG:boto:Port: 8000 2023-05-24T21:11:37.166 DEBUG:boto:Params: {} 2023-05-24T21:11:37.166 DEBUG:boto:Token: None 2023-05-24T21:11:37.167 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/ 2023-05-24T21:11:37.167 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Yq7x3Vk2BaiZhXUVMszjtFsbHVw= 2023-05-24T21:11:37.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Yq7x3Vk2BaiZhXUVMszjtFsbHVw=', 'Content-Length': '0'} 2023-05-24T21:11:37.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3bdd058b67edc59-00646e7d89-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.171 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:37.172 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.172 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.173 DEBUG:boto:Method: HEAD 2023-05-24T21:11:37.173 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:11:37.173 DEBUG:boto:Data: 2023-05-24T21:11:37.174 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.174 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:37.175 DEBUG:boto:Port: 8001 2023-05-24T21:11:37.175 DEBUG:boto:Params: {} 2023-05-24T21:11:37.175 DEBUG:boto:Token: None 2023-05-24T21:11:37.176 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj 2023-05-24T21:11:37.176 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:trVDb6z7e9C2FiTIR87GPhzUXuE= 2023-05-24T21:11:37.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:trVDb6z7e9C2FiTIR87GPhzUXuE=', 'Content-Length': '0'} 2023-05-24T21:11:37.180 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c60f40488e510a1a-00646e7d89-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.180 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.181 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.181 DEBUG:boto:Method: HEAD 2023-05-24T21:11:37.181 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:11:37.182 DEBUG:boto:Data: 2023-05-24T21:11:37.182 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.182 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:37.183 DEBUG:boto:Port: 8000 2023-05-24T21:11:37.183 DEBUG:boto:Params: {} 2023-05-24T21:11:37.183 DEBUG:boto:Token: None 2023-05-24T21:11:37.184 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj 2023-05-24T21:11:37.184 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:trVDb6z7e9C2FiTIR87GPhzUXuE= 2023-05-24T21:11:37.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:trVDb6z7e9C2FiTIR87GPhzUXuE=', 'Content-Length': '0'} 2023-05-24T21:11:37.188 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003c8fbd140f4ee6d8-00646e7d89-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:37.189 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.189 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.189 DEBUG:boto:path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:37.190 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:37.190 DEBUG:boto:Method: GET 2023-05-24T21:11:37.190 DEBUG:boto:Path: /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:37.191 DEBUG:boto:Data: 2023-05-24T21:11:37.191 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.192 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:37.192 DEBUG:boto:Port: 8001 2023-05-24T21:11:37.192 DEBUG:boto:Params: {} 2023-05-24T21:11:37.193 DEBUG:boto:Token: None 2023-05-24T21:11:37.193 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:37.194 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:qL19ViArSYW5jPKAJUwyd8qHF20= 2023-05-24T21:11:37.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qL19ViArSYW5jPKAJUwyd8qHF20=', 'Content-Length': '0'} 2023-05-24T21:11:37.196 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000038eda514102bb178-00646e7d89-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.197 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.197 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.197 DEBUG:boto:path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:37.198 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:37.198 DEBUG:boto:Method: GET 2023-05-24T21:11:37.198 DEBUG:boto:Path: /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:37.199 DEBUG:boto:Data: 2023-05-24T21:11:37.199 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.200 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:37.200 DEBUG:boto:Port: 8000 2023-05-24T21:11:37.200 DEBUG:boto:Params: {} 2023-05-24T21:11:37.201 DEBUG:boto:Token: None 2023-05-24T21:11:37.201 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:37.202 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:qL19ViArSYW5jPKAJUwyd8qHF20= 2023-05-24T21:11:37.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qL19ViArSYW5jPKAJUwyd8qHF20=', 'Content-Length': '0'} 2023-05-24T21:11:37.204 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000274450dd4f212274-00646e7d89-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.204 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:37.205 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:37.205 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:37.205 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:37.206 DEBUG:boto:Method: GET 2023-05-24T21:11:37.206 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:37.207 DEBUG:boto:Data: 2023-05-24T21:11:37.207 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.207 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:37.208 DEBUG:boto:Port: 8001 2023-05-24T21:11:37.208 DEBUG:boto:Params: {} 2023-05-24T21:11:37.209 DEBUG:boto:Token: None 2023-05-24T21:11:37.209 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/ 2023-05-24T21:11:37.209 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Yq7x3Vk2BaiZhXUVMszjtFsbHVw= 2023-05-24T21:11:37.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Yq7x3Vk2BaiZhXUVMszjtFsbHVw=', 'Content-Length': '0'} 2023-05-24T21:11:37.212 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000858b5ebcebc66e28-00646e7d89-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.213 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:37.214 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:37.214 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:37.214 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:37.215 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:37.215 DEBUG:boto:Method: GET 2023-05-24T21:11:37.215 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:37.216 DEBUG:boto:Data: 2023-05-24T21:11:37.216 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.216 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:37.217 DEBUG:boto:Port: 8000 2023-05-24T21:11:37.217 DEBUG:boto:Params: {} 2023-05-24T21:11:37.218 DEBUG:boto:Token: None 2023-05-24T21:11:37.218 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/ 2023-05-24T21:11:37.218 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Yq7x3Vk2BaiZhXUVMszjtFsbHVw= 2023-05-24T21:11:37.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Yq7x3Vk2BaiZhXUVMszjtFsbHVw=', 'Content-Length': '0'} 2023-05-24T21:11:37.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e785052a7e1bfd66-00646e7d89-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.222 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:37.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:37.223 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.223 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.224 DEBUG:boto:path=/qskrps-13/obj?versionId=-H.jz834T6ck7fPleRHP.k9CJzkLgis 2023-05-24T21:11:37.224 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=-H.jz834T6ck7fPleRHP.k9CJzkLgis 2023-05-24T21:11:37.224 DEBUG:boto:Method: GET 2023-05-24T21:11:37.225 DEBUG:boto:Path: /qskrps-13/obj?versionId=-H.jz834T6ck7fPleRHP.k9CJzkLgis 2023-05-24T21:11:37.225 DEBUG:boto:Data: 2023-05-24T21:11:37.225 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.226 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:37.226 DEBUG:boto:Port: 8001 2023-05-24T21:11:37.227 DEBUG:boto:Params: {} 2023-05-24T21:11:37.227 DEBUG:boto:Token: None 2023-05-24T21:11:37.227 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj?versionId=-H.jz834T6ck7fPleRHP.k9CJzkLgis 2023-05-24T21:11:37.228 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:TSApO9MqYYwzK12p3qzlZ77siTk= 2023-05-24T21:11:37.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:TSApO9MqYYwzK12p3qzlZ77siTk=', 'Content-Length': '0'} 2023-05-24T21:11:37.230 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', '-H.jz834T6ck7fPleRHP.k9CJzkLgis'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000049f03f2ccfc75acc-00646e7d89-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.272 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.273 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.274 DEBUG:boto:path=/qskrps-13/obj?versionId=-H.jz834T6ck7fPleRHP.k9CJzkLgis 2023-05-24T21:11:37.274 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=-H.jz834T6ck7fPleRHP.k9CJzkLgis 2023-05-24T21:11:37.274 DEBUG:boto:Method: GET 2023-05-24T21:11:37.275 DEBUG:boto:Path: /qskrps-13/obj?versionId=-H.jz834T6ck7fPleRHP.k9CJzkLgis 2023-05-24T21:11:37.275 DEBUG:boto:Data: 2023-05-24T21:11:37.276 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.276 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:37.276 DEBUG:boto:Port: 8000 2023-05-24T21:11:37.277 DEBUG:boto:Params: {} 2023-05-24T21:11:37.277 DEBUG:boto:Token: None 2023-05-24T21:11:37.278 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj?versionId=-H.jz834T6ck7fPleRHP.k9CJzkLgis 2023-05-24T21:11:37.278 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:TSApO9MqYYwzK12p3qzlZ77siTk= 2023-05-24T21:11:37.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:TSApO9MqYYwzK12p3qzlZ77siTk=', 'Content-Length': '0'} 2023-05-24T21:11:37.280 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', '-H.jz834T6ck7fPleRHP.k9CJzkLgis'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e841e325da4b99b5-00646e7d89-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.281 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.282 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.282 DEBUG:boto:path=/qskrps-13/obj?versionId=-H.jz834T6ck7fPleRHP.k9CJzkLgis 2023-05-24T21:11:37.282 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=-H.jz834T6ck7fPleRHP.k9CJzkLgis 2023-05-24T21:11:37.283 DEBUG:boto:Method: HEAD 2023-05-24T21:11:37.283 DEBUG:boto:Path: /qskrps-13/obj?versionId=-H.jz834T6ck7fPleRHP.k9CJzkLgis 2023-05-24T21:11:37.283 DEBUG:boto:Data: 2023-05-24T21:11:37.284 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.284 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:37.284 DEBUG:boto:Port: 8001 2023-05-24T21:11:37.285 DEBUG:boto:Params: {} 2023-05-24T21:11:37.285 DEBUG:boto:Token: None 2023-05-24T21:11:37.285 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj?versionId=-H.jz834T6ck7fPleRHP.k9CJzkLgis 2023-05-24T21:11:37.286 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:VqzqYlY9TDxvC3Pmgvlb3QLfBB4= 2023-05-24T21:11:37.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:VqzqYlY9TDxvC3Pmgvlb3QLfBB4=', 'Content-Length': '0'} 2023-05-24T21:11:37.288 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', '-H.jz834T6ck7fPleRHP.k9CJzkLgis'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007969764390027eca-00646e7d89-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.288 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.289 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.289 DEBUG:boto:path=/qskrps-13/obj?versionId=-H.jz834T6ck7fPleRHP.k9CJzkLgis 2023-05-24T21:11:37.290 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=-H.jz834T6ck7fPleRHP.k9CJzkLgis 2023-05-24T21:11:37.290 DEBUG:boto:Method: HEAD 2023-05-24T21:11:37.290 DEBUG:boto:Path: /qskrps-13/obj?versionId=-H.jz834T6ck7fPleRHP.k9CJzkLgis 2023-05-24T21:11:37.291 DEBUG:boto:Data: 2023-05-24T21:11:37.291 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.291 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:37.292 DEBUG:boto:Port: 8000 2023-05-24T21:11:37.292 DEBUG:boto:Params: {} 2023-05-24T21:11:37.292 DEBUG:boto:Token: None 2023-05-24T21:11:37.293 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj?versionId=-H.jz834T6ck7fPleRHP.k9CJzkLgis 2023-05-24T21:11:37.293 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:VqzqYlY9TDxvC3Pmgvlb3QLfBB4= 2023-05-24T21:11:37.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:VqzqYlY9TDxvC3Pmgvlb3QLfBB4=', 'Content-Length': '0'} 2023-05-24T21:11:37.295 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', '-H.jz834T6ck7fPleRHP.k9CJzkLgis'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000084680f2885d86e90-00646e7d89-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:37.296 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.297 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.297 DEBUG:boto:path=/qskrps-13/obj?versionId=-H.jz834T6ck7fPleRHP.k9CJzkLgis 2023-05-24T21:11:37.297 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=-H.jz834T6ck7fPleRHP.k9CJzkLgis 2023-05-24T21:11:37.298 DEBUG:boto:Method: GET 2023-05-24T21:11:37.298 DEBUG:boto:Path: /qskrps-13/obj?versionId=-H.jz834T6ck7fPleRHP.k9CJzkLgis 2023-05-24T21:11:37.299 DEBUG:boto:Data: 2023-05-24T21:11:37.299 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.299 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:37.300 DEBUG:boto:Port: 8001 2023-05-24T21:11:37.300 DEBUG:boto:Params: {} 2023-05-24T21:11:37.300 DEBUG:boto:Token: None 2023-05-24T21:11:37.301 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj?versionId=-H.jz834T6ck7fPleRHP.k9CJzkLgis 2023-05-24T21:11:37.301 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:TSApO9MqYYwzK12p3qzlZ77siTk= 2023-05-24T21:11:37.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:TSApO9MqYYwzK12p3qzlZ77siTk=', 'Content-Length': '0'} 2023-05-24T21:11:37.303 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', '-H.jz834T6ck7fPleRHP.k9CJzkLgis'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000da4a5516a30ace0c-00646e7d89-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.304 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.305 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.305 DEBUG:boto:path=/qskrps-13/obj?versionId=-H.jz834T6ck7fPleRHP.k9CJzkLgis 2023-05-24T21:11:37.305 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=-H.jz834T6ck7fPleRHP.k9CJzkLgis 2023-05-24T21:11:37.306 DEBUG:boto:Method: GET 2023-05-24T21:11:37.306 DEBUG:boto:Path: /qskrps-13/obj?versionId=-H.jz834T6ck7fPleRHP.k9CJzkLgis 2023-05-24T21:11:37.306 DEBUG:boto:Data: 2023-05-24T21:11:37.307 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.307 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:37.307 DEBUG:boto:Port: 8000 2023-05-24T21:11:37.308 DEBUG:boto:Params: {} 2023-05-24T21:11:37.308 DEBUG:boto:Token: None 2023-05-24T21:11:37.309 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj?versionId=-H.jz834T6ck7fPleRHP.k9CJzkLgis 2023-05-24T21:11:37.309 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:TSApO9MqYYwzK12p3qzlZ77siTk= 2023-05-24T21:11:37.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:TSApO9MqYYwzK12p3qzlZ77siTk=', 'Content-Length': '0'} 2023-05-24T21:11:37.311 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', '-H.jz834T6ck7fPleRHP.k9CJzkLgis'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000da090413a3b4d9eb-00646e7d89-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.312 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:37.313 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:37.313 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:37.313 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:37.314 DEBUG:boto:Method: GET 2023-05-24T21:11:37.314 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:37.314 DEBUG:boto:Data: 2023-05-24T21:11:37.315 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.315 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:37.315 DEBUG:boto:Port: 8001 2023-05-24T21:11:37.316 DEBUG:boto:Params: {} 2023-05-24T21:11:37.316 DEBUG:boto:Token: None 2023-05-24T21:11:37.317 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/ 2023-05-24T21:11:37.317 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Yq7x3Vk2BaiZhXUVMszjtFsbHVw= 2023-05-24T21:11:37.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Yq7x3Vk2BaiZhXUVMszjtFsbHVw=', 'Content-Length': '0'} 2023-05-24T21:11:37.320 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ccee31732030a7e-00646e7d89-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.320 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:37.321 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:37.322 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:37.322 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:37.322 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:37.323 DEBUG:boto:Method: GET 2023-05-24T21:11:37.323 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:37.323 DEBUG:boto:Data: 2023-05-24T21:11:37.324 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.324 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:37.325 DEBUG:boto:Port: 8000 2023-05-24T21:11:37.325 DEBUG:boto:Params: {} 2023-05-24T21:11:37.325 DEBUG:boto:Token: None 2023-05-24T21:11:37.326 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/ 2023-05-24T21:11:37.326 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Yq7x3Vk2BaiZhXUVMszjtFsbHVw= 2023-05-24T21:11:37.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Yq7x3Vk2BaiZhXUVMszjtFsbHVw=', 'Content-Length': '0'} 2023-05-24T21:11:37.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005db32423c148451d-00646e7d89-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.329 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:37.330 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.330 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.331 DEBUG:boto:Method: HEAD 2023-05-24T21:11:37.331 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:11:37.331 DEBUG:boto:Data: 2023-05-24T21:11:37.332 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.332 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:37.332 DEBUG:boto:Port: 8001 2023-05-24T21:11:37.333 DEBUG:boto:Params: {} 2023-05-24T21:11:37.333 DEBUG:boto:Token: None 2023-05-24T21:11:37.334 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj 2023-05-24T21:11:37.334 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:trVDb6z7e9C2FiTIR87GPhzUXuE= 2023-05-24T21:11:37.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:trVDb6z7e9C2FiTIR87GPhzUXuE=', 'Content-Length': '0'} 2023-05-24T21:11:37.337 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000081057f3c9bb50cdf-00646e7d89-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.338 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.338 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.339 DEBUG:boto:Method: HEAD 2023-05-24T21:11:37.339 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:11:37.339 DEBUG:boto:Data: 2023-05-24T21:11:37.340 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.340 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:37.340 DEBUG:boto:Port: 8000 2023-05-24T21:11:37.341 DEBUG:boto:Params: {} 2023-05-24T21:11:37.341 DEBUG:boto:Token: None 2023-05-24T21:11:37.341 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj 2023-05-24T21:11:37.342 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:trVDb6z7e9C2FiTIR87GPhzUXuE= 2023-05-24T21:11:37.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:trVDb6z7e9C2FiTIR87GPhzUXuE=', 'Content-Length': '0'} 2023-05-24T21:11:37.345 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ba368456397d50b3-00646e7d89-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:37.346 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.346 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.347 DEBUG:boto:path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:37.347 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:37.348 DEBUG:boto:Method: GET 2023-05-24T21:11:37.348 DEBUG:boto:Path: /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:37.348 DEBUG:boto:Data: 2023-05-24T21:11:37.349 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.349 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:37.349 DEBUG:boto:Port: 8001 2023-05-24T21:11:37.350 DEBUG:boto:Params: {} 2023-05-24T21:11:37.350 DEBUG:boto:Token: None 2023-05-24T21:11:37.351 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:37.351 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:qL19ViArSYW5jPKAJUwyd8qHF20= 2023-05-24T21:11:37.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qL19ViArSYW5jPKAJUwyd8qHF20=', 'Content-Length': '0'} 2023-05-24T21:11:37.353 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000027c87bb6174290b6-00646e7d89-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.354 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.354 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.355 DEBUG:boto:path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:37.355 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:37.355 DEBUG:boto:Method: GET 2023-05-24T21:11:37.356 DEBUG:boto:Path: /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:37.356 DEBUG:boto:Data: 2023-05-24T21:11:37.356 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.357 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:37.357 DEBUG:boto:Port: 8000 2023-05-24T21:11:37.358 DEBUG:boto:Params: {} 2023-05-24T21:11:37.358 DEBUG:boto:Token: None 2023-05-24T21:11:37.358 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:37.359 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:qL19ViArSYW5jPKAJUwyd8qHF20= 2023-05-24T21:11:37.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qL19ViArSYW5jPKAJUwyd8qHF20=', 'Content-Length': '0'} 2023-05-24T21:11:37.361 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9105ed73f4629c0-00646e7d89-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.404 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:37.405 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:37.405 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:37.406 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:37.406 DEBUG:boto:Method: GET 2023-05-24T21:11:37.406 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:37.407 DEBUG:boto:Data: 2023-05-24T21:11:37.407 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.407 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:37.408 DEBUG:boto:Port: 8001 2023-05-24T21:11:37.408 DEBUG:boto:Params: {} 2023-05-24T21:11:37.409 DEBUG:boto:Token: None 2023-05-24T21:11:37.409 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/ 2023-05-24T21:11:37.410 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Yq7x3Vk2BaiZhXUVMszjtFsbHVw= 2023-05-24T21:11:37.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Yq7x3Vk2BaiZhXUVMszjtFsbHVw=', 'Content-Length': '0'} 2023-05-24T21:11:37.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003acf59591aac322-00646e7d89-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.413 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:37.414 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:37.414 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:37.414 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:37.415 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:37.415 DEBUG:boto:Method: GET 2023-05-24T21:11:37.415 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:37.416 DEBUG:boto:Data: 2023-05-24T21:11:37.416 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.416 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:37.417 DEBUG:boto:Port: 8000 2023-05-24T21:11:37.417 DEBUG:boto:Params: {} 2023-05-24T21:11:37.418 DEBUG:boto:Token: None 2023-05-24T21:11:37.418 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/ 2023-05-24T21:11:37.418 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Yq7x3Vk2BaiZhXUVMszjtFsbHVw= 2023-05-24T21:11:37.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Yq7x3Vk2BaiZhXUVMszjtFsbHVw=', 'Content-Length': '0'} 2023-05-24T21:11:37.421 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041bee8e0325b3ac8-00646e7d89-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.422 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:37.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:37.423 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.423 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.424 DEBUG:boto:path=/qskrps-13/obj?versionId=ORI9tDTIyFkUt3B9j46aV8ZBUYqr2Kp 2023-05-24T21:11:37.424 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=ORI9tDTIyFkUt3B9j46aV8ZBUYqr2Kp 2023-05-24T21:11:37.424 DEBUG:boto:Method: GET 2023-05-24T21:11:37.425 DEBUG:boto:Path: /qskrps-13/obj?versionId=ORI9tDTIyFkUt3B9j46aV8ZBUYqr2Kp 2023-05-24T21:11:37.425 DEBUG:boto:Data: 2023-05-24T21:11:37.425 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.426 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:37.426 DEBUG:boto:Port: 8001 2023-05-24T21:11:37.427 DEBUG:boto:Params: {} 2023-05-24T21:11:37.427 DEBUG:boto:Token: None 2023-05-24T21:11:37.427 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj?versionId=ORI9tDTIyFkUt3B9j46aV8ZBUYqr2Kp 2023-05-24T21:11:37.428 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Gad9HYgMfNaSmgJwNEkeifDI+js= 2023-05-24T21:11:37.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Gad9HYgMfNaSmgJwNEkeifDI+js=', 'Content-Length': '0'} 2023-05-24T21:11:37.430 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'ORI9tDTIyFkUt3B9j46aV8ZBUYqr2Kp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000095b3f7f25de928b2-00646e7d89-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.472 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.473 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.473 DEBUG:boto:path=/qskrps-13/obj?versionId=ORI9tDTIyFkUt3B9j46aV8ZBUYqr2Kp 2023-05-24T21:11:37.473 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=ORI9tDTIyFkUt3B9j46aV8ZBUYqr2Kp 2023-05-24T21:11:37.474 DEBUG:boto:Method: GET 2023-05-24T21:11:37.474 DEBUG:boto:Path: /qskrps-13/obj?versionId=ORI9tDTIyFkUt3B9j46aV8ZBUYqr2Kp 2023-05-24T21:11:37.475 DEBUG:boto:Data: 2023-05-24T21:11:37.475 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.475 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:37.476 DEBUG:boto:Port: 8000 2023-05-24T21:11:37.476 DEBUG:boto:Params: {} 2023-05-24T21:11:37.476 DEBUG:boto:Token: None 2023-05-24T21:11:37.477 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj?versionId=ORI9tDTIyFkUt3B9j46aV8ZBUYqr2Kp 2023-05-24T21:11:37.477 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Gad9HYgMfNaSmgJwNEkeifDI+js= 2023-05-24T21:11:37.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Gad9HYgMfNaSmgJwNEkeifDI+js=', 'Content-Length': '0'} 2023-05-24T21:11:37.480 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'ORI9tDTIyFkUt3B9j46aV8ZBUYqr2Kp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e6471d1ef479064d-00646e7d89-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.481 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.481 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.481 DEBUG:boto:path=/qskrps-13/obj?versionId=ORI9tDTIyFkUt3B9j46aV8ZBUYqr2Kp 2023-05-24T21:11:37.482 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=ORI9tDTIyFkUt3B9j46aV8ZBUYqr2Kp 2023-05-24T21:11:37.482 DEBUG:boto:Method: HEAD 2023-05-24T21:11:37.482 DEBUG:boto:Path: /qskrps-13/obj?versionId=ORI9tDTIyFkUt3B9j46aV8ZBUYqr2Kp 2023-05-24T21:11:37.483 DEBUG:boto:Data: 2023-05-24T21:11:37.483 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.483 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:37.484 DEBUG:boto:Port: 8001 2023-05-24T21:11:37.484 DEBUG:boto:Params: {} 2023-05-24T21:11:37.484 DEBUG:boto:Token: None 2023-05-24T21:11:37.485 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj?versionId=ORI9tDTIyFkUt3B9j46aV8ZBUYqr2Kp 2023-05-24T21:11:37.485 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:faqw4bg39OSpm0Ox8tFPKiheTYs= 2023-05-24T21:11:37.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:faqw4bg39OSpm0Ox8tFPKiheTYs=', 'Content-Length': '0'} 2023-05-24T21:11:37.487 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'ORI9tDTIyFkUt3B9j46aV8ZBUYqr2Kp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002466e587ad33584d-00646e7d89-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.488 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.488 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.488 DEBUG:boto:path=/qskrps-13/obj?versionId=ORI9tDTIyFkUt3B9j46aV8ZBUYqr2Kp 2023-05-24T21:11:37.489 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=ORI9tDTIyFkUt3B9j46aV8ZBUYqr2Kp 2023-05-24T21:11:37.489 DEBUG:boto:Method: HEAD 2023-05-24T21:11:37.490 DEBUG:boto:Path: /qskrps-13/obj?versionId=ORI9tDTIyFkUt3B9j46aV8ZBUYqr2Kp 2023-05-24T21:11:37.490 DEBUG:boto:Data: 2023-05-24T21:11:37.490 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.491 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:37.491 DEBUG:boto:Port: 8000 2023-05-24T21:11:37.491 DEBUG:boto:Params: {} 2023-05-24T21:11:37.492 DEBUG:boto:Token: None 2023-05-24T21:11:37.492 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj?versionId=ORI9tDTIyFkUt3B9j46aV8ZBUYqr2Kp 2023-05-24T21:11:37.492 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:faqw4bg39OSpm0Ox8tFPKiheTYs= 2023-05-24T21:11:37.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:faqw4bg39OSpm0Ox8tFPKiheTYs=', 'Content-Length': '0'} 2023-05-24T21:11:37.494 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'ORI9tDTIyFkUt3B9j46aV8ZBUYqr2Kp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005225ebb1a69350aa-00646e7d89-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:37.495 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.496 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.496 DEBUG:boto:path=/qskrps-13/obj?versionId=ORI9tDTIyFkUt3B9j46aV8ZBUYqr2Kp 2023-05-24T21:11:37.496 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=ORI9tDTIyFkUt3B9j46aV8ZBUYqr2Kp 2023-05-24T21:11:37.497 DEBUG:boto:Method: GET 2023-05-24T21:11:37.497 DEBUG:boto:Path: /qskrps-13/obj?versionId=ORI9tDTIyFkUt3B9j46aV8ZBUYqr2Kp 2023-05-24T21:11:37.498 DEBUG:boto:Data: 2023-05-24T21:11:37.498 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.498 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:37.499 DEBUG:boto:Port: 8001 2023-05-24T21:11:37.499 DEBUG:boto:Params: {} 2023-05-24T21:11:37.499 DEBUG:boto:Token: None 2023-05-24T21:11:37.500 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj?versionId=ORI9tDTIyFkUt3B9j46aV8ZBUYqr2Kp 2023-05-24T21:11:37.500 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Gad9HYgMfNaSmgJwNEkeifDI+js= 2023-05-24T21:11:37.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Gad9HYgMfNaSmgJwNEkeifDI+js=', 'Content-Length': '0'} 2023-05-24T21:11:37.503 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'ORI9tDTIyFkUt3B9j46aV8ZBUYqr2Kp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e328711ddb96862a-00646e7d89-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.503 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.504 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.504 DEBUG:boto:path=/qskrps-13/obj?versionId=ORI9tDTIyFkUt3B9j46aV8ZBUYqr2Kp 2023-05-24T21:11:37.504 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=ORI9tDTIyFkUt3B9j46aV8ZBUYqr2Kp 2023-05-24T21:11:37.505 DEBUG:boto:Method: GET 2023-05-24T21:11:37.505 DEBUG:boto:Path: /qskrps-13/obj?versionId=ORI9tDTIyFkUt3B9j46aV8ZBUYqr2Kp 2023-05-24T21:11:37.505 DEBUG:boto:Data: 2023-05-24T21:11:37.506 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.506 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:37.506 DEBUG:boto:Port: 8000 2023-05-24T21:11:37.507 DEBUG:boto:Params: {} 2023-05-24T21:11:37.507 DEBUG:boto:Token: None 2023-05-24T21:11:37.508 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj?versionId=ORI9tDTIyFkUt3B9j46aV8ZBUYqr2Kp 2023-05-24T21:11:37.508 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Gad9HYgMfNaSmgJwNEkeifDI+js= 2023-05-24T21:11:37.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Gad9HYgMfNaSmgJwNEkeifDI+js=', 'Content-Length': '0'} 2023-05-24T21:11:37.510 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'ORI9tDTIyFkUt3B9j46aV8ZBUYqr2Kp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007aec7ddc8cc5fd91-00646e7d89-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.511 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:37.511 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:37.512 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:37.512 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:37.513 DEBUG:boto:Method: GET 2023-05-24T21:11:37.513 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:37.513 DEBUG:boto:Data: 2023-05-24T21:11:37.514 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.514 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:37.514 DEBUG:boto:Port: 8001 2023-05-24T21:11:37.515 DEBUG:boto:Params: {} 2023-05-24T21:11:37.515 DEBUG:boto:Token: None 2023-05-24T21:11:37.515 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/ 2023-05-24T21:11:37.516 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Yq7x3Vk2BaiZhXUVMszjtFsbHVw= 2023-05-24T21:11:37.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Yq7x3Vk2BaiZhXUVMszjtFsbHVw=', 'Content-Length': '0'} 2023-05-24T21:11:37.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000070779bf5b763711-00646e7d89-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.519 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:37.520 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:37.520 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:37.521 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:37.521 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:37.521 DEBUG:boto:Method: GET 2023-05-24T21:11:37.522 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:37.522 DEBUG:boto:Data: 2023-05-24T21:11:37.522 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.523 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:37.523 DEBUG:boto:Port: 8000 2023-05-24T21:11:37.524 DEBUG:boto:Params: {} 2023-05-24T21:11:37.524 DEBUG:boto:Token: None 2023-05-24T21:11:37.524 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/ 2023-05-24T21:11:37.525 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Yq7x3Vk2BaiZhXUVMszjtFsbHVw= 2023-05-24T21:11:37.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Yq7x3Vk2BaiZhXUVMszjtFsbHVw=', 'Content-Length': '0'} 2023-05-24T21:11:37.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012c984490cf1ed05-00646e7d89-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.528 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:37.529 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.529 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.529 DEBUG:boto:Method: HEAD 2023-05-24T21:11:37.530 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:11:37.530 DEBUG:boto:Data: 2023-05-24T21:11:37.530 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.531 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:37.531 DEBUG:boto:Port: 8001 2023-05-24T21:11:37.531 DEBUG:boto:Params: {} 2023-05-24T21:11:37.532 DEBUG:boto:Token: None 2023-05-24T21:11:37.532 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj 2023-05-24T21:11:37.533 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:trVDb6z7e9C2FiTIR87GPhzUXuE= 2023-05-24T21:11:37.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:trVDb6z7e9C2FiTIR87GPhzUXuE=', 'Content-Length': '0'} 2023-05-24T21:11:37.536 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bd352f7f68b0c89f-00646e7d89-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.536 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.537 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.537 DEBUG:boto:Method: HEAD 2023-05-24T21:11:37.538 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:11:37.538 DEBUG:boto:Data: 2023-05-24T21:11:37.538 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.539 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:37.539 DEBUG:boto:Port: 8000 2023-05-24T21:11:37.539 DEBUG:boto:Params: {} 2023-05-24T21:11:37.540 DEBUG:boto:Token: None 2023-05-24T21:11:37.540 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj 2023-05-24T21:11:37.540 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:trVDb6z7e9C2FiTIR87GPhzUXuE= 2023-05-24T21:11:37.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:trVDb6z7e9C2FiTIR87GPhzUXuE=', 'Content-Length': '0'} 2023-05-24T21:11:37.544 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002f97f06001ed4fdd-00646e7d89-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:37.544 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.545 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.545 DEBUG:boto:path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:37.546 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:37.546 DEBUG:boto:Method: GET 2023-05-24T21:11:37.546 DEBUG:boto:Path: /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:37.547 DEBUG:boto:Data: 2023-05-24T21:11:37.547 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.547 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:37.548 DEBUG:boto:Port: 8001 2023-05-24T21:11:37.548 DEBUG:boto:Params: {} 2023-05-24T21:11:37.548 DEBUG:boto:Token: None 2023-05-24T21:11:37.549 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:37.549 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:qL19ViArSYW5jPKAJUwyd8qHF20= 2023-05-24T21:11:37.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qL19ViArSYW5jPKAJUwyd8qHF20=', 'Content-Length': '0'} 2023-05-24T21:11:37.551 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c01b327472b14711-00646e7d89-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.552 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.552 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.553 DEBUG:boto:path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:37.553 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:37.554 DEBUG:boto:Method: GET 2023-05-24T21:11:37.554 DEBUG:boto:Path: /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:37.554 DEBUG:boto:Data: 2023-05-24T21:11:37.555 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.555 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:37.555 DEBUG:boto:Port: 8000 2023-05-24T21:11:37.556 DEBUG:boto:Params: {} 2023-05-24T21:11:37.556 DEBUG:boto:Token: None 2023-05-24T21:11:37.557 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:37.557 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:qL19ViArSYW5jPKAJUwyd8qHF20= 2023-05-24T21:11:37.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qL19ViArSYW5jPKAJUwyd8qHF20=', 'Content-Length': '0'} 2023-05-24T21:11:37.559 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000096bc41f5e831aeb-00646e7d89-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.560 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:37.560 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:37.561 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:37.561 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:37.561 DEBUG:boto:Method: GET 2023-05-24T21:11:37.562 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:37.562 DEBUG:boto:Data: 2023-05-24T21:11:37.562 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.563 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:37.563 DEBUG:boto:Port: 8001 2023-05-24T21:11:37.564 DEBUG:boto:Params: {} 2023-05-24T21:11:37.564 DEBUG:boto:Token: None 2023-05-24T21:11:37.564 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/ 2023-05-24T21:11:37.565 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Yq7x3Vk2BaiZhXUVMszjtFsbHVw= 2023-05-24T21:11:37.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Yq7x3Vk2BaiZhXUVMszjtFsbHVw=', 'Content-Length': '0'} 2023-05-24T21:11:37.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c70088d24c2e5f50-00646e7d89-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.568 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:37.569 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:37.570 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:37.570 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:37.570 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:37.571 DEBUG:boto:Method: GET 2023-05-24T21:11:37.571 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:37.572 DEBUG:boto:Data: 2023-05-24T21:11:37.572 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.572 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:37.573 DEBUG:boto:Port: 8000 2023-05-24T21:11:37.573 DEBUG:boto:Params: {} 2023-05-24T21:11:37.573 DEBUG:boto:Token: None 2023-05-24T21:11:37.574 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/ 2023-05-24T21:11:37.574 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Yq7x3Vk2BaiZhXUVMszjtFsbHVw= 2023-05-24T21:11:37.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Yq7x3Vk2BaiZhXUVMszjtFsbHVw=', 'Content-Length': '0'} 2023-05-24T21:11:37.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007467a9a931ca8724-00646e7d89-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.578 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:37.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:37.579 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.579 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.579 DEBUG:boto:path=/qskrps-13/obj?versionId=QzHAWShKdleKYlXrNH1akYPxtOhaV6z 2023-05-24T21:11:37.580 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=QzHAWShKdleKYlXrNH1akYPxtOhaV6z 2023-05-24T21:11:37.580 DEBUG:boto:Method: GET 2023-05-24T21:11:37.581 DEBUG:boto:Path: /qskrps-13/obj?versionId=QzHAWShKdleKYlXrNH1akYPxtOhaV6z 2023-05-24T21:11:37.581 DEBUG:boto:Data: 2023-05-24T21:11:37.581 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.582 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:37.582 DEBUG:boto:Port: 8001 2023-05-24T21:11:37.582 DEBUG:boto:Params: {} 2023-05-24T21:11:37.583 DEBUG:boto:Token: None 2023-05-24T21:11:37.583 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj?versionId=QzHAWShKdleKYlXrNH1akYPxtOhaV6z 2023-05-24T21:11:37.584 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:8Iw1U0/Y6pF0VzIzDCdP7Olklis= 2023-05-24T21:11:37.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:8Iw1U0/Y6pF0VzIzDCdP7Olklis=', 'Content-Length': '0'} 2023-05-24T21:11:37.586 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'QzHAWShKdleKYlXrNH1akYPxtOhaV6z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008e71b0605a20bb0c-00646e7d89-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.587 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.587 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.588 DEBUG:boto:path=/qskrps-13/obj?versionId=QzHAWShKdleKYlXrNH1akYPxtOhaV6z 2023-05-24T21:11:37.588 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=QzHAWShKdleKYlXrNH1akYPxtOhaV6z 2023-05-24T21:11:37.589 DEBUG:boto:Method: GET 2023-05-24T21:11:37.589 DEBUG:boto:Path: /qskrps-13/obj?versionId=QzHAWShKdleKYlXrNH1akYPxtOhaV6z 2023-05-24T21:11:37.589 DEBUG:boto:Data: 2023-05-24T21:11:37.590 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.590 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:37.591 DEBUG:boto:Port: 8000 2023-05-24T21:11:37.591 DEBUG:boto:Params: {} 2023-05-24T21:11:37.591 DEBUG:boto:Token: None 2023-05-24T21:11:37.592 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj?versionId=QzHAWShKdleKYlXrNH1akYPxtOhaV6z 2023-05-24T21:11:37.592 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:8Iw1U0/Y6pF0VzIzDCdP7Olklis= 2023-05-24T21:11:37.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:8Iw1U0/Y6pF0VzIzDCdP7Olklis=', 'Content-Length': '0'} 2023-05-24T21:11:37.595 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'QzHAWShKdleKYlXrNH1akYPxtOhaV6z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ddbfbe8db7f332fe-00646e7d89-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.595 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.596 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.596 DEBUG:boto:path=/qskrps-13/obj?versionId=QzHAWShKdleKYlXrNH1akYPxtOhaV6z 2023-05-24T21:11:37.596 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=QzHAWShKdleKYlXrNH1akYPxtOhaV6z 2023-05-24T21:11:37.597 DEBUG:boto:Method: HEAD 2023-05-24T21:11:37.597 DEBUG:boto:Path: /qskrps-13/obj?versionId=QzHAWShKdleKYlXrNH1akYPxtOhaV6z 2023-05-24T21:11:37.597 DEBUG:boto:Data: 2023-05-24T21:11:37.598 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.598 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:37.598 DEBUG:boto:Port: 8001 2023-05-24T21:11:37.599 DEBUG:boto:Params: {} 2023-05-24T21:11:37.599 DEBUG:boto:Token: None 2023-05-24T21:11:37.600 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj?versionId=QzHAWShKdleKYlXrNH1akYPxtOhaV6z 2023-05-24T21:11:37.600 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:k6mPbPdPVR3Bg8i5DWZKS7V0N2Y= 2023-05-24T21:11:37.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:k6mPbPdPVR3Bg8i5DWZKS7V0N2Y=', 'Content-Length': '0'} 2023-05-24T21:11:37.602 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'QzHAWShKdleKYlXrNH1akYPxtOhaV6z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006b0390c0c21c0f27-00646e7d89-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.603 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.603 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.604 DEBUG:boto:path=/qskrps-13/obj?versionId=QzHAWShKdleKYlXrNH1akYPxtOhaV6z 2023-05-24T21:11:37.604 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=QzHAWShKdleKYlXrNH1akYPxtOhaV6z 2023-05-24T21:11:37.604 DEBUG:boto:Method: HEAD 2023-05-24T21:11:37.605 DEBUG:boto:Path: /qskrps-13/obj?versionId=QzHAWShKdleKYlXrNH1akYPxtOhaV6z 2023-05-24T21:11:37.605 DEBUG:boto:Data: 2023-05-24T21:11:37.605 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.606 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:37.606 DEBUG:boto:Port: 8000 2023-05-24T21:11:37.606 DEBUG:boto:Params: {} 2023-05-24T21:11:37.607 DEBUG:boto:Token: None 2023-05-24T21:11:37.607 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj?versionId=QzHAWShKdleKYlXrNH1akYPxtOhaV6z 2023-05-24T21:11:37.608 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:k6mPbPdPVR3Bg8i5DWZKS7V0N2Y= 2023-05-24T21:11:37.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:k6mPbPdPVR3Bg8i5DWZKS7V0N2Y=', 'Content-Length': '0'} 2023-05-24T21:11:37.610 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'QzHAWShKdleKYlXrNH1akYPxtOhaV6z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000029062892028dae1-00646e7d89-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:37.611 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.611 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.612 DEBUG:boto:path=/qskrps-13/obj?versionId=QzHAWShKdleKYlXrNH1akYPxtOhaV6z 2023-05-24T21:11:37.612 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=QzHAWShKdleKYlXrNH1akYPxtOhaV6z 2023-05-24T21:11:37.612 DEBUG:boto:Method: GET 2023-05-24T21:11:37.613 DEBUG:boto:Path: /qskrps-13/obj?versionId=QzHAWShKdleKYlXrNH1akYPxtOhaV6z 2023-05-24T21:11:37.613 DEBUG:boto:Data: 2023-05-24T21:11:37.613 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.614 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:37.614 DEBUG:boto:Port: 8001 2023-05-24T21:11:37.615 DEBUG:boto:Params: {} 2023-05-24T21:11:37.615 DEBUG:boto:Token: None 2023-05-24T21:11:37.615 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj?versionId=QzHAWShKdleKYlXrNH1akYPxtOhaV6z 2023-05-24T21:11:37.616 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:8Iw1U0/Y6pF0VzIzDCdP7Olklis= 2023-05-24T21:11:37.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:8Iw1U0/Y6pF0VzIzDCdP7Olklis=', 'Content-Length': '0'} 2023-05-24T21:11:37.618 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'QzHAWShKdleKYlXrNH1akYPxtOhaV6z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004d2a75bb08860ecd-00646e7d89-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.619 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.619 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.619 DEBUG:boto:path=/qskrps-13/obj?versionId=QzHAWShKdleKYlXrNH1akYPxtOhaV6z 2023-05-24T21:11:37.620 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=QzHAWShKdleKYlXrNH1akYPxtOhaV6z 2023-05-24T21:11:37.620 DEBUG:boto:Method: GET 2023-05-24T21:11:37.621 DEBUG:boto:Path: /qskrps-13/obj?versionId=QzHAWShKdleKYlXrNH1akYPxtOhaV6z 2023-05-24T21:11:37.621 DEBUG:boto:Data: 2023-05-24T21:11:37.621 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.622 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:37.622 DEBUG:boto:Port: 8000 2023-05-24T21:11:37.622 DEBUG:boto:Params: {} 2023-05-24T21:11:37.623 DEBUG:boto:Token: None 2023-05-24T21:11:37.623 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj?versionId=QzHAWShKdleKYlXrNH1akYPxtOhaV6z 2023-05-24T21:11:37.624 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:8Iw1U0/Y6pF0VzIzDCdP7Olklis= 2023-05-24T21:11:37.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:8Iw1U0/Y6pF0VzIzDCdP7Olklis=', 'Content-Length': '0'} 2023-05-24T21:11:37.626 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'QzHAWShKdleKYlXrNH1akYPxtOhaV6z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b3dfd01f0634cba7-00646e7d89-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.627 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:37.627 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:37.627 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:37.628 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:37.628 DEBUG:boto:Method: GET 2023-05-24T21:11:37.629 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:37.629 DEBUG:boto:Data: 2023-05-24T21:11:37.629 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.630 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:37.630 DEBUG:boto:Port: 8001 2023-05-24T21:11:37.630 DEBUG:boto:Params: {} 2023-05-24T21:11:37.631 DEBUG:boto:Token: None 2023-05-24T21:11:37.631 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/ 2023-05-24T21:11:37.632 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Yq7x3Vk2BaiZhXUVMszjtFsbHVw= 2023-05-24T21:11:37.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Yq7x3Vk2BaiZhXUVMszjtFsbHVw=', 'Content-Length': '0'} 2023-05-24T21:11:37.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d449e477ad489b48-00646e7d89-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.635 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:37.636 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:37.636 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:37.637 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:37.637 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:37.637 DEBUG:boto:Method: GET 2023-05-24T21:11:37.638 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:37.638 DEBUG:boto:Data: 2023-05-24T21:11:37.639 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.639 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:37.639 DEBUG:boto:Port: 8000 2023-05-24T21:11:37.640 DEBUG:boto:Params: {} 2023-05-24T21:11:37.640 DEBUG:boto:Token: None 2023-05-24T21:11:37.640 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/ 2023-05-24T21:11:37.641 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Yq7x3Vk2BaiZhXUVMszjtFsbHVw= 2023-05-24T21:11:37.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Yq7x3Vk2BaiZhXUVMszjtFsbHVw=', 'Content-Length': '0'} 2023-05-24T21:11:37.644 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005808b046e4b8aa6b-00646e7d89-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.644 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:37.645 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.645 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.646 DEBUG:boto:Method: HEAD 2023-05-24T21:11:37.646 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:11:37.646 DEBUG:boto:Data: 2023-05-24T21:11:37.647 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.647 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:37.647 DEBUG:boto:Port: 8001 2023-05-24T21:11:37.648 DEBUG:boto:Params: {} 2023-05-24T21:11:37.648 DEBUG:boto:Token: None 2023-05-24T21:11:37.648 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj 2023-05-24T21:11:37.649 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:trVDb6z7e9C2FiTIR87GPhzUXuE= 2023-05-24T21:11:37.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:trVDb6z7e9C2FiTIR87GPhzUXuE=', 'Content-Length': '0'} 2023-05-24T21:11:37.652 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000094dd867a63dbf326-00646e7d89-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.653 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.653 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.654 DEBUG:boto:Method: HEAD 2023-05-24T21:11:37.654 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:11:37.654 DEBUG:boto:Data: 2023-05-24T21:11:37.655 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.655 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:37.655 DEBUG:boto:Port: 8000 2023-05-24T21:11:37.656 DEBUG:boto:Params: {} 2023-05-24T21:11:37.656 DEBUG:boto:Token: None 2023-05-24T21:11:37.657 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj 2023-05-24T21:11:37.657 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:trVDb6z7e9C2FiTIR87GPhzUXuE= 2023-05-24T21:11:37.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:trVDb6z7e9C2FiTIR87GPhzUXuE=', 'Content-Length': '0'} 2023-05-24T21:11:37.660 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a6cf965a71f47df4-00646e7d89-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:37.661 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.662 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.662 DEBUG:boto:path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:37.663 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:37.663 DEBUG:boto:Method: GET 2023-05-24T21:11:37.663 DEBUG:boto:Path: /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:37.664 DEBUG:boto:Data: 2023-05-24T21:11:37.664 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.664 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:37.665 DEBUG:boto:Port: 8001 2023-05-24T21:11:37.665 DEBUG:boto:Params: {} 2023-05-24T21:11:37.666 DEBUG:boto:Token: None 2023-05-24T21:11:37.666 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:37.666 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:qL19ViArSYW5jPKAJUwyd8qHF20= 2023-05-24T21:11:37.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qL19ViArSYW5jPKAJUwyd8qHF20=', 'Content-Length': '0'} 2023-05-24T21:11:37.669 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c1330710f560d2ed-00646e7d89-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.670 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.670 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.670 DEBUG:boto:path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:37.671 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:37.671 DEBUG:boto:Method: GET 2023-05-24T21:11:37.671 DEBUG:boto:Path: /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:37.672 DEBUG:boto:Data: 2023-05-24T21:11:37.672 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.673 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:37.673 DEBUG:boto:Port: 8000 2023-05-24T21:11:37.673 DEBUG:boto:Params: {} 2023-05-24T21:11:37.674 DEBUG:boto:Token: None 2023-05-24T21:11:37.674 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:37.675 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:qL19ViArSYW5jPKAJUwyd8qHF20= 2023-05-24T21:11:37.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qL19ViArSYW5jPKAJUwyd8qHF20=', 'Content-Length': '0'} 2023-05-24T21:11:37.677 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000292053b3ee2dfb83-00646e7d89-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.720 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:37.721 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:37.721 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:37.722 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:37.722 DEBUG:boto:Method: GET 2023-05-24T21:11:37.723 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:37.723 DEBUG:boto:Data: 2023-05-24T21:11:37.723 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.724 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:37.724 DEBUG:boto:Port: 8001 2023-05-24T21:11:37.724 DEBUG:boto:Params: {} 2023-05-24T21:11:37.725 DEBUG:boto:Token: None 2023-05-24T21:11:37.725 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/ 2023-05-24T21:11:37.726 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Yq7x3Vk2BaiZhXUVMszjtFsbHVw= 2023-05-24T21:11:37.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Yq7x3Vk2BaiZhXUVMszjtFsbHVw=', 'Content-Length': '0'} 2023-05-24T21:11:37.729 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000caec15cbf769119a-00646e7d89-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.730 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:37.730 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:37.731 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:37.731 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:37.731 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:37.732 DEBUG:boto:Method: GET 2023-05-24T21:11:37.732 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:37.733 DEBUG:boto:Data: 2023-05-24T21:11:37.733 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.733 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:37.734 DEBUG:boto:Port: 8000 2023-05-24T21:11:37.734 DEBUG:boto:Params: {} 2023-05-24T21:11:37.735 DEBUG:boto:Token: None 2023-05-24T21:11:37.735 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/ 2023-05-24T21:11:37.736 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Yq7x3Vk2BaiZhXUVMszjtFsbHVw= 2023-05-24T21:11:37.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Yq7x3Vk2BaiZhXUVMszjtFsbHVw=', 'Content-Length': '0'} 2023-05-24T21:11:37.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf07a710c2b7211d-00646e7d89-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.739 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:37.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:37.740 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.740 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.741 DEBUG:boto:path=/qskrps-13/obj?versionId=b9BUNjISbr2Ry0Y4wv-brHNozZY03Qi 2023-05-24T21:11:37.741 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=b9BUNjISbr2Ry0Y4wv-brHNozZY03Qi 2023-05-24T21:11:37.741 DEBUG:boto:Method: GET 2023-05-24T21:11:37.742 DEBUG:boto:Path: /qskrps-13/obj?versionId=b9BUNjISbr2Ry0Y4wv-brHNozZY03Qi 2023-05-24T21:11:37.742 DEBUG:boto:Data: 2023-05-24T21:11:37.742 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.743 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:37.743 DEBUG:boto:Port: 8001 2023-05-24T21:11:37.744 DEBUG:boto:Params: {} 2023-05-24T21:11:37.744 DEBUG:boto:Token: None 2023-05-24T21:11:37.744 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj?versionId=b9BUNjISbr2Ry0Y4wv-brHNozZY03Qi 2023-05-24T21:11:37.745 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:1LeZtTpxy2Nf+CIjbvrD0FEqDGk= 2023-05-24T21:11:37.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:1LeZtTpxy2Nf+CIjbvrD0FEqDGk=', 'Content-Length': '0'} 2023-05-24T21:11:37.747 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'b9BUNjISbr2Ry0Y4wv-brHNozZY03Qi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000095ea3db7b34d6481-00646e7d89-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.788 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.789 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.789 DEBUG:boto:path=/qskrps-13/obj?versionId=b9BUNjISbr2Ry0Y4wv-brHNozZY03Qi 2023-05-24T21:11:37.790 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=b9BUNjISbr2Ry0Y4wv-brHNozZY03Qi 2023-05-24T21:11:37.790 DEBUG:boto:Method: GET 2023-05-24T21:11:37.790 DEBUG:boto:Path: /qskrps-13/obj?versionId=b9BUNjISbr2Ry0Y4wv-brHNozZY03Qi 2023-05-24T21:11:37.791 DEBUG:boto:Data: 2023-05-24T21:11:37.791 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.792 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:37.792 DEBUG:boto:Port: 8000 2023-05-24T21:11:37.792 DEBUG:boto:Params: {} 2023-05-24T21:11:37.793 DEBUG:boto:Token: None 2023-05-24T21:11:37.793 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj?versionId=b9BUNjISbr2Ry0Y4wv-brHNozZY03Qi 2023-05-24T21:11:37.794 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:1LeZtTpxy2Nf+CIjbvrD0FEqDGk= 2023-05-24T21:11:37.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:1LeZtTpxy2Nf+CIjbvrD0FEqDGk=', 'Content-Length': '0'} 2023-05-24T21:11:37.796 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'b9BUNjISbr2Ry0Y4wv-brHNozZY03Qi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d64943536ea49ced-00646e7d89-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.797 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.798 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.798 DEBUG:boto:path=/qskrps-13/obj?versionId=b9BUNjISbr2Ry0Y4wv-brHNozZY03Qi 2023-05-24T21:11:37.798 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=b9BUNjISbr2Ry0Y4wv-brHNozZY03Qi 2023-05-24T21:11:37.799 DEBUG:boto:Method: HEAD 2023-05-24T21:11:37.799 DEBUG:boto:Path: /qskrps-13/obj?versionId=b9BUNjISbr2Ry0Y4wv-brHNozZY03Qi 2023-05-24T21:11:37.799 DEBUG:boto:Data: 2023-05-24T21:11:37.800 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.800 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:37.800 DEBUG:boto:Port: 8001 2023-05-24T21:11:37.801 DEBUG:boto:Params: {} 2023-05-24T21:11:37.801 DEBUG:boto:Token: None 2023-05-24T21:11:37.801 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj?versionId=b9BUNjISbr2Ry0Y4wv-brHNozZY03Qi 2023-05-24T21:11:37.802 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:FEo+NoaiQVJ9dZK6JbvCm6KJAeY= 2023-05-24T21:11:37.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:FEo+NoaiQVJ9dZK6JbvCm6KJAeY=', 'Content-Length': '0'} 2023-05-24T21:11:37.804 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'b9BUNjISbr2Ry0Y4wv-brHNozZY03Qi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000034337440eb5d96a1-00646e7d89-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.804 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.805 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.805 DEBUG:boto:path=/qskrps-13/obj?versionId=b9BUNjISbr2Ry0Y4wv-brHNozZY03Qi 2023-05-24T21:11:37.806 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=b9BUNjISbr2Ry0Y4wv-brHNozZY03Qi 2023-05-24T21:11:37.806 DEBUG:boto:Method: HEAD 2023-05-24T21:11:37.806 DEBUG:boto:Path: /qskrps-13/obj?versionId=b9BUNjISbr2Ry0Y4wv-brHNozZY03Qi 2023-05-24T21:11:37.807 DEBUG:boto:Data: 2023-05-24T21:11:37.807 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.807 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:37.807 DEBUG:boto:Port: 8000 2023-05-24T21:11:37.808 DEBUG:boto:Params: {} 2023-05-24T21:11:37.808 DEBUG:boto:Token: None 2023-05-24T21:11:37.809 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj?versionId=b9BUNjISbr2Ry0Y4wv-brHNozZY03Qi 2023-05-24T21:11:37.809 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:FEo+NoaiQVJ9dZK6JbvCm6KJAeY= 2023-05-24T21:11:37.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:FEo+NoaiQVJ9dZK6JbvCm6KJAeY=', 'Content-Length': '0'} 2023-05-24T21:11:37.811 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'b9BUNjISbr2Ry0Y4wv-brHNozZY03Qi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000002f4770a7394cb91-00646e7d89-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:37.812 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.813 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.813 DEBUG:boto:path=/qskrps-13/obj?versionId=b9BUNjISbr2Ry0Y4wv-brHNozZY03Qi 2023-05-24T21:11:37.813 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=b9BUNjISbr2Ry0Y4wv-brHNozZY03Qi 2023-05-24T21:11:37.814 DEBUG:boto:Method: GET 2023-05-24T21:11:37.814 DEBUG:boto:Path: /qskrps-13/obj?versionId=b9BUNjISbr2Ry0Y4wv-brHNozZY03Qi 2023-05-24T21:11:37.815 DEBUG:boto:Data: 2023-05-24T21:11:37.815 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.815 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:37.816 DEBUG:boto:Port: 8001 2023-05-24T21:11:37.816 DEBUG:boto:Params: {} 2023-05-24T21:11:37.816 DEBUG:boto:Token: None 2023-05-24T21:11:37.817 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj?versionId=b9BUNjISbr2Ry0Y4wv-brHNozZY03Qi 2023-05-24T21:11:37.817 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:1LeZtTpxy2Nf+CIjbvrD0FEqDGk= 2023-05-24T21:11:37.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:1LeZtTpxy2Nf+CIjbvrD0FEqDGk=', 'Content-Length': '0'} 2023-05-24T21:11:37.820 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'b9BUNjISbr2Ry0Y4wv-brHNozZY03Qi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cd4e87df883e082f-00646e7d89-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.820 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.821 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.821 DEBUG:boto:path=/qskrps-13/obj?versionId=b9BUNjISbr2Ry0Y4wv-brHNozZY03Qi 2023-05-24T21:11:37.821 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=b9BUNjISbr2Ry0Y4wv-brHNozZY03Qi 2023-05-24T21:11:37.822 DEBUG:boto:Method: GET 2023-05-24T21:11:37.822 DEBUG:boto:Path: /qskrps-13/obj?versionId=b9BUNjISbr2Ry0Y4wv-brHNozZY03Qi 2023-05-24T21:11:37.823 DEBUG:boto:Data: 2023-05-24T21:11:37.823 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.823 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:37.824 DEBUG:boto:Port: 8000 2023-05-24T21:11:37.824 DEBUG:boto:Params: {} 2023-05-24T21:11:37.824 DEBUG:boto:Token: None 2023-05-24T21:11:37.825 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj?versionId=b9BUNjISbr2Ry0Y4wv-brHNozZY03Qi 2023-05-24T21:11:37.825 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:1LeZtTpxy2Nf+CIjbvrD0FEqDGk= 2023-05-24T21:11:37.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:1LeZtTpxy2Nf+CIjbvrD0FEqDGk=', 'Content-Length': '0'} 2023-05-24T21:11:37.828 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'b9BUNjISbr2Ry0Y4wv-brHNozZY03Qi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000010bad554712c3abc-00646e7d89-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.828 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:37.829 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:37.829 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:37.830 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:37.830 DEBUG:boto:Method: GET 2023-05-24T21:11:37.830 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:37.831 DEBUG:boto:Data: 2023-05-24T21:11:37.831 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.832 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:37.832 DEBUG:boto:Port: 8001 2023-05-24T21:11:37.832 DEBUG:boto:Params: {} 2023-05-24T21:11:37.833 DEBUG:boto:Token: None 2023-05-24T21:11:37.833 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/ 2023-05-24T21:11:37.834 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Yq7x3Vk2BaiZhXUVMszjtFsbHVw= 2023-05-24T21:11:37.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Yq7x3Vk2BaiZhXUVMszjtFsbHVw=', 'Content-Length': '0'} 2023-05-24T21:11:37.836 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c87c204f7886c6c-00646e7d89-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.837 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:37.838 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:37.838 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:37.838 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:37.839 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:37.839 DEBUG:boto:Method: GET 2023-05-24T21:11:37.840 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:37.840 DEBUG:boto:Data: 2023-05-24T21:11:37.840 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.841 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:37.841 DEBUG:boto:Port: 8000 2023-05-24T21:11:37.841 DEBUG:boto:Params: {} 2023-05-24T21:11:37.842 DEBUG:boto:Token: None 2023-05-24T21:11:37.842 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/ 2023-05-24T21:11:37.843 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Yq7x3Vk2BaiZhXUVMszjtFsbHVw= 2023-05-24T21:11:37.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Yq7x3Vk2BaiZhXUVMszjtFsbHVw=', 'Content-Length': '0'} 2023-05-24T21:11:37.845 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d152090ffc5d35d2-00646e7d89-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.846 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:37.846 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.847 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.847 DEBUG:boto:Method: HEAD 2023-05-24T21:11:37.848 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:11:37.848 DEBUG:boto:Data: 2023-05-24T21:11:37.848 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.849 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:37.849 DEBUG:boto:Port: 8001 2023-05-24T21:11:37.849 DEBUG:boto:Params: {} 2023-05-24T21:11:37.850 DEBUG:boto:Token: None 2023-05-24T21:11:37.850 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj 2023-05-24T21:11:37.851 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:trVDb6z7e9C2FiTIR87GPhzUXuE= 2023-05-24T21:11:37.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:trVDb6z7e9C2FiTIR87GPhzUXuE=', 'Content-Length': '0'} 2023-05-24T21:11:37.854 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000da3781c17b75bbb-00646e7d89-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.855 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.855 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.855 DEBUG:boto:Method: HEAD 2023-05-24T21:11:37.856 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:11:37.856 DEBUG:boto:Data: 2023-05-24T21:11:37.856 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.857 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:37.857 DEBUG:boto:Port: 8000 2023-05-24T21:11:37.857 DEBUG:boto:Params: {} 2023-05-24T21:11:37.858 DEBUG:boto:Token: None 2023-05-24T21:11:37.858 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj 2023-05-24T21:11:37.859 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:trVDb6z7e9C2FiTIR87GPhzUXuE= 2023-05-24T21:11:37.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:trVDb6z7e9C2FiTIR87GPhzUXuE=', 'Content-Length': '0'} 2023-05-24T21:11:37.862 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cec0a278d6d6468a-00646e7d89-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:37.863 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.863 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.863 DEBUG:boto:path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:37.864 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:37.864 DEBUG:boto:Method: GET 2023-05-24T21:11:37.865 DEBUG:boto:Path: /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:37.865 DEBUG:boto:Data: 2023-05-24T21:11:37.865 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.866 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:37.866 DEBUG:boto:Port: 8001 2023-05-24T21:11:37.866 DEBUG:boto:Params: {} 2023-05-24T21:11:37.867 DEBUG:boto:Token: None 2023-05-24T21:11:37.867 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:37.868 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:qL19ViArSYW5jPKAJUwyd8qHF20= 2023-05-24T21:11:37.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qL19ViArSYW5jPKAJUwyd8qHF20=', 'Content-Length': '0'} 2023-05-24T21:11:37.870 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b35c2c0c0331a4a2-00646e7d89-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.912 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.913 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.913 DEBUG:boto:path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:37.914 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:37.914 DEBUG:boto:Method: GET 2023-05-24T21:11:37.914 DEBUG:boto:Path: /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:37.915 DEBUG:boto:Data: 2023-05-24T21:11:37.915 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.915 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:37.916 DEBUG:boto:Port: 8000 2023-05-24T21:11:37.916 DEBUG:boto:Params: {} 2023-05-24T21:11:37.917 DEBUG:boto:Token: None 2023-05-24T21:11:37.917 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:37.918 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:qL19ViArSYW5jPKAJUwyd8qHF20= 2023-05-24T21:11:37.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qL19ViArSYW5jPKAJUwyd8qHF20=', 'Content-Length': '0'} 2023-05-24T21:11:37.920 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000b81e6e74cb54139-00646e7d89-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.921 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:37.921 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:37.921 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:37.922 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:37.922 DEBUG:boto:Method: GET 2023-05-24T21:11:37.923 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:37.923 DEBUG:boto:Data: 2023-05-24T21:11:37.923 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.924 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:37.924 DEBUG:boto:Port: 8001 2023-05-24T21:11:37.924 DEBUG:boto:Params: {} 2023-05-24T21:11:37.925 DEBUG:boto:Token: None 2023-05-24T21:11:37.925 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/ 2023-05-24T21:11:37.926 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Yq7x3Vk2BaiZhXUVMszjtFsbHVw= 2023-05-24T21:11:37.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Yq7x3Vk2BaiZhXUVMszjtFsbHVw=', 'Content-Length': '0'} 2023-05-24T21:11:37.928 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004eea4125af0e5ce1-00646e7d89-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.929 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:37.930 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:37.930 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:37.930 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:37.931 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:37.931 DEBUG:boto:Method: GET 2023-05-24T21:11:37.932 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:37.932 DEBUG:boto:Data: 2023-05-24T21:11:37.932 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.933 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:37.933 DEBUG:boto:Port: 8000 2023-05-24T21:11:37.933 DEBUG:boto:Params: {} 2023-05-24T21:11:37.934 DEBUG:boto:Token: None 2023-05-24T21:11:37.934 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/ 2023-05-24T21:11:37.935 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Yq7x3Vk2BaiZhXUVMszjtFsbHVw= 2023-05-24T21:11:37.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Yq7x3Vk2BaiZhXUVMszjtFsbHVw=', 'Content-Length': '0'} 2023-05-24T21:11:37.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad2ec61efee9e0ca-00646e7d89-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.938 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:37.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:37.939 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.940 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.940 DEBUG:boto:path=/qskrps-13/obj?versionId=U-3Niwfe1WdmuD.qX1nCRvbymdYC9Ns 2023-05-24T21:11:37.940 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=U-3Niwfe1WdmuD.qX1nCRvbymdYC9Ns 2023-05-24T21:11:37.941 DEBUG:boto:Method: GET 2023-05-24T21:11:37.941 DEBUG:boto:Path: /qskrps-13/obj?versionId=U-3Niwfe1WdmuD.qX1nCRvbymdYC9Ns 2023-05-24T21:11:37.942 DEBUG:boto:Data: 2023-05-24T21:11:37.942 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.942 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:37.943 DEBUG:boto:Port: 8001 2023-05-24T21:11:37.943 DEBUG:boto:Params: {} 2023-05-24T21:11:37.943 DEBUG:boto:Token: None 2023-05-24T21:11:37.944 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj?versionId=U-3Niwfe1WdmuD.qX1nCRvbymdYC9Ns 2023-05-24T21:11:37.944 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:CP5MF2zVVN0VvWyrFUiiKGgxXCw= 2023-05-24T21:11:37.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:CP5MF2zVVN0VvWyrFUiiKGgxXCw=', 'Content-Length': '0'} 2023-05-24T21:11:37.947 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'U-3Niwfe1WdmuD.qX1nCRvbymdYC9Ns'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000031c5b7fe904f216a-00646e7d89-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.947 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.948 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.948 DEBUG:boto:path=/qskrps-13/obj?versionId=U-3Niwfe1WdmuD.qX1nCRvbymdYC9Ns 2023-05-24T21:11:37.948 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=U-3Niwfe1WdmuD.qX1nCRvbymdYC9Ns 2023-05-24T21:11:37.949 DEBUG:boto:Method: GET 2023-05-24T21:11:37.949 DEBUG:boto:Path: /qskrps-13/obj?versionId=U-3Niwfe1WdmuD.qX1nCRvbymdYC9Ns 2023-05-24T21:11:37.949 DEBUG:boto:Data: 2023-05-24T21:11:37.950 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.950 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:37.950 DEBUG:boto:Port: 8000 2023-05-24T21:11:37.951 DEBUG:boto:Params: {} 2023-05-24T21:11:37.951 DEBUG:boto:Token: None 2023-05-24T21:11:37.952 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj?versionId=U-3Niwfe1WdmuD.qX1nCRvbymdYC9Ns 2023-05-24T21:11:37.952 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:CP5MF2zVVN0VvWyrFUiiKGgxXCw= 2023-05-24T21:11:37.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:CP5MF2zVVN0VvWyrFUiiKGgxXCw=', 'Content-Length': '0'} 2023-05-24T21:11:37.955 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'U-3Niwfe1WdmuD.qX1nCRvbymdYC9Ns'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000547c9dac30b6369f-00646e7d89-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.955 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.956 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.956 DEBUG:boto:path=/qskrps-13/obj?versionId=U-3Niwfe1WdmuD.qX1nCRvbymdYC9Ns 2023-05-24T21:11:37.956 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=U-3Niwfe1WdmuD.qX1nCRvbymdYC9Ns 2023-05-24T21:11:37.957 DEBUG:boto:Method: HEAD 2023-05-24T21:11:37.957 DEBUG:boto:Path: /qskrps-13/obj?versionId=U-3Niwfe1WdmuD.qX1nCRvbymdYC9Ns 2023-05-24T21:11:37.957 DEBUG:boto:Data: 2023-05-24T21:11:37.958 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.958 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:37.958 DEBUG:boto:Port: 8001 2023-05-24T21:11:37.959 DEBUG:boto:Params: {} 2023-05-24T21:11:37.959 DEBUG:boto:Token: None 2023-05-24T21:11:37.959 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj?versionId=U-3Niwfe1WdmuD.qX1nCRvbymdYC9Ns 2023-05-24T21:11:37.960 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:fWhvc4cxH5tEfT0jUHqIfSZaaR0= 2023-05-24T21:11:37.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:fWhvc4cxH5tEfT0jUHqIfSZaaR0=', 'Content-Length': '0'} 2023-05-24T21:11:37.962 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'U-3Niwfe1WdmuD.qX1nCRvbymdYC9Ns'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e8fa69d0fc8c4a86-00646e7d89-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.963 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.963 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.963 DEBUG:boto:path=/qskrps-13/obj?versionId=U-3Niwfe1WdmuD.qX1nCRvbymdYC9Ns 2023-05-24T21:11:37.964 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=U-3Niwfe1WdmuD.qX1nCRvbymdYC9Ns 2023-05-24T21:11:37.964 DEBUG:boto:Method: HEAD 2023-05-24T21:11:37.965 DEBUG:boto:Path: /qskrps-13/obj?versionId=U-3Niwfe1WdmuD.qX1nCRvbymdYC9Ns 2023-05-24T21:11:37.965 DEBUG:boto:Data: 2023-05-24T21:11:37.965 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.966 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:37.966 DEBUG:boto:Port: 8000 2023-05-24T21:11:37.967 DEBUG:boto:Params: {} 2023-05-24T21:11:37.967 DEBUG:boto:Token: None 2023-05-24T21:11:37.967 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj?versionId=U-3Niwfe1WdmuD.qX1nCRvbymdYC9Ns 2023-05-24T21:11:37.968 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:fWhvc4cxH5tEfT0jUHqIfSZaaR0= 2023-05-24T21:11:37.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:fWhvc4cxH5tEfT0jUHqIfSZaaR0=', 'Content-Length': '0'} 2023-05-24T21:11:37.970 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'U-3Niwfe1WdmuD.qX1nCRvbymdYC9Ns'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000033b2afc4e1995645-00646e7d89-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:37.971 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.971 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.971 DEBUG:boto:path=/qskrps-13/obj?versionId=U-3Niwfe1WdmuD.qX1nCRvbymdYC9Ns 2023-05-24T21:11:37.972 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=U-3Niwfe1WdmuD.qX1nCRvbymdYC9Ns 2023-05-24T21:11:37.972 DEBUG:boto:Method: GET 2023-05-24T21:11:37.973 DEBUG:boto:Path: /qskrps-13/obj?versionId=U-3Niwfe1WdmuD.qX1nCRvbymdYC9Ns 2023-05-24T21:11:37.973 DEBUG:boto:Data: 2023-05-24T21:11:37.973 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.974 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:37.974 DEBUG:boto:Port: 8001 2023-05-24T21:11:37.974 DEBUG:boto:Params: {} 2023-05-24T21:11:37.975 DEBUG:boto:Token: None 2023-05-24T21:11:37.975 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj?versionId=U-3Niwfe1WdmuD.qX1nCRvbymdYC9Ns 2023-05-24T21:11:37.976 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:CP5MF2zVVN0VvWyrFUiiKGgxXCw= 2023-05-24T21:11:37.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:CP5MF2zVVN0VvWyrFUiiKGgxXCw=', 'Content-Length': '0'} 2023-05-24T21:11:37.978 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'U-3Niwfe1WdmuD.qX1nCRvbymdYC9Ns'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000be1287026c741e60-00646e7d89-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.979 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:37.979 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:37.979 DEBUG:boto:path=/qskrps-13/obj?versionId=U-3Niwfe1WdmuD.qX1nCRvbymdYC9Ns 2023-05-24T21:11:37.980 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=U-3Niwfe1WdmuD.qX1nCRvbymdYC9Ns 2023-05-24T21:11:37.980 DEBUG:boto:Method: GET 2023-05-24T21:11:37.981 DEBUG:boto:Path: /qskrps-13/obj?versionId=U-3Niwfe1WdmuD.qX1nCRvbymdYC9Ns 2023-05-24T21:11:37.981 DEBUG:boto:Data: 2023-05-24T21:11:37.981 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.982 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:37.982 DEBUG:boto:Port: 8000 2023-05-24T21:11:37.982 DEBUG:boto:Params: {} 2023-05-24T21:11:37.983 DEBUG:boto:Token: None 2023-05-24T21:11:37.983 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/obj?versionId=U-3Niwfe1WdmuD.qX1nCRvbymdYC9Ns 2023-05-24T21:11:37.984 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:CP5MF2zVVN0VvWyrFUiiKGgxXCw= 2023-05-24T21:11:37.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:CP5MF2zVVN0VvWyrFUiiKGgxXCw=', 'Content-Length': '0'} 2023-05-24T21:11:37.986 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'U-3Niwfe1WdmuD.qX1nCRvbymdYC9Ns'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000738f14701ed2f726-00646e7d89-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.987 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:37.987 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:37.987 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:37.988 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:37.988 DEBUG:boto:Method: GET 2023-05-24T21:11:37.988 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:37.989 DEBUG:boto:Data: 2023-05-24T21:11:37.989 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.989 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:37.990 DEBUG:boto:Port: 8001 2023-05-24T21:11:37.990 DEBUG:boto:Params: {} 2023-05-24T21:11:37.991 DEBUG:boto:Token: None 2023-05-24T21:11:37.991 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:37 GMT /qskrps-13/ 2023-05-24T21:11:37.992 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Yq7x3Vk2BaiZhXUVMszjtFsbHVw= 2023-05-24T21:11:37.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Yq7x3Vk2BaiZhXUVMszjtFsbHVw=', 'Content-Length': '0'} 2023-05-24T21:11:37.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e452ef1e705976a7-00646e7d89-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:37.995 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:37.996 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:37.996 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:37.996 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:37.997 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:37.997 DEBUG:boto:Method: GET 2023-05-24T21:11:37.998 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:37.998 DEBUG:boto:Data: 2023-05-24T21:11:37.998 DEBUG:boto:Headers: {} 2023-05-24T21:11:37.999 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:37.999 DEBUG:boto:Port: 8000 2023-05-24T21:11:37.999 DEBUG:boto:Params: {} 2023-05-24T21:11:38.000 DEBUG:boto:Token: None 2023-05-24T21:11:38.000 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/ 2023-05-24T21:11:38.001 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5e2RJlSUQ8Y5t134TUlreZP6Nas= 2023-05-24T21:11:38.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5e2RJlSUQ8Y5t134TUlreZP6Nas=', 'Content-Length': '0'} 2023-05-24T21:11:38.003 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000119a0a12e9aef632-00646e7d89-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.004 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:38.004 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.005 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.005 DEBUG:boto:Method: HEAD 2023-05-24T21:11:38.006 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:11:38.006 DEBUG:boto:Data: 2023-05-24T21:11:38.006 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.007 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:38.007 DEBUG:boto:Port: 8001 2023-05-24T21:11:38.007 DEBUG:boto:Params: {} 2023-05-24T21:11:38.008 DEBUG:boto:Token: None 2023-05-24T21:11:38.008 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj 2023-05-24T21:11:38.008 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ed3FFLYS1b/XkKX9hkntV4II6A8= 2023-05-24T21:11:38.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ed3FFLYS1b/XkKX9hkntV4II6A8=', 'Content-Length': '0'} 2023-05-24T21:11:38.012 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000baade92d749a7c53-00646e7d8a-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.012 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.013 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.013 DEBUG:boto:Method: HEAD 2023-05-24T21:11:38.013 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:11:38.014 DEBUG:boto:Data: 2023-05-24T21:11:38.014 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.014 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:38.015 DEBUG:boto:Port: 8000 2023-05-24T21:11:38.015 DEBUG:boto:Params: {} 2023-05-24T21:11:38.015 DEBUG:boto:Token: None 2023-05-24T21:11:38.016 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj 2023-05-24T21:11:38.016 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ed3FFLYS1b/XkKX9hkntV4II6A8= 2023-05-24T21:11:38.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ed3FFLYS1b/XkKX9hkntV4II6A8=', 'Content-Length': '0'} 2023-05-24T21:11:38.020 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000095470c6081e15a0e-00646e7d8a-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:38.021 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.021 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.021 DEBUG:boto:path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.022 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.022 DEBUG:boto:Method: GET 2023-05-24T21:11:38.022 DEBUG:boto:Path: /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.023 DEBUG:boto:Data: 2023-05-24T21:11:38.023 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.024 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:38.024 DEBUG:boto:Port: 8001 2023-05-24T21:11:38.024 DEBUG:boto:Params: {} 2023-05-24T21:11:38.025 DEBUG:boto:Token: None 2023-05-24T21:11:38.025 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.026 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Qy3zwRpx5lVG+UJdWDSmNUuL1CE= 2023-05-24T21:11:38.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Qy3zwRpx5lVG+UJdWDSmNUuL1CE=', 'Content-Length': '0'} 2023-05-24T21:11:38.028 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce3d43faf7827b97-00646e7d8a-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.029 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.029 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.029 DEBUG:boto:path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.030 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.030 DEBUG:boto:Method: GET 2023-05-24T21:11:38.030 DEBUG:boto:Path: /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.031 DEBUG:boto:Data: 2023-05-24T21:11:38.031 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.031 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:38.032 DEBUG:boto:Port: 8000 2023-05-24T21:11:38.032 DEBUG:boto:Params: {} 2023-05-24T21:11:38.033 DEBUG:boto:Token: None 2023-05-24T21:11:38.033 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.033 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Qy3zwRpx5lVG+UJdWDSmNUuL1CE= 2023-05-24T21:11:38.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Qy3zwRpx5lVG+UJdWDSmNUuL1CE=', 'Content-Length': '0'} 2023-05-24T21:11:38.035 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009e9a5769d35af0de-00646e7d8a-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.036 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:38.037 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:38.037 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:38.037 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:38.038 DEBUG:boto:Method: GET 2023-05-24T21:11:38.038 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:38.038 DEBUG:boto:Data: 2023-05-24T21:11:38.039 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.039 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:38.039 DEBUG:boto:Port: 8001 2023-05-24T21:11:38.040 DEBUG:boto:Params: {} 2023-05-24T21:11:38.040 DEBUG:boto:Token: None 2023-05-24T21:11:38.041 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/ 2023-05-24T21:11:38.041 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5e2RJlSUQ8Y5t134TUlreZP6Nas= 2023-05-24T21:11:38.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5e2RJlSUQ8Y5t134TUlreZP6Nas=', 'Content-Length': '0'} 2023-05-24T21:11:38.044 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd2c062fb5e82fb3-00646e7d8a-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.044 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:38.045 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:38.045 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:38.046 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:38.046 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:38.046 DEBUG:boto:Method: GET 2023-05-24T21:11:38.047 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:38.047 DEBUG:boto:Data: 2023-05-24T21:11:38.048 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.048 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:38.048 DEBUG:boto:Port: 8000 2023-05-24T21:11:38.049 DEBUG:boto:Params: {} 2023-05-24T21:11:38.049 DEBUG:boto:Token: None 2023-05-24T21:11:38.049 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/ 2023-05-24T21:11:38.050 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5e2RJlSUQ8Y5t134TUlreZP6Nas= 2023-05-24T21:11:38.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5e2RJlSUQ8Y5t134TUlreZP6Nas=', 'Content-Length': '0'} 2023-05-24T21:11:38.053 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dee52459afda0523-00646e7d8a-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.053 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:38.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:38.054 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.055 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.055 DEBUG:boto:path=/qskrps-13/obj?versionId=hcen3y8E05rcJbSPsSSwDQfkEHk1NI7 2023-05-24T21:11:38.055 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=hcen3y8E05rcJbSPsSSwDQfkEHk1NI7 2023-05-24T21:11:38.056 DEBUG:boto:Method: GET 2023-05-24T21:11:38.056 DEBUG:boto:Path: /qskrps-13/obj?versionId=hcen3y8E05rcJbSPsSSwDQfkEHk1NI7 2023-05-24T21:11:38.056 DEBUG:boto:Data: 2023-05-24T21:11:38.057 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.057 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:38.058 DEBUG:boto:Port: 8001 2023-05-24T21:11:38.058 DEBUG:boto:Params: {} 2023-05-24T21:11:38.058 DEBUG:boto:Token: None 2023-05-24T21:11:38.059 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj?versionId=hcen3y8E05rcJbSPsSSwDQfkEHk1NI7 2023-05-24T21:11:38.059 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:7f8PhdW2+VsLHuFb9+aAMnK8cWo= 2023-05-24T21:11:38.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:7f8PhdW2+VsLHuFb9+aAMnK8cWo=', 'Content-Length': '0'} 2023-05-24T21:11:38.061 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'hcen3y8E05rcJbSPsSSwDQfkEHk1NI7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000062138382d5f17810-00646e7d8a-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.062 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.063 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.063 DEBUG:boto:path=/qskrps-13/obj?versionId=hcen3y8E05rcJbSPsSSwDQfkEHk1NI7 2023-05-24T21:11:38.063 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=hcen3y8E05rcJbSPsSSwDQfkEHk1NI7 2023-05-24T21:11:38.064 DEBUG:boto:Method: GET 2023-05-24T21:11:38.064 DEBUG:boto:Path: /qskrps-13/obj?versionId=hcen3y8E05rcJbSPsSSwDQfkEHk1NI7 2023-05-24T21:11:38.064 DEBUG:boto:Data: 2023-05-24T21:11:38.065 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.065 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:38.065 DEBUG:boto:Port: 8000 2023-05-24T21:11:38.066 DEBUG:boto:Params: {} 2023-05-24T21:11:38.066 DEBUG:boto:Token: None 2023-05-24T21:11:38.067 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj?versionId=hcen3y8E05rcJbSPsSSwDQfkEHk1NI7 2023-05-24T21:11:38.067 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:7f8PhdW2+VsLHuFb9+aAMnK8cWo= 2023-05-24T21:11:38.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:7f8PhdW2+VsLHuFb9+aAMnK8cWo=', 'Content-Length': '0'} 2023-05-24T21:11:38.069 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'hcen3y8E05rcJbSPsSSwDQfkEHk1NI7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f2db76ee014ff60a-00646e7d8a-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.070 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.070 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.070 DEBUG:boto:path=/qskrps-13/obj?versionId=hcen3y8E05rcJbSPsSSwDQfkEHk1NI7 2023-05-24T21:11:38.071 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=hcen3y8E05rcJbSPsSSwDQfkEHk1NI7 2023-05-24T21:11:38.071 DEBUG:boto:Method: HEAD 2023-05-24T21:11:38.071 DEBUG:boto:Path: /qskrps-13/obj?versionId=hcen3y8E05rcJbSPsSSwDQfkEHk1NI7 2023-05-24T21:11:38.072 DEBUG:boto:Data: 2023-05-24T21:11:38.072 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.072 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:38.073 DEBUG:boto:Port: 8001 2023-05-24T21:11:38.073 DEBUG:boto:Params: {} 2023-05-24T21:11:38.073 DEBUG:boto:Token: None 2023-05-24T21:11:38.074 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj?versionId=hcen3y8E05rcJbSPsSSwDQfkEHk1NI7 2023-05-24T21:11:38.074 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:xlR6YMlXDPge7pguLTf0QFF6GF4= 2023-05-24T21:11:38.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:xlR6YMlXDPge7pguLTf0QFF6GF4=', 'Content-Length': '0'} 2023-05-24T21:11:38.076 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'hcen3y8E05rcJbSPsSSwDQfkEHk1NI7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000076e2aacb5aed4383-00646e7d8a-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.077 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.077 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.077 DEBUG:boto:path=/qskrps-13/obj?versionId=hcen3y8E05rcJbSPsSSwDQfkEHk1NI7 2023-05-24T21:11:38.078 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=hcen3y8E05rcJbSPsSSwDQfkEHk1NI7 2023-05-24T21:11:38.078 DEBUG:boto:Method: HEAD 2023-05-24T21:11:38.078 DEBUG:boto:Path: /qskrps-13/obj?versionId=hcen3y8E05rcJbSPsSSwDQfkEHk1NI7 2023-05-24T21:11:38.079 DEBUG:boto:Data: 2023-05-24T21:11:38.079 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.079 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:38.080 DEBUG:boto:Port: 8000 2023-05-24T21:11:38.080 DEBUG:boto:Params: {} 2023-05-24T21:11:38.080 DEBUG:boto:Token: None 2023-05-24T21:11:38.081 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj?versionId=hcen3y8E05rcJbSPsSSwDQfkEHk1NI7 2023-05-24T21:11:38.081 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:xlR6YMlXDPge7pguLTf0QFF6GF4= 2023-05-24T21:11:38.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:xlR6YMlXDPge7pguLTf0QFF6GF4=', 'Content-Length': '0'} 2023-05-24T21:11:38.083 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'hcen3y8E05rcJbSPsSSwDQfkEHk1NI7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a793ecd1a2f46223-00646e7d8a-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:38.084 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.085 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.085 DEBUG:boto:path=/qskrps-13/obj?versionId=hcen3y8E05rcJbSPsSSwDQfkEHk1NI7 2023-05-24T21:11:38.085 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=hcen3y8E05rcJbSPsSSwDQfkEHk1NI7 2023-05-24T21:11:38.086 DEBUG:boto:Method: GET 2023-05-24T21:11:38.086 DEBUG:boto:Path: /qskrps-13/obj?versionId=hcen3y8E05rcJbSPsSSwDQfkEHk1NI7 2023-05-24T21:11:38.086 DEBUG:boto:Data: 2023-05-24T21:11:38.087 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.087 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:38.087 DEBUG:boto:Port: 8001 2023-05-24T21:11:38.088 DEBUG:boto:Params: {} 2023-05-24T21:11:38.088 DEBUG:boto:Token: None 2023-05-24T21:11:38.089 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj?versionId=hcen3y8E05rcJbSPsSSwDQfkEHk1NI7 2023-05-24T21:11:38.089 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:7f8PhdW2+VsLHuFb9+aAMnK8cWo= 2023-05-24T21:11:38.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:7f8PhdW2+VsLHuFb9+aAMnK8cWo=', 'Content-Length': '0'} 2023-05-24T21:11:38.091 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'hcen3y8E05rcJbSPsSSwDQfkEHk1NI7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001507f26725571dfe-00646e7d8a-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.092 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.092 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.093 DEBUG:boto:path=/qskrps-13/obj?versionId=hcen3y8E05rcJbSPsSSwDQfkEHk1NI7 2023-05-24T21:11:38.093 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=hcen3y8E05rcJbSPsSSwDQfkEHk1NI7 2023-05-24T21:11:38.094 DEBUG:boto:Method: GET 2023-05-24T21:11:38.094 DEBUG:boto:Path: /qskrps-13/obj?versionId=hcen3y8E05rcJbSPsSSwDQfkEHk1NI7 2023-05-24T21:11:38.094 DEBUG:boto:Data: 2023-05-24T21:11:38.095 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.095 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:38.096 DEBUG:boto:Port: 8000 2023-05-24T21:11:38.096 DEBUG:boto:Params: {} 2023-05-24T21:11:38.097 DEBUG:boto:Token: None 2023-05-24T21:11:38.097 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj?versionId=hcen3y8E05rcJbSPsSSwDQfkEHk1NI7 2023-05-24T21:11:38.097 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:7f8PhdW2+VsLHuFb9+aAMnK8cWo= 2023-05-24T21:11:38.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:7f8PhdW2+VsLHuFb9+aAMnK8cWo=', 'Content-Length': '0'} 2023-05-24T21:11:38.099 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'hcen3y8E05rcJbSPsSSwDQfkEHk1NI7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a122a103cc6a07fb-00646e7d8a-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.100 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:38.100 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:38.101 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:38.101 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:38.102 DEBUG:boto:Method: GET 2023-05-24T21:11:38.102 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:38.102 DEBUG:boto:Data: 2023-05-24T21:11:38.103 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.103 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:38.103 DEBUG:boto:Port: 8001 2023-05-24T21:11:38.104 DEBUG:boto:Params: {} 2023-05-24T21:11:38.104 DEBUG:boto:Token: None 2023-05-24T21:11:38.105 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/ 2023-05-24T21:11:38.105 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5e2RJlSUQ8Y5t134TUlreZP6Nas= 2023-05-24T21:11:38.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5e2RJlSUQ8Y5t134TUlreZP6Nas=', 'Content-Length': '0'} 2023-05-24T21:11:38.108 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c5d26cf3dc9732a-00646e7d8a-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.108 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:38.109 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:38.109 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:38.110 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:38.110 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:38.110 DEBUG:boto:Method: GET 2023-05-24T21:11:38.111 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:38.111 DEBUG:boto:Data: 2023-05-24T21:11:38.111 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.112 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:38.112 DEBUG:boto:Port: 8000 2023-05-24T21:11:38.113 DEBUG:boto:Params: {} 2023-05-24T21:11:38.113 DEBUG:boto:Token: None 2023-05-24T21:11:38.113 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/ 2023-05-24T21:11:38.114 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5e2RJlSUQ8Y5t134TUlreZP6Nas= 2023-05-24T21:11:38.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5e2RJlSUQ8Y5t134TUlreZP6Nas=', 'Content-Length': '0'} 2023-05-24T21:11:38.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce1e59461c780425-00646e7d8a-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.117 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:38.118 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.118 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.119 DEBUG:boto:Method: HEAD 2023-05-24T21:11:38.119 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:11:38.119 DEBUG:boto:Data: 2023-05-24T21:11:38.120 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.120 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:38.120 DEBUG:boto:Port: 8001 2023-05-24T21:11:38.121 DEBUG:boto:Params: {} 2023-05-24T21:11:38.121 DEBUG:boto:Token: None 2023-05-24T21:11:38.121 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj 2023-05-24T21:11:38.122 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ed3FFLYS1b/XkKX9hkntV4II6A8= 2023-05-24T21:11:38.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ed3FFLYS1b/XkKX9hkntV4II6A8=', 'Content-Length': '0'} 2023-05-24T21:11:38.125 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cd1e2cd03baac702-00646e7d8a-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.126 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.126 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.126 DEBUG:boto:Method: HEAD 2023-05-24T21:11:38.127 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:11:38.127 DEBUG:boto:Data: 2023-05-24T21:11:38.127 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.128 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:38.128 DEBUG:boto:Port: 8000 2023-05-24T21:11:38.129 DEBUG:boto:Params: {} 2023-05-24T21:11:38.129 DEBUG:boto:Token: None 2023-05-24T21:11:38.129 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj 2023-05-24T21:11:38.130 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ed3FFLYS1b/XkKX9hkntV4II6A8= 2023-05-24T21:11:38.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ed3FFLYS1b/XkKX9hkntV4II6A8=', 'Content-Length': '0'} 2023-05-24T21:11:38.133 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000559e8cafe1d4194b-00646e7d8a-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:38.134 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.134 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.134 DEBUG:boto:path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.135 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.135 DEBUG:boto:Method: GET 2023-05-24T21:11:38.136 DEBUG:boto:Path: /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.136 DEBUG:boto:Data: 2023-05-24T21:11:38.136 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.137 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:38.137 DEBUG:boto:Port: 8001 2023-05-24T21:11:38.137 DEBUG:boto:Params: {} 2023-05-24T21:11:38.138 DEBUG:boto:Token: None 2023-05-24T21:11:38.138 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.139 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Qy3zwRpx5lVG+UJdWDSmNUuL1CE= 2023-05-24T21:11:38.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Qy3zwRpx5lVG+UJdWDSmNUuL1CE=', 'Content-Length': '0'} 2023-05-24T21:11:38.141 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e4c137727f3d5b60-00646e7d8a-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.188 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.189 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.189 DEBUG:boto:path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.189 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.190 DEBUG:boto:Method: GET 2023-05-24T21:11:38.190 DEBUG:boto:Path: /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.190 DEBUG:boto:Data: 2023-05-24T21:11:38.191 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.191 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:38.191 DEBUG:boto:Port: 8000 2023-05-24T21:11:38.192 DEBUG:boto:Params: {} 2023-05-24T21:11:38.192 DEBUG:boto:Token: None 2023-05-24T21:11:38.193 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.193 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Qy3zwRpx5lVG+UJdWDSmNUuL1CE= 2023-05-24T21:11:38.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Qy3zwRpx5lVG+UJdWDSmNUuL1CE=', 'Content-Length': '0'} 2023-05-24T21:11:38.195 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000570cc36631af4f04-00646e7d8a-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.196 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:38.196 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:38.197 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:38.197 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:38.197 DEBUG:boto:Method: GET 2023-05-24T21:11:38.198 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:38.198 DEBUG:boto:Data: 2023-05-24T21:11:38.198 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.199 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:38.199 DEBUG:boto:Port: 8001 2023-05-24T21:11:38.200 DEBUG:boto:Params: {} 2023-05-24T21:11:38.200 DEBUG:boto:Token: None 2023-05-24T21:11:38.200 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/ 2023-05-24T21:11:38.201 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5e2RJlSUQ8Y5t134TUlreZP6Nas= 2023-05-24T21:11:38.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5e2RJlSUQ8Y5t134TUlreZP6Nas=', 'Content-Length': '0'} 2023-05-24T21:11:38.203 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b206c044ea59237-00646e7d8a-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.204 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:38.205 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:38.205 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:38.205 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:38.206 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:38.206 DEBUG:boto:Method: GET 2023-05-24T21:11:38.207 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:38.207 DEBUG:boto:Data: 2023-05-24T21:11:38.207 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.208 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:38.208 DEBUG:boto:Port: 8000 2023-05-24T21:11:38.208 DEBUG:boto:Params: {} 2023-05-24T21:11:38.209 DEBUG:boto:Token: None 2023-05-24T21:11:38.209 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/ 2023-05-24T21:11:38.210 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5e2RJlSUQ8Y5t134TUlreZP6Nas= 2023-05-24T21:11:38.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5e2RJlSUQ8Y5t134TUlreZP6Nas=', 'Content-Length': '0'} 2023-05-24T21:11:38.212 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038306fc10730cdb8-00646e7d8a-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.213 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:38.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:38.214 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.214 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.215 DEBUG:boto:path=/qskrps-13/obj?versionId=LsSugEEr8NPamOV4ZLrgX.pixmT7G5y 2023-05-24T21:11:38.215 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=LsSugEEr8NPamOV4ZLrgX.pixmT7G5y 2023-05-24T21:11:38.215 DEBUG:boto:Method: GET 2023-05-24T21:11:38.216 DEBUG:boto:Path: /qskrps-13/obj?versionId=LsSugEEr8NPamOV4ZLrgX.pixmT7G5y 2023-05-24T21:11:38.216 DEBUG:boto:Data: 2023-05-24T21:11:38.216 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.217 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:38.217 DEBUG:boto:Port: 8001 2023-05-24T21:11:38.217 DEBUG:boto:Params: {} 2023-05-24T21:11:38.218 DEBUG:boto:Token: None 2023-05-24T21:11:38.218 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj?versionId=LsSugEEr8NPamOV4ZLrgX.pixmT7G5y 2023-05-24T21:11:38.219 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:9CfNZP9saRZj7d3ILOzv2gs4cfo= 2023-05-24T21:11:38.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:9CfNZP9saRZj7d3ILOzv2gs4cfo=', 'Content-Length': '0'} 2023-05-24T21:11:38.221 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'LsSugEEr8NPamOV4ZLrgX.pixmT7G5y'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001ba5b7ea4b04874b-00646e7d8a-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.222 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.222 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.222 DEBUG:boto:path=/qskrps-13/obj?versionId=LsSugEEr8NPamOV4ZLrgX.pixmT7G5y 2023-05-24T21:11:38.223 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=LsSugEEr8NPamOV4ZLrgX.pixmT7G5y 2023-05-24T21:11:38.223 DEBUG:boto:Method: GET 2023-05-24T21:11:38.224 DEBUG:boto:Path: /qskrps-13/obj?versionId=LsSugEEr8NPamOV4ZLrgX.pixmT7G5y 2023-05-24T21:11:38.224 DEBUG:boto:Data: 2023-05-24T21:11:38.224 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.225 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:38.225 DEBUG:boto:Port: 8000 2023-05-24T21:11:38.225 DEBUG:boto:Params: {} 2023-05-24T21:11:38.226 DEBUG:boto:Token: None 2023-05-24T21:11:38.226 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj?versionId=LsSugEEr8NPamOV4ZLrgX.pixmT7G5y 2023-05-24T21:11:38.227 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:9CfNZP9saRZj7d3ILOzv2gs4cfo= 2023-05-24T21:11:38.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:9CfNZP9saRZj7d3ILOzv2gs4cfo=', 'Content-Length': '0'} 2023-05-24T21:11:38.229 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'LsSugEEr8NPamOV4ZLrgX.pixmT7G5y'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006b51375d14d04e34-00646e7d8a-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.230 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.230 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.230 DEBUG:boto:path=/qskrps-13/obj?versionId=LsSugEEr8NPamOV4ZLrgX.pixmT7G5y 2023-05-24T21:11:38.231 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=LsSugEEr8NPamOV4ZLrgX.pixmT7G5y 2023-05-24T21:11:38.231 DEBUG:boto:Method: HEAD 2023-05-24T21:11:38.231 DEBUG:boto:Path: /qskrps-13/obj?versionId=LsSugEEr8NPamOV4ZLrgX.pixmT7G5y 2023-05-24T21:11:38.232 DEBUG:boto:Data: 2023-05-24T21:11:38.232 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.232 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:38.233 DEBUG:boto:Port: 8001 2023-05-24T21:11:38.233 DEBUG:boto:Params: {} 2023-05-24T21:11:38.233 DEBUG:boto:Token: None 2023-05-24T21:11:38.234 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj?versionId=LsSugEEr8NPamOV4ZLrgX.pixmT7G5y 2023-05-24T21:11:38.234 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:B7yGip6KBembet8Pi+/Ao+ew5yg= 2023-05-24T21:11:38.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:B7yGip6KBembet8Pi+/Ao+ew5yg=', 'Content-Length': '0'} 2023-05-24T21:11:38.236 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'LsSugEEr8NPamOV4ZLrgX.pixmT7G5y'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000798f0b88d6423588-00646e7d8a-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.237 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.237 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.238 DEBUG:boto:path=/qskrps-13/obj?versionId=LsSugEEr8NPamOV4ZLrgX.pixmT7G5y 2023-05-24T21:11:38.238 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=LsSugEEr8NPamOV4ZLrgX.pixmT7G5y 2023-05-24T21:11:38.238 DEBUG:boto:Method: HEAD 2023-05-24T21:11:38.239 DEBUG:boto:Path: /qskrps-13/obj?versionId=LsSugEEr8NPamOV4ZLrgX.pixmT7G5y 2023-05-24T21:11:38.239 DEBUG:boto:Data: 2023-05-24T21:11:38.239 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.240 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:38.240 DEBUG:boto:Port: 8000 2023-05-24T21:11:38.240 DEBUG:boto:Params: {} 2023-05-24T21:11:38.241 DEBUG:boto:Token: None 2023-05-24T21:11:38.241 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj?versionId=LsSugEEr8NPamOV4ZLrgX.pixmT7G5y 2023-05-24T21:11:38.242 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:B7yGip6KBembet8Pi+/Ao+ew5yg= 2023-05-24T21:11:38.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:B7yGip6KBembet8Pi+/Ao+ew5yg=', 'Content-Length': '0'} 2023-05-24T21:11:38.244 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'LsSugEEr8NPamOV4ZLrgX.pixmT7G5y'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000000b606b35a0bcb1e-00646e7d8a-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:38.245 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.245 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.246 DEBUG:boto:path=/qskrps-13/obj?versionId=LsSugEEr8NPamOV4ZLrgX.pixmT7G5y 2023-05-24T21:11:38.246 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=LsSugEEr8NPamOV4ZLrgX.pixmT7G5y 2023-05-24T21:11:38.246 DEBUG:boto:Method: GET 2023-05-24T21:11:38.247 DEBUG:boto:Path: /qskrps-13/obj?versionId=LsSugEEr8NPamOV4ZLrgX.pixmT7G5y 2023-05-24T21:11:38.247 DEBUG:boto:Data: 2023-05-24T21:11:38.247 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.248 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:38.248 DEBUG:boto:Port: 8001 2023-05-24T21:11:38.249 DEBUG:boto:Params: {} 2023-05-24T21:11:38.249 DEBUG:boto:Token: None 2023-05-24T21:11:38.249 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj?versionId=LsSugEEr8NPamOV4ZLrgX.pixmT7G5y 2023-05-24T21:11:38.250 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:9CfNZP9saRZj7d3ILOzv2gs4cfo= 2023-05-24T21:11:38.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:9CfNZP9saRZj7d3ILOzv2gs4cfo=', 'Content-Length': '0'} 2023-05-24T21:11:38.252 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'LsSugEEr8NPamOV4ZLrgX.pixmT7G5y'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000089891abe6a538f2f-00646e7d8a-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.292 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.293 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.293 DEBUG:boto:path=/qskrps-13/obj?versionId=LsSugEEr8NPamOV4ZLrgX.pixmT7G5y 2023-05-24T21:11:38.293 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=LsSugEEr8NPamOV4ZLrgX.pixmT7G5y 2023-05-24T21:11:38.294 DEBUG:boto:Method: GET 2023-05-24T21:11:38.294 DEBUG:boto:Path: /qskrps-13/obj?versionId=LsSugEEr8NPamOV4ZLrgX.pixmT7G5y 2023-05-24T21:11:38.294 DEBUG:boto:Data: 2023-05-24T21:11:38.295 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.295 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:38.296 DEBUG:boto:Port: 8000 2023-05-24T21:11:38.296 DEBUG:boto:Params: {} 2023-05-24T21:11:38.296 DEBUG:boto:Token: None 2023-05-24T21:11:38.297 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj?versionId=LsSugEEr8NPamOV4ZLrgX.pixmT7G5y 2023-05-24T21:11:38.297 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:9CfNZP9saRZj7d3ILOzv2gs4cfo= 2023-05-24T21:11:38.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:9CfNZP9saRZj7d3ILOzv2gs4cfo=', 'Content-Length': '0'} 2023-05-24T21:11:38.299 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'LsSugEEr8NPamOV4ZLrgX.pixmT7G5y'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000724b99b3398305de-00646e7d8a-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.300 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:38.300 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:38.301 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:38.301 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:38.301 DEBUG:boto:Method: GET 2023-05-24T21:11:38.302 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:38.302 DEBUG:boto:Data: 2023-05-24T21:11:38.302 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.303 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:38.303 DEBUG:boto:Port: 8001 2023-05-24T21:11:38.304 DEBUG:boto:Params: {} 2023-05-24T21:11:38.304 DEBUG:boto:Token: None 2023-05-24T21:11:38.304 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/ 2023-05-24T21:11:38.305 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5e2RJlSUQ8Y5t134TUlreZP6Nas= 2023-05-24T21:11:38.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5e2RJlSUQ8Y5t134TUlreZP6Nas=', 'Content-Length': '0'} 2023-05-24T21:11:38.307 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007dbb891a4f5ffa5f-00646e7d8a-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.308 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:38.309 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:38.309 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:38.309 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:38.310 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:38.310 DEBUG:boto:Method: GET 2023-05-24T21:11:38.310 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:38.311 DEBUG:boto:Data: 2023-05-24T21:11:38.311 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.312 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:38.312 DEBUG:boto:Port: 8000 2023-05-24T21:11:38.312 DEBUG:boto:Params: {} 2023-05-24T21:11:38.313 DEBUG:boto:Token: None 2023-05-24T21:11:38.313 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/ 2023-05-24T21:11:38.313 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5e2RJlSUQ8Y5t134TUlreZP6Nas= 2023-05-24T21:11:38.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5e2RJlSUQ8Y5t134TUlreZP6Nas=', 'Content-Length': '0'} 2023-05-24T21:11:38.316 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b9ee66e2bdb408d-00646e7d8a-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.317 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:38.317 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.318 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.318 DEBUG:boto:Method: HEAD 2023-05-24T21:11:38.319 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:11:38.319 DEBUG:boto:Data: 2023-05-24T21:11:38.319 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.320 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:38.320 DEBUG:boto:Port: 8001 2023-05-24T21:11:38.320 DEBUG:boto:Params: {} 2023-05-24T21:11:38.321 DEBUG:boto:Token: None 2023-05-24T21:11:38.321 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj 2023-05-24T21:11:38.321 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ed3FFLYS1b/XkKX9hkntV4II6A8= 2023-05-24T21:11:38.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ed3FFLYS1b/XkKX9hkntV4II6A8=', 'Content-Length': '0'} 2023-05-24T21:11:38.325 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000020867ec639c09be4-00646e7d8a-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.325 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.326 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.326 DEBUG:boto:Method: HEAD 2023-05-24T21:11:38.326 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:11:38.327 DEBUG:boto:Data: 2023-05-24T21:11:38.327 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.327 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:38.328 DEBUG:boto:Port: 8000 2023-05-24T21:11:38.328 DEBUG:boto:Params: {} 2023-05-24T21:11:38.329 DEBUG:boto:Token: None 2023-05-24T21:11:38.329 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj 2023-05-24T21:11:38.329 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ed3FFLYS1b/XkKX9hkntV4II6A8= 2023-05-24T21:11:38.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ed3FFLYS1b/XkKX9hkntV4II6A8=', 'Content-Length': '0'} 2023-05-24T21:11:38.332 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e719fbf11972e5d9-00646e7d8a-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:38.333 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.334 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.334 DEBUG:boto:path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.334 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.335 DEBUG:boto:Method: GET 2023-05-24T21:11:38.335 DEBUG:boto:Path: /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.335 DEBUG:boto:Data: 2023-05-24T21:11:38.336 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.336 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:38.337 DEBUG:boto:Port: 8001 2023-05-24T21:11:38.337 DEBUG:boto:Params: {} 2023-05-24T21:11:38.337 DEBUG:boto:Token: None 2023-05-24T21:11:38.338 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.338 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Qy3zwRpx5lVG+UJdWDSmNUuL1CE= 2023-05-24T21:11:38.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Qy3zwRpx5lVG+UJdWDSmNUuL1CE=', 'Content-Length': '0'} 2023-05-24T21:11:38.340 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ea268671e5f2f544-00646e7d8a-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.341 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.341 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.342 DEBUG:boto:path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.342 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.342 DEBUG:boto:Method: GET 2023-05-24T21:11:38.343 DEBUG:boto:Path: /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.343 DEBUG:boto:Data: 2023-05-24T21:11:38.344 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.344 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:38.345 DEBUG:boto:Port: 8000 2023-05-24T21:11:38.345 DEBUG:boto:Params: {} 2023-05-24T21:11:38.345 DEBUG:boto:Token: None 2023-05-24T21:11:38.346 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.346 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Qy3zwRpx5lVG+UJdWDSmNUuL1CE= 2023-05-24T21:11:38.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Qy3zwRpx5lVG+UJdWDSmNUuL1CE=', 'Content-Length': '0'} 2023-05-24T21:11:38.348 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000050b1012866c36892-00646e7d8a-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.392 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:38.393 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:38.393 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:38.394 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:38.394 DEBUG:boto:Method: GET 2023-05-24T21:11:38.394 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:38.395 DEBUG:boto:Data: 2023-05-24T21:11:38.395 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.395 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:38.396 DEBUG:boto:Port: 8001 2023-05-24T21:11:38.396 DEBUG:boto:Params: {} 2023-05-24T21:11:38.397 DEBUG:boto:Token: None 2023-05-24T21:11:38.397 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/ 2023-05-24T21:11:38.397 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5e2RJlSUQ8Y5t134TUlreZP6Nas= 2023-05-24T21:11:38.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5e2RJlSUQ8Y5t134TUlreZP6Nas=', 'Content-Length': '0'} 2023-05-24T21:11:38.401 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003985b9608e84b5b1-00646e7d8a-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.401 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:38.402 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:38.402 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:38.403 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:38.403 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:38.403 DEBUG:boto:Method: GET 2023-05-24T21:11:38.404 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:38.404 DEBUG:boto:Data: 2023-05-24T21:11:38.405 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.405 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:38.405 DEBUG:boto:Port: 8000 2023-05-24T21:11:38.406 DEBUG:boto:Params: {} 2023-05-24T21:11:38.406 DEBUG:boto:Token: None 2023-05-24T21:11:38.406 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/ 2023-05-24T21:11:38.407 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5e2RJlSUQ8Y5t134TUlreZP6Nas= 2023-05-24T21:11:38.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5e2RJlSUQ8Y5t134TUlreZP6Nas=', 'Content-Length': '0'} 2023-05-24T21:11:38.410 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d87d12c194c3defb-00646e7d8a-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.410 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:38.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:38.411 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.412 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.412 DEBUG:boto:path=/qskrps-13/obj?versionId=c.ug7Ld36X8.TaQXWIjev99m-BKd5qJ 2023-05-24T21:11:38.413 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=c.ug7Ld36X8.TaQXWIjev99m-BKd5qJ 2023-05-24T21:11:38.413 DEBUG:boto:Method: GET 2023-05-24T21:11:38.413 DEBUG:boto:Path: /qskrps-13/obj?versionId=c.ug7Ld36X8.TaQXWIjev99m-BKd5qJ 2023-05-24T21:11:38.414 DEBUG:boto:Data: 2023-05-24T21:11:38.414 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.414 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:38.415 DEBUG:boto:Port: 8001 2023-05-24T21:11:38.415 DEBUG:boto:Params: {} 2023-05-24T21:11:38.416 DEBUG:boto:Token: None 2023-05-24T21:11:38.416 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj?versionId=c.ug7Ld36X8.TaQXWIjev99m-BKd5qJ 2023-05-24T21:11:38.416 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:6efMqK7jw7JSNyFWZIO5CrzAGj0= 2023-05-24T21:11:38.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:6efMqK7jw7JSNyFWZIO5CrzAGj0=', 'Content-Length': '0'} 2023-05-24T21:11:38.419 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'c.ug7Ld36X8.TaQXWIjev99m-BKd5qJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000422d31f541a21a4b-00646e7d8a-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.460 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.461 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.461 DEBUG:boto:path=/qskrps-13/obj?versionId=c.ug7Ld36X8.TaQXWIjev99m-BKd5qJ 2023-05-24T21:11:38.462 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=c.ug7Ld36X8.TaQXWIjev99m-BKd5qJ 2023-05-24T21:11:38.462 DEBUG:boto:Method: GET 2023-05-24T21:11:38.463 DEBUG:boto:Path: /qskrps-13/obj?versionId=c.ug7Ld36X8.TaQXWIjev99m-BKd5qJ 2023-05-24T21:11:38.463 DEBUG:boto:Data: 2023-05-24T21:11:38.463 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.464 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:38.464 DEBUG:boto:Port: 8000 2023-05-24T21:11:38.464 DEBUG:boto:Params: {} 2023-05-24T21:11:38.465 DEBUG:boto:Token: None 2023-05-24T21:11:38.465 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj?versionId=c.ug7Ld36X8.TaQXWIjev99m-BKd5qJ 2023-05-24T21:11:38.466 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:6efMqK7jw7JSNyFWZIO5CrzAGj0= 2023-05-24T21:11:38.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:6efMqK7jw7JSNyFWZIO5CrzAGj0=', 'Content-Length': '0'} 2023-05-24T21:11:38.468 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'c.ug7Ld36X8.TaQXWIjev99m-BKd5qJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1aef29362b65c95-00646e7d8a-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.469 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.469 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.469 DEBUG:boto:path=/qskrps-13/obj?versionId=c.ug7Ld36X8.TaQXWIjev99m-BKd5qJ 2023-05-24T21:11:38.470 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=c.ug7Ld36X8.TaQXWIjev99m-BKd5qJ 2023-05-24T21:11:38.470 DEBUG:boto:Method: HEAD 2023-05-24T21:11:38.470 DEBUG:boto:Path: /qskrps-13/obj?versionId=c.ug7Ld36X8.TaQXWIjev99m-BKd5qJ 2023-05-24T21:11:38.471 DEBUG:boto:Data: 2023-05-24T21:11:38.471 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.471 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:38.472 DEBUG:boto:Port: 8001 2023-05-24T21:11:38.472 DEBUG:boto:Params: {} 2023-05-24T21:11:38.472 DEBUG:boto:Token: None 2023-05-24T21:11:38.473 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj?versionId=c.ug7Ld36X8.TaQXWIjev99m-BKd5qJ 2023-05-24T21:11:38.473 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:YH5CUtJiJlM/PWRGp7J9NxzkQUE= 2023-05-24T21:11:38.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:YH5CUtJiJlM/PWRGp7J9NxzkQUE=', 'Content-Length': '0'} 2023-05-24T21:11:38.476 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'c.ug7Ld36X8.TaQXWIjev99m-BKd5qJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000070e5e27bc55aa7e4-00646e7d8a-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.476 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.477 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.477 DEBUG:boto:path=/qskrps-13/obj?versionId=c.ug7Ld36X8.TaQXWIjev99m-BKd5qJ 2023-05-24T21:11:38.477 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=c.ug7Ld36X8.TaQXWIjev99m-BKd5qJ 2023-05-24T21:11:38.478 DEBUG:boto:Method: HEAD 2023-05-24T21:11:38.478 DEBUG:boto:Path: /qskrps-13/obj?versionId=c.ug7Ld36X8.TaQXWIjev99m-BKd5qJ 2023-05-24T21:11:38.478 DEBUG:boto:Data: 2023-05-24T21:11:38.479 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.479 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:38.479 DEBUG:boto:Port: 8000 2023-05-24T21:11:38.480 DEBUG:boto:Params: {} 2023-05-24T21:11:38.480 DEBUG:boto:Token: None 2023-05-24T21:11:38.480 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj?versionId=c.ug7Ld36X8.TaQXWIjev99m-BKd5qJ 2023-05-24T21:11:38.481 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:YH5CUtJiJlM/PWRGp7J9NxzkQUE= 2023-05-24T21:11:38.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:YH5CUtJiJlM/PWRGp7J9NxzkQUE=', 'Content-Length': '0'} 2023-05-24T21:11:38.483 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'c.ug7Ld36X8.TaQXWIjev99m-BKd5qJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e3649ebeb85519f2-00646e7d8a-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:38.484 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.484 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.485 DEBUG:boto:path=/qskrps-13/obj?versionId=c.ug7Ld36X8.TaQXWIjev99m-BKd5qJ 2023-05-24T21:11:38.485 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=c.ug7Ld36X8.TaQXWIjev99m-BKd5qJ 2023-05-24T21:11:38.486 DEBUG:boto:Method: GET 2023-05-24T21:11:38.486 DEBUG:boto:Path: /qskrps-13/obj?versionId=c.ug7Ld36X8.TaQXWIjev99m-BKd5qJ 2023-05-24T21:11:38.486 DEBUG:boto:Data: 2023-05-24T21:11:38.487 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.487 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:38.487 DEBUG:boto:Port: 8001 2023-05-24T21:11:38.488 DEBUG:boto:Params: {} 2023-05-24T21:11:38.488 DEBUG:boto:Token: None 2023-05-24T21:11:38.488 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj?versionId=c.ug7Ld36X8.TaQXWIjev99m-BKd5qJ 2023-05-24T21:11:38.489 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:6efMqK7jw7JSNyFWZIO5CrzAGj0= 2023-05-24T21:11:38.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:6efMqK7jw7JSNyFWZIO5CrzAGj0=', 'Content-Length': '0'} 2023-05-24T21:11:38.491 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'c.ug7Ld36X8.TaQXWIjev99m-BKd5qJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000984cbed5cbd9139f-00646e7d8a-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.492 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.492 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.493 DEBUG:boto:path=/qskrps-13/obj?versionId=c.ug7Ld36X8.TaQXWIjev99m-BKd5qJ 2023-05-24T21:11:38.493 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=c.ug7Ld36X8.TaQXWIjev99m-BKd5qJ 2023-05-24T21:11:38.493 DEBUG:boto:Method: GET 2023-05-24T21:11:38.494 DEBUG:boto:Path: /qskrps-13/obj?versionId=c.ug7Ld36X8.TaQXWIjev99m-BKd5qJ 2023-05-24T21:11:38.494 DEBUG:boto:Data: 2023-05-24T21:11:38.494 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.495 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:38.495 DEBUG:boto:Port: 8000 2023-05-24T21:11:38.495 DEBUG:boto:Params: {} 2023-05-24T21:11:38.496 DEBUG:boto:Token: None 2023-05-24T21:11:38.505 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj?versionId=c.ug7Ld36X8.TaQXWIjev99m-BKd5qJ 2023-05-24T21:11:38.506 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:6efMqK7jw7JSNyFWZIO5CrzAGj0= 2023-05-24T21:11:38.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:6efMqK7jw7JSNyFWZIO5CrzAGj0=', 'Content-Length': '0'} 2023-05-24T21:11:38.508 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'c.ug7Ld36X8.TaQXWIjev99m-BKd5qJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005644e804b2a6ff88-00646e7d8a-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.509 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:38.509 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:38.510 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:38.510 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:38.510 DEBUG:boto:Method: GET 2023-05-24T21:11:38.511 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:38.511 DEBUG:boto:Data: 2023-05-24T21:11:38.512 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.512 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:38.512 DEBUG:boto:Port: 8001 2023-05-24T21:11:38.513 DEBUG:boto:Params: {} 2023-05-24T21:11:38.513 DEBUG:boto:Token: None 2023-05-24T21:11:38.513 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/ 2023-05-24T21:11:38.514 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5e2RJlSUQ8Y5t134TUlreZP6Nas= 2023-05-24T21:11:38.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5e2RJlSUQ8Y5t134TUlreZP6Nas=', 'Content-Length': '0'} 2023-05-24T21:11:38.517 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7d6bd0806d18caa-00646e7d8a-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.517 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:38.518 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:38.518 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:38.519 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:38.519 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:38.519 DEBUG:boto:Method: GET 2023-05-24T21:11:38.520 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:38.520 DEBUG:boto:Data: 2023-05-24T21:11:38.521 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.521 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:38.521 DEBUG:boto:Port: 8000 2023-05-24T21:11:38.522 DEBUG:boto:Params: {} 2023-05-24T21:11:38.522 DEBUG:boto:Token: None 2023-05-24T21:11:38.522 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/ 2023-05-24T21:11:38.523 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5e2RJlSUQ8Y5t134TUlreZP6Nas= 2023-05-24T21:11:38.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5e2RJlSUQ8Y5t134TUlreZP6Nas=', 'Content-Length': '0'} 2023-05-24T21:11:38.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec1b48a1361125db-00646e7d8a-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.526 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:38.527 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.527 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.528 DEBUG:boto:Method: HEAD 2023-05-24T21:11:38.528 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:11:38.528 DEBUG:boto:Data: 2023-05-24T21:11:38.529 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.529 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:38.529 DEBUG:boto:Port: 8001 2023-05-24T21:11:38.530 DEBUG:boto:Params: {} 2023-05-24T21:11:38.530 DEBUG:boto:Token: None 2023-05-24T21:11:38.530 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj 2023-05-24T21:11:38.531 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ed3FFLYS1b/XkKX9hkntV4II6A8= 2023-05-24T21:11:38.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ed3FFLYS1b/XkKX9hkntV4II6A8=', 'Content-Length': '0'} 2023-05-24T21:11:38.534 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002f665c4d94d30ad1-00646e7d8a-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.535 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.535 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.536 DEBUG:boto:Method: HEAD 2023-05-24T21:11:38.536 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:11:38.537 DEBUG:boto:Data: 2023-05-24T21:11:38.537 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.537 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:38.537 DEBUG:boto:Port: 8000 2023-05-24T21:11:38.538 DEBUG:boto:Params: {} 2023-05-24T21:11:38.538 DEBUG:boto:Token: None 2023-05-24T21:11:38.539 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj 2023-05-24T21:11:38.539 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ed3FFLYS1b/XkKX9hkntV4II6A8= 2023-05-24T21:11:38.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ed3FFLYS1b/XkKX9hkntV4II6A8=', 'Content-Length': '0'} 2023-05-24T21:11:38.542 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001d79e1eb63d73075-00646e7d8a-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:38.543 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.544 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.544 DEBUG:boto:path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.544 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.545 DEBUG:boto:Method: GET 2023-05-24T21:11:38.545 DEBUG:boto:Path: /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.546 DEBUG:boto:Data: 2023-05-24T21:11:38.546 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.546 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:38.547 DEBUG:boto:Port: 8001 2023-05-24T21:11:38.547 DEBUG:boto:Params: {} 2023-05-24T21:11:38.547 DEBUG:boto:Token: None 2023-05-24T21:11:38.548 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.548 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Qy3zwRpx5lVG+UJdWDSmNUuL1CE= 2023-05-24T21:11:38.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Qy3zwRpx5lVG+UJdWDSmNUuL1CE=', 'Content-Length': '0'} 2023-05-24T21:11:38.551 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa19c167596c9a16-00646e7d8a-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.592 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.593 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.593 DEBUG:boto:path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.594 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.594 DEBUG:boto:Method: GET 2023-05-24T21:11:38.595 DEBUG:boto:Path: /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.595 DEBUG:boto:Data: 2023-05-24T21:11:38.595 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.596 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:38.596 DEBUG:boto:Port: 8000 2023-05-24T21:11:38.596 DEBUG:boto:Params: {} 2023-05-24T21:11:38.597 DEBUG:boto:Token: None 2023-05-24T21:11:38.597 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.598 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Qy3zwRpx5lVG+UJdWDSmNUuL1CE= 2023-05-24T21:11:38.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Qy3zwRpx5lVG+UJdWDSmNUuL1CE=', 'Content-Length': '0'} 2023-05-24T21:11:38.600 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005b089d4c38fd0196-00646e7d8a-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.601 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:38.601 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:38.602 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:38.602 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:38.602 DEBUG:boto:Method: GET 2023-05-24T21:11:38.603 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:38.603 DEBUG:boto:Data: 2023-05-24T21:11:38.603 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.604 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:38.604 DEBUG:boto:Port: 8001 2023-05-24T21:11:38.604 DEBUG:boto:Params: {} 2023-05-24T21:11:38.605 DEBUG:boto:Token: None 2023-05-24T21:11:38.605 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/ 2023-05-24T21:11:38.606 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5e2RJlSUQ8Y5t134TUlreZP6Nas= 2023-05-24T21:11:38.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5e2RJlSUQ8Y5t134TUlreZP6Nas=', 'Content-Length': '0'} 2023-05-24T21:11:38.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007df35bbf704d4d4a-00646e7d8a-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.609 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:38.610 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:38.610 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:38.611 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:38.611 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:38.611 DEBUG:boto:Method: GET 2023-05-24T21:11:38.612 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:38.612 DEBUG:boto:Data: 2023-05-24T21:11:38.612 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.613 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:38.613 DEBUG:boto:Port: 8000 2023-05-24T21:11:38.613 DEBUG:boto:Params: {} 2023-05-24T21:11:38.614 DEBUG:boto:Token: None 2023-05-24T21:11:38.614 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/ 2023-05-24T21:11:38.615 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5e2RJlSUQ8Y5t134TUlreZP6Nas= 2023-05-24T21:11:38.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5e2RJlSUQ8Y5t134TUlreZP6Nas=', 'Content-Length': '0'} 2023-05-24T21:11:38.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dad702094834e435-00646e7d8a-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.618 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:38.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:38.619 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.619 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.620 DEBUG:boto:path=/qskrps-13/obj?versionId=DkBjLdO6nXireb7JdIhAsx00H76MZGD 2023-05-24T21:11:38.620 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=DkBjLdO6nXireb7JdIhAsx00H76MZGD 2023-05-24T21:11:38.620 DEBUG:boto:Method: GET 2023-05-24T21:11:38.621 DEBUG:boto:Path: /qskrps-13/obj?versionId=DkBjLdO6nXireb7JdIhAsx00H76MZGD 2023-05-24T21:11:38.621 DEBUG:boto:Data: 2023-05-24T21:11:38.621 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.622 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:38.622 DEBUG:boto:Port: 8001 2023-05-24T21:11:38.623 DEBUG:boto:Params: {} 2023-05-24T21:11:38.623 DEBUG:boto:Token: None 2023-05-24T21:11:38.623 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj?versionId=DkBjLdO6nXireb7JdIhAsx00H76MZGD 2023-05-24T21:11:38.624 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:7XqrYWsmUoXrTz/5teoipy4VF6Q= 2023-05-24T21:11:38.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:7XqrYWsmUoXrTz/5teoipy4VF6Q=', 'Content-Length': '0'} 2023-05-24T21:11:38.626 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'DkBjLdO6nXireb7JdIhAsx00H76MZGD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bf85c64fd12914c4-00646e7d8a-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.627 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.627 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.628 DEBUG:boto:path=/qskrps-13/obj?versionId=DkBjLdO6nXireb7JdIhAsx00H76MZGD 2023-05-24T21:11:38.628 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=DkBjLdO6nXireb7JdIhAsx00H76MZGD 2023-05-24T21:11:38.628 DEBUG:boto:Method: GET 2023-05-24T21:11:38.629 DEBUG:boto:Path: /qskrps-13/obj?versionId=DkBjLdO6nXireb7JdIhAsx00H76MZGD 2023-05-24T21:11:38.629 DEBUG:boto:Data: 2023-05-24T21:11:38.629 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.630 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:38.630 DEBUG:boto:Port: 8000 2023-05-24T21:11:38.631 DEBUG:boto:Params: {} 2023-05-24T21:11:38.631 DEBUG:boto:Token: None 2023-05-24T21:11:38.631 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj?versionId=DkBjLdO6nXireb7JdIhAsx00H76MZGD 2023-05-24T21:11:38.632 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:7XqrYWsmUoXrTz/5teoipy4VF6Q= 2023-05-24T21:11:38.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:7XqrYWsmUoXrTz/5teoipy4VF6Q=', 'Content-Length': '0'} 2023-05-24T21:11:38.634 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'DkBjLdO6nXireb7JdIhAsx00H76MZGD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000066eb27b9154ed849-00646e7d8a-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.635 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.635 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.635 DEBUG:boto:path=/qskrps-13/obj?versionId=DkBjLdO6nXireb7JdIhAsx00H76MZGD 2023-05-24T21:11:38.636 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=DkBjLdO6nXireb7JdIhAsx00H76MZGD 2023-05-24T21:11:38.636 DEBUG:boto:Method: HEAD 2023-05-24T21:11:38.636 DEBUG:boto:Path: /qskrps-13/obj?versionId=DkBjLdO6nXireb7JdIhAsx00H76MZGD 2023-05-24T21:11:38.637 DEBUG:boto:Data: 2023-05-24T21:11:38.637 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.637 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:38.638 DEBUG:boto:Port: 8001 2023-05-24T21:11:38.638 DEBUG:boto:Params: {} 2023-05-24T21:11:38.638 DEBUG:boto:Token: None 2023-05-24T21:11:38.639 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj?versionId=DkBjLdO6nXireb7JdIhAsx00H76MZGD 2023-05-24T21:11:38.639 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:KReHeseLEGJTLC5fb1TZvrvSwf8= 2023-05-24T21:11:38.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:KReHeseLEGJTLC5fb1TZvrvSwf8=', 'Content-Length': '0'} 2023-05-24T21:11:38.641 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'DkBjLdO6nXireb7JdIhAsx00H76MZGD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000092a9ed8759aff22-00646e7d8a-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.642 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.642 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.643 DEBUG:boto:path=/qskrps-13/obj?versionId=DkBjLdO6nXireb7JdIhAsx00H76MZGD 2023-05-24T21:11:38.643 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=DkBjLdO6nXireb7JdIhAsx00H76MZGD 2023-05-24T21:11:38.643 DEBUG:boto:Method: HEAD 2023-05-24T21:11:38.644 DEBUG:boto:Path: /qskrps-13/obj?versionId=DkBjLdO6nXireb7JdIhAsx00H76MZGD 2023-05-24T21:11:38.644 DEBUG:boto:Data: 2023-05-24T21:11:38.644 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.645 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:38.645 DEBUG:boto:Port: 8000 2023-05-24T21:11:38.645 DEBUG:boto:Params: {} 2023-05-24T21:11:38.646 DEBUG:boto:Token: None 2023-05-24T21:11:38.646 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj?versionId=DkBjLdO6nXireb7JdIhAsx00H76MZGD 2023-05-24T21:11:38.646 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:KReHeseLEGJTLC5fb1TZvrvSwf8= 2023-05-24T21:11:38.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:KReHeseLEGJTLC5fb1TZvrvSwf8=', 'Content-Length': '0'} 2023-05-24T21:11:38.648 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'DkBjLdO6nXireb7JdIhAsx00H76MZGD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ba8e5769c3e34f30-00646e7d8a-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:38.649 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.650 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.650 DEBUG:boto:path=/qskrps-13/obj?versionId=DkBjLdO6nXireb7JdIhAsx00H76MZGD 2023-05-24T21:11:38.650 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=DkBjLdO6nXireb7JdIhAsx00H76MZGD 2023-05-24T21:11:38.651 DEBUG:boto:Method: GET 2023-05-24T21:11:38.651 DEBUG:boto:Path: /qskrps-13/obj?versionId=DkBjLdO6nXireb7JdIhAsx00H76MZGD 2023-05-24T21:11:38.651 DEBUG:boto:Data: 2023-05-24T21:11:38.652 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.652 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:38.653 DEBUG:boto:Port: 8001 2023-05-24T21:11:38.653 DEBUG:boto:Params: {} 2023-05-24T21:11:38.653 DEBUG:boto:Token: None 2023-05-24T21:11:38.654 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj?versionId=DkBjLdO6nXireb7JdIhAsx00H76MZGD 2023-05-24T21:11:38.654 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:7XqrYWsmUoXrTz/5teoipy4VF6Q= 2023-05-24T21:11:38.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:7XqrYWsmUoXrTz/5teoipy4VF6Q=', 'Content-Length': '0'} 2023-05-24T21:11:38.657 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'DkBjLdO6nXireb7JdIhAsx00H76MZGD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e68d06304ef2187e-00646e7d8a-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.658 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.658 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.659 DEBUG:boto:path=/qskrps-13/obj?versionId=DkBjLdO6nXireb7JdIhAsx00H76MZGD 2023-05-24T21:11:38.659 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=DkBjLdO6nXireb7JdIhAsx00H76MZGD 2023-05-24T21:11:38.659 DEBUG:boto:Method: GET 2023-05-24T21:11:38.660 DEBUG:boto:Path: /qskrps-13/obj?versionId=DkBjLdO6nXireb7JdIhAsx00H76MZGD 2023-05-24T21:11:38.660 DEBUG:boto:Data: 2023-05-24T21:11:38.661 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.661 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:38.662 DEBUG:boto:Port: 8000 2023-05-24T21:11:38.662 DEBUG:boto:Params: {} 2023-05-24T21:11:38.662 DEBUG:boto:Token: None 2023-05-24T21:11:38.663 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj?versionId=DkBjLdO6nXireb7JdIhAsx00H76MZGD 2023-05-24T21:11:38.663 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:7XqrYWsmUoXrTz/5teoipy4VF6Q= 2023-05-24T21:11:38.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:7XqrYWsmUoXrTz/5teoipy4VF6Q=', 'Content-Length': '0'} 2023-05-24T21:11:38.666 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'DkBjLdO6nXireb7JdIhAsx00H76MZGD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f8b69b8d566630a7-00646e7d8a-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.666 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:38.667 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:38.667 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:38.667 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:38.668 DEBUG:boto:Method: GET 2023-05-24T21:11:38.668 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:38.669 DEBUG:boto:Data: 2023-05-24T21:11:38.669 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.669 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:38.670 DEBUG:boto:Port: 8001 2023-05-24T21:11:38.670 DEBUG:boto:Params: {} 2023-05-24T21:11:38.670 DEBUG:boto:Token: None 2023-05-24T21:11:38.671 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/ 2023-05-24T21:11:38.671 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5e2RJlSUQ8Y5t134TUlreZP6Nas= 2023-05-24T21:11:38.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5e2RJlSUQ8Y5t134TUlreZP6Nas=', 'Content-Length': '0'} 2023-05-24T21:11:38.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091eedab868fa0410-00646e7d8a-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.716 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:38.717 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:38.717 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:38.718 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:38.718 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:38.719 DEBUG:boto:Method: GET 2023-05-24T21:11:38.719 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:38.719 DEBUG:boto:Data: 2023-05-24T21:11:38.720 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.720 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:38.720 DEBUG:boto:Port: 8000 2023-05-24T21:11:38.721 DEBUG:boto:Params: {} 2023-05-24T21:11:38.721 DEBUG:boto:Token: None 2023-05-24T21:11:38.722 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/ 2023-05-24T21:11:38.722 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5e2RJlSUQ8Y5t134TUlreZP6Nas= 2023-05-24T21:11:38.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5e2RJlSUQ8Y5t134TUlreZP6Nas=', 'Content-Length': '0'} 2023-05-24T21:11:38.725 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070d26054441cd1e5-00646e7d8a-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.725 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:38.726 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.727 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.727 DEBUG:boto:Method: HEAD 2023-05-24T21:11:38.727 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:11:38.728 DEBUG:boto:Data: 2023-05-24T21:11:38.728 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.728 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:38.729 DEBUG:boto:Port: 8001 2023-05-24T21:11:38.729 DEBUG:boto:Params: {} 2023-05-24T21:11:38.729 DEBUG:boto:Token: None 2023-05-24T21:11:38.730 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj 2023-05-24T21:11:38.730 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ed3FFLYS1b/XkKX9hkntV4II6A8= 2023-05-24T21:11:38.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ed3FFLYS1b/XkKX9hkntV4II6A8=', 'Content-Length': '0'} 2023-05-24T21:11:38.734 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000296cf94ebd13e9fa-00646e7d8a-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.734 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.735 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.735 DEBUG:boto:Method: HEAD 2023-05-24T21:11:38.736 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:11:38.736 DEBUG:boto:Data: 2023-05-24T21:11:38.736 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.737 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:38.737 DEBUG:boto:Port: 8000 2023-05-24T21:11:38.737 DEBUG:boto:Params: {} 2023-05-24T21:11:38.738 DEBUG:boto:Token: None 2023-05-24T21:11:38.738 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj 2023-05-24T21:11:38.738 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ed3FFLYS1b/XkKX9hkntV4II6A8= 2023-05-24T21:11:38.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ed3FFLYS1b/XkKX9hkntV4II6A8=', 'Content-Length': '0'} 2023-05-24T21:11:38.742 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000036f0bf89939b75e5-00646e7d8a-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:38.743 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.743 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.744 DEBUG:boto:path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.744 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.744 DEBUG:boto:Method: GET 2023-05-24T21:11:38.745 DEBUG:boto:Path: /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.745 DEBUG:boto:Data: 2023-05-24T21:11:38.746 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.746 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:38.746 DEBUG:boto:Port: 8001 2023-05-24T21:11:38.747 DEBUG:boto:Params: {} 2023-05-24T21:11:38.747 DEBUG:boto:Token: None 2023-05-24T21:11:38.748 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.748 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Qy3zwRpx5lVG+UJdWDSmNUuL1CE= 2023-05-24T21:11:38.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Qy3zwRpx5lVG+UJdWDSmNUuL1CE=', 'Content-Length': '0'} 2023-05-24T21:11:38.750 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000004b5e511f0ccb3af-00646e7d8a-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.751 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.751 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.752 DEBUG:boto:path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.752 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.753 DEBUG:boto:Method: GET 2023-05-24T21:11:38.753 DEBUG:boto:Path: /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.753 DEBUG:boto:Data: 2023-05-24T21:11:38.754 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.754 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:38.755 DEBUG:boto:Port: 8000 2023-05-24T21:11:38.755 DEBUG:boto:Params: {} 2023-05-24T21:11:38.755 DEBUG:boto:Token: None 2023-05-24T21:11:38.756 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.756 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Qy3zwRpx5lVG+UJdWDSmNUuL1CE= 2023-05-24T21:11:38.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Qy3zwRpx5lVG+UJdWDSmNUuL1CE=', 'Content-Length': '0'} 2023-05-24T21:11:38.759 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000094b71cf3b0c19e9e-00646e7d8a-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.804 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:38.805 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:38.805 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:38.806 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:38.806 DEBUG:boto:Method: GET 2023-05-24T21:11:38.806 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:38.807 DEBUG:boto:Data: 2023-05-24T21:11:38.807 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.808 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:38.808 DEBUG:boto:Port: 8001 2023-05-24T21:11:38.808 DEBUG:boto:Params: {} 2023-05-24T21:11:38.809 DEBUG:boto:Token: None 2023-05-24T21:11:38.809 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/ 2023-05-24T21:11:38.810 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5e2RJlSUQ8Y5t134TUlreZP6Nas= 2023-05-24T21:11:38.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5e2RJlSUQ8Y5t134TUlreZP6Nas=', 'Content-Length': '0'} 2023-05-24T21:11:38.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018783fc29b635078-00646e7d8a-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.813 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:38.814 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:38.814 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:38.815 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:38.815 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:38.815 DEBUG:boto:Method: GET 2023-05-24T21:11:38.816 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:38.816 DEBUG:boto:Data: 2023-05-24T21:11:38.816 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.817 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:38.817 DEBUG:boto:Port: 8000 2023-05-24T21:11:38.818 DEBUG:boto:Params: {} 2023-05-24T21:11:38.818 DEBUG:boto:Token: None 2023-05-24T21:11:38.818 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/ 2023-05-24T21:11:38.819 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5e2RJlSUQ8Y5t134TUlreZP6Nas= 2023-05-24T21:11:38.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5e2RJlSUQ8Y5t134TUlreZP6Nas=', 'Content-Length': '0'} 2023-05-24T21:11:38.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a9682ec94812a5f-00646e7d8a-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.822 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:38.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:38.823 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.824 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.824 DEBUG:boto:path=/qskrps-13/obj?versionId=U4zUwdtKVyScPebZNZT8NHXc8rQ.8f- 2023-05-24T21:11:38.824 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=U4zUwdtKVyScPebZNZT8NHXc8rQ.8f- 2023-05-24T21:11:38.825 DEBUG:boto:Method: GET 2023-05-24T21:11:38.825 DEBUG:boto:Path: /qskrps-13/obj?versionId=U4zUwdtKVyScPebZNZT8NHXc8rQ.8f- 2023-05-24T21:11:38.825 DEBUG:boto:Data: 2023-05-24T21:11:38.826 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.826 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:38.827 DEBUG:boto:Port: 8001 2023-05-24T21:11:38.827 DEBUG:boto:Params: {} 2023-05-24T21:11:38.827 DEBUG:boto:Token: None 2023-05-24T21:11:38.828 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj?versionId=U4zUwdtKVyScPebZNZT8NHXc8rQ.8f- 2023-05-24T21:11:38.828 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:yNYrikZsb1k/oi0y7/sGFdewPfM= 2023-05-24T21:11:38.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:yNYrikZsb1k/oi0y7/sGFdewPfM=', 'Content-Length': '0'} 2023-05-24T21:11:38.831 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'U4zUwdtKVyScPebZNZT8NHXc8rQ.8f-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005c72104f97530c8c-00646e7d8a-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.872 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.873 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.873 DEBUG:boto:path=/qskrps-13/obj?versionId=U4zUwdtKVyScPebZNZT8NHXc8rQ.8f- 2023-05-24T21:11:38.874 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=U4zUwdtKVyScPebZNZT8NHXc8rQ.8f- 2023-05-24T21:11:38.874 DEBUG:boto:Method: GET 2023-05-24T21:11:38.874 DEBUG:boto:Path: /qskrps-13/obj?versionId=U4zUwdtKVyScPebZNZT8NHXc8rQ.8f- 2023-05-24T21:11:38.875 DEBUG:boto:Data: 2023-05-24T21:11:38.875 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.875 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:38.876 DEBUG:boto:Port: 8000 2023-05-24T21:11:38.876 DEBUG:boto:Params: {} 2023-05-24T21:11:38.877 DEBUG:boto:Token: None 2023-05-24T21:11:38.877 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj?versionId=U4zUwdtKVyScPebZNZT8NHXc8rQ.8f- 2023-05-24T21:11:38.878 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:yNYrikZsb1k/oi0y7/sGFdewPfM= 2023-05-24T21:11:38.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:yNYrikZsb1k/oi0y7/sGFdewPfM=', 'Content-Length': '0'} 2023-05-24T21:11:38.880 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'U4zUwdtKVyScPebZNZT8NHXc8rQ.8f-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000933e466dee0d085f-00646e7d8a-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.881 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.881 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.881 DEBUG:boto:path=/qskrps-13/obj?versionId=U4zUwdtKVyScPebZNZT8NHXc8rQ.8f- 2023-05-24T21:11:38.882 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=U4zUwdtKVyScPebZNZT8NHXc8rQ.8f- 2023-05-24T21:11:38.882 DEBUG:boto:Method: HEAD 2023-05-24T21:11:38.882 DEBUG:boto:Path: /qskrps-13/obj?versionId=U4zUwdtKVyScPebZNZT8NHXc8rQ.8f- 2023-05-24T21:11:38.883 DEBUG:boto:Data: 2023-05-24T21:11:38.883 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.883 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:38.884 DEBUG:boto:Port: 8001 2023-05-24T21:11:38.884 DEBUG:boto:Params: {} 2023-05-24T21:11:38.885 DEBUG:boto:Token: None 2023-05-24T21:11:38.885 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj?versionId=U4zUwdtKVyScPebZNZT8NHXc8rQ.8f- 2023-05-24T21:11:38.885 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:eE6tX3e18vrh0cj7Ctpu91oYaBY= 2023-05-24T21:11:38.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:eE6tX3e18vrh0cj7Ctpu91oYaBY=', 'Content-Length': '0'} 2023-05-24T21:11:38.887 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'U4zUwdtKVyScPebZNZT8NHXc8rQ.8f-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000d98b792141515cc-00646e7d8a-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.888 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.888 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.889 DEBUG:boto:path=/qskrps-13/obj?versionId=U4zUwdtKVyScPebZNZT8NHXc8rQ.8f- 2023-05-24T21:11:38.889 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=U4zUwdtKVyScPebZNZT8NHXc8rQ.8f- 2023-05-24T21:11:38.889 DEBUG:boto:Method: HEAD 2023-05-24T21:11:38.890 DEBUG:boto:Path: /qskrps-13/obj?versionId=U4zUwdtKVyScPebZNZT8NHXc8rQ.8f- 2023-05-24T21:11:38.890 DEBUG:boto:Data: 2023-05-24T21:11:38.890 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.891 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:38.891 DEBUG:boto:Port: 8000 2023-05-24T21:11:38.891 DEBUG:boto:Params: {} 2023-05-24T21:11:38.892 DEBUG:boto:Token: None 2023-05-24T21:11:38.892 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj?versionId=U4zUwdtKVyScPebZNZT8NHXc8rQ.8f- 2023-05-24T21:11:38.893 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:eE6tX3e18vrh0cj7Ctpu91oYaBY= 2023-05-24T21:11:38.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:eE6tX3e18vrh0cj7Ctpu91oYaBY=', 'Content-Length': '0'} 2023-05-24T21:11:38.895 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'U4zUwdtKVyScPebZNZT8NHXc8rQ.8f-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c1f2c249c99b1d2c-00646e7d8a-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:38.896 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.896 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.896 DEBUG:boto:path=/qskrps-13/obj?versionId=U4zUwdtKVyScPebZNZT8NHXc8rQ.8f- 2023-05-24T21:11:38.897 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=U4zUwdtKVyScPebZNZT8NHXc8rQ.8f- 2023-05-24T21:11:38.897 DEBUG:boto:Method: GET 2023-05-24T21:11:38.897 DEBUG:boto:Path: /qskrps-13/obj?versionId=U4zUwdtKVyScPebZNZT8NHXc8rQ.8f- 2023-05-24T21:11:38.898 DEBUG:boto:Data: 2023-05-24T21:11:38.898 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.898 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:38.899 DEBUG:boto:Port: 8001 2023-05-24T21:11:38.899 DEBUG:boto:Params: {} 2023-05-24T21:11:38.900 DEBUG:boto:Token: None 2023-05-24T21:11:38.900 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj?versionId=U4zUwdtKVyScPebZNZT8NHXc8rQ.8f- 2023-05-24T21:11:38.900 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:yNYrikZsb1k/oi0y7/sGFdewPfM= 2023-05-24T21:11:38.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:yNYrikZsb1k/oi0y7/sGFdewPfM=', 'Content-Length': '0'} 2023-05-24T21:11:38.903 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'U4zUwdtKVyScPebZNZT8NHXc8rQ.8f-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dca485ca7808bed8-00646e7d8a-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.903 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.904 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.904 DEBUG:boto:path=/qskrps-13/obj?versionId=U4zUwdtKVyScPebZNZT8NHXc8rQ.8f- 2023-05-24T21:11:38.905 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=U4zUwdtKVyScPebZNZT8NHXc8rQ.8f- 2023-05-24T21:11:38.905 DEBUG:boto:Method: GET 2023-05-24T21:11:38.905 DEBUG:boto:Path: /qskrps-13/obj?versionId=U4zUwdtKVyScPebZNZT8NHXc8rQ.8f- 2023-05-24T21:11:38.906 DEBUG:boto:Data: 2023-05-24T21:11:38.906 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.906 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:38.907 DEBUG:boto:Port: 8000 2023-05-24T21:11:38.907 DEBUG:boto:Params: {} 2023-05-24T21:11:38.907 DEBUG:boto:Token: None 2023-05-24T21:11:38.908 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj?versionId=U4zUwdtKVyScPebZNZT8NHXc8rQ.8f- 2023-05-24T21:11:38.908 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:yNYrikZsb1k/oi0y7/sGFdewPfM= 2023-05-24T21:11:38.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:yNYrikZsb1k/oi0y7/sGFdewPfM=', 'Content-Length': '0'} 2023-05-24T21:11:38.910 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'U4zUwdtKVyScPebZNZT8NHXc8rQ.8f-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000251d980cdf5d027a-00646e7d8a-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.911 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:38.912 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:38.912 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:38.912 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:38.913 DEBUG:boto:Method: GET 2023-05-24T21:11:38.913 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:38.913 DEBUG:boto:Data: 2023-05-24T21:11:38.914 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.914 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:38.914 DEBUG:boto:Port: 8001 2023-05-24T21:11:38.915 DEBUG:boto:Params: {} 2023-05-24T21:11:38.915 DEBUG:boto:Token: None 2023-05-24T21:11:38.916 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/ 2023-05-24T21:11:38.916 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5e2RJlSUQ8Y5t134TUlreZP6Nas= 2023-05-24T21:11:38.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5e2RJlSUQ8Y5t134TUlreZP6Nas=', 'Content-Length': '0'} 2023-05-24T21:11:38.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036918cc4c11c3b37-00646e7d8a-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.919 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:38.920 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:38.920 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:38.921 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:38.921 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:38.921 DEBUG:boto:Method: GET 2023-05-24T21:11:38.922 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:38.922 DEBUG:boto:Data: 2023-05-24T21:11:38.922 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.923 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:38.923 DEBUG:boto:Port: 8000 2023-05-24T21:11:38.923 DEBUG:boto:Params: {} 2023-05-24T21:11:38.924 DEBUG:boto:Token: None 2023-05-24T21:11:38.924 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/ 2023-05-24T21:11:38.925 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5e2RJlSUQ8Y5t134TUlreZP6Nas= 2023-05-24T21:11:38.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5e2RJlSUQ8Y5t134TUlreZP6Nas=', 'Content-Length': '0'} 2023-05-24T21:11:38.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e0b1059f8917a8a-00646e7d8a-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.928 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:38.929 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.929 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.929 DEBUG:boto:Method: HEAD 2023-05-24T21:11:38.930 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:11:38.930 DEBUG:boto:Data: 2023-05-24T21:11:38.930 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.931 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:38.931 DEBUG:boto:Port: 8001 2023-05-24T21:11:38.931 DEBUG:boto:Params: {} 2023-05-24T21:11:38.932 DEBUG:boto:Token: None 2023-05-24T21:11:38.932 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj 2023-05-24T21:11:38.932 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ed3FFLYS1b/XkKX9hkntV4II6A8= 2023-05-24T21:11:38.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ed3FFLYS1b/XkKX9hkntV4II6A8=', 'Content-Length': '0'} 2023-05-24T21:11:38.936 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d854c031e6dc92d7-00646e7d8a-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.936 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.937 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.937 DEBUG:boto:Method: HEAD 2023-05-24T21:11:38.937 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:11:38.938 DEBUG:boto:Data: 2023-05-24T21:11:38.938 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.938 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:38.939 DEBUG:boto:Port: 8000 2023-05-24T21:11:38.939 DEBUG:boto:Params: {} 2023-05-24T21:11:38.939 DEBUG:boto:Token: None 2023-05-24T21:11:38.940 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj 2023-05-24T21:11:38.940 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ed3FFLYS1b/XkKX9hkntV4II6A8= 2023-05-24T21:11:38.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ed3FFLYS1b/XkKX9hkntV4II6A8=', 'Content-Length': '0'} 2023-05-24T21:11:38.943 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e6b089250553bc3-00646e7d8a-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:38.944 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.945 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.945 DEBUG:boto:path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.945 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.946 DEBUG:boto:Method: GET 2023-05-24T21:11:38.946 DEBUG:boto:Path: /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.946 DEBUG:boto:Data: 2023-05-24T21:11:38.947 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.947 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:38.948 DEBUG:boto:Port: 8001 2023-05-24T21:11:38.948 DEBUG:boto:Params: {} 2023-05-24T21:11:38.948 DEBUG:boto:Token: None 2023-05-24T21:11:38.949 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.949 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Qy3zwRpx5lVG+UJdWDSmNUuL1CE= 2023-05-24T21:11:38.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Qy3zwRpx5lVG+UJdWDSmNUuL1CE=', 'Content-Length': '0'} 2023-05-24T21:11:38.951 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f5eaabc02bd3869-00646e7d8a-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.952 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.953 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.953 DEBUG:boto:path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.953 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.954 DEBUG:boto:Method: GET 2023-05-24T21:11:38.954 DEBUG:boto:Path: /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.954 DEBUG:boto:Data: 2023-05-24T21:11:38.955 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.955 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:38.956 DEBUG:boto:Port: 8000 2023-05-24T21:11:38.956 DEBUG:boto:Params: {} 2023-05-24T21:11:38.956 DEBUG:boto:Token: None 2023-05-24T21:11:38.957 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:38.957 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Qy3zwRpx5lVG+UJdWDSmNUuL1CE= 2023-05-24T21:11:38.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Qy3zwRpx5lVG+UJdWDSmNUuL1CE=', 'Content-Length': '0'} 2023-05-24T21:11:38.959 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000369ac559fbbad552-00646e7d8a-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.960 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:38.960 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:38.961 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:38.961 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:38.961 DEBUG:boto:Method: GET 2023-05-24T21:11:38.962 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:38.962 DEBUG:boto:Data: 2023-05-24T21:11:38.963 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.963 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:38.963 DEBUG:boto:Port: 8001 2023-05-24T21:11:38.964 DEBUG:boto:Params: {} 2023-05-24T21:11:38.964 DEBUG:boto:Token: None 2023-05-24T21:11:38.964 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/ 2023-05-24T21:11:38.965 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5e2RJlSUQ8Y5t134TUlreZP6Nas= 2023-05-24T21:11:38.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5e2RJlSUQ8Y5t134TUlreZP6Nas=', 'Content-Length': '0'} 2023-05-24T21:11:38.968 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec6be1dd0bc72586-00646e7d8a-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.968 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:38.969 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:38.970 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:38.970 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:38.970 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:38.971 DEBUG:boto:Method: GET 2023-05-24T21:11:38.971 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:38.971 DEBUG:boto:Data: 2023-05-24T21:11:38.972 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.972 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:38.972 DEBUG:boto:Port: 8000 2023-05-24T21:11:38.973 DEBUG:boto:Params: {} 2023-05-24T21:11:38.973 DEBUG:boto:Token: None 2023-05-24T21:11:38.974 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/ 2023-05-24T21:11:38.974 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5e2RJlSUQ8Y5t134TUlreZP6Nas= 2023-05-24T21:11:38.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5e2RJlSUQ8Y5t134TUlreZP6Nas=', 'Content-Length': '0'} 2023-05-24T21:11:38.977 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd70d551ffc96f35-00646e7d8a-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.978 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:38.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:38.979 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.979 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.980 DEBUG:boto:path=/qskrps-13/obj?versionId=1BSwhaY-MjwOOLy73m-3KiPjCrRBsod 2023-05-24T21:11:38.980 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=1BSwhaY-MjwOOLy73m-3KiPjCrRBsod 2023-05-24T21:11:38.980 DEBUG:boto:Method: GET 2023-05-24T21:11:38.981 DEBUG:boto:Path: /qskrps-13/obj?versionId=1BSwhaY-MjwOOLy73m-3KiPjCrRBsod 2023-05-24T21:11:38.981 DEBUG:boto:Data: 2023-05-24T21:11:38.981 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.982 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:38.982 DEBUG:boto:Port: 8001 2023-05-24T21:11:38.983 DEBUG:boto:Params: {} 2023-05-24T21:11:38.983 DEBUG:boto:Token: None 2023-05-24T21:11:38.984 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj?versionId=1BSwhaY-MjwOOLy73m-3KiPjCrRBsod 2023-05-24T21:11:38.984 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:3wwKjWnIchHuGogZan1g6oATWns= 2023-05-24T21:11:38.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:3wwKjWnIchHuGogZan1g6oATWns=', 'Content-Length': '0'} 2023-05-24T21:11:38.986 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', '1BSwhaY-MjwOOLy73m-3KiPjCrRBsod'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fdb29a903abc75a2-00646e7d8a-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:38.987 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:38.987 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:38.988 DEBUG:boto:path=/qskrps-13/obj?versionId=1BSwhaY-MjwOOLy73m-3KiPjCrRBsod 2023-05-24T21:11:38.988 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=1BSwhaY-MjwOOLy73m-3KiPjCrRBsod 2023-05-24T21:11:38.989 DEBUG:boto:Method: GET 2023-05-24T21:11:38.989 DEBUG:boto:Path: /qskrps-13/obj?versionId=1BSwhaY-MjwOOLy73m-3KiPjCrRBsod 2023-05-24T21:11:38.989 DEBUG:boto:Data: 2023-05-24T21:11:38.990 DEBUG:boto:Headers: {} 2023-05-24T21:11:38.990 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:38.990 DEBUG:boto:Port: 8000 2023-05-24T21:11:38.991 DEBUG:boto:Params: {} 2023-05-24T21:11:38.991 DEBUG:boto:Token: None 2023-05-24T21:11:38.991 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:38 GMT /qskrps-13/obj?versionId=1BSwhaY-MjwOOLy73m-3KiPjCrRBsod 2023-05-24T21:11:38.992 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:3wwKjWnIchHuGogZan1g6oATWns= 2023-05-24T21:11:38.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:3wwKjWnIchHuGogZan1g6oATWns=', 'Content-Length': '0'} 2023-05-24T21:11:38.994 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', '1BSwhaY-MjwOOLy73m-3KiPjCrRBsod'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000096bdf593c7961ef2-00646e7d8a-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.036 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:39.037 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:39.037 DEBUG:boto:path=/qskrps-13/obj?versionId=1BSwhaY-MjwOOLy73m-3KiPjCrRBsod 2023-05-24T21:11:39.038 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=1BSwhaY-MjwOOLy73m-3KiPjCrRBsod 2023-05-24T21:11:39.038 DEBUG:boto:Method: HEAD 2023-05-24T21:11:39.038 DEBUG:boto:Path: /qskrps-13/obj?versionId=1BSwhaY-MjwOOLy73m-3KiPjCrRBsod 2023-05-24T21:11:39.039 DEBUG:boto:Data: 2023-05-24T21:11:39.039 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.039 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:39.040 DEBUG:boto:Port: 8001 2023-05-24T21:11:39.040 DEBUG:boto:Params: {} 2023-05-24T21:11:39.040 DEBUG:boto:Token: None 2023-05-24T21:11:39.041 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:39 GMT /qskrps-13/obj?versionId=1BSwhaY-MjwOOLy73m-3KiPjCrRBsod 2023-05-24T21:11:39.041 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:KoDupllhnpthRGS1CFwB6wc9f/0= 2023-05-24T21:11:39.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:KoDupllhnpthRGS1CFwB6wc9f/0=', 'Content-Length': '0'} 2023-05-24T21:11:39.044 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', '1BSwhaY-MjwOOLy73m-3KiPjCrRBsod'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002e3a26779e7ef7c1-00646e7d8b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.044 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:39.045 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:39.045 DEBUG:boto:path=/qskrps-13/obj?versionId=1BSwhaY-MjwOOLy73m-3KiPjCrRBsod 2023-05-24T21:11:39.045 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=1BSwhaY-MjwOOLy73m-3KiPjCrRBsod 2023-05-24T21:11:39.046 DEBUG:boto:Method: HEAD 2023-05-24T21:11:39.046 DEBUG:boto:Path: /qskrps-13/obj?versionId=1BSwhaY-MjwOOLy73m-3KiPjCrRBsod 2023-05-24T21:11:39.046 DEBUG:boto:Data: 2023-05-24T21:11:39.047 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.047 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:39.048 DEBUG:boto:Port: 8000 2023-05-24T21:11:39.048 DEBUG:boto:Params: {} 2023-05-24T21:11:39.049 DEBUG:boto:Token: None 2023-05-24T21:11:39.049 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:39 GMT /qskrps-13/obj?versionId=1BSwhaY-MjwOOLy73m-3KiPjCrRBsod 2023-05-24T21:11:39.050 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:KoDupllhnpthRGS1CFwB6wc9f/0= 2023-05-24T21:11:39.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:KoDupllhnpthRGS1CFwB6wc9f/0=', 'Content-Length': '0'} 2023-05-24T21:11:39.052 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', '1BSwhaY-MjwOOLy73m-3KiPjCrRBsod'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ac2daa5ba362cebf-00646e7d8b-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:39.053 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:39.053 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:39.054 DEBUG:boto:path=/qskrps-13/obj?versionId=1BSwhaY-MjwOOLy73m-3KiPjCrRBsod 2023-05-24T21:11:39.055 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=1BSwhaY-MjwOOLy73m-3KiPjCrRBsod 2023-05-24T21:11:39.055 DEBUG:boto:Method: GET 2023-05-24T21:11:39.056 DEBUG:boto:Path: /qskrps-13/obj?versionId=1BSwhaY-MjwOOLy73m-3KiPjCrRBsod 2023-05-24T21:11:39.056 DEBUG:boto:Data: 2023-05-24T21:11:39.057 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.057 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:39.058 DEBUG:boto:Port: 8001 2023-05-24T21:11:39.059 DEBUG:boto:Params: {} 2023-05-24T21:11:39.059 DEBUG:boto:Token: None 2023-05-24T21:11:39.060 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:39 GMT /qskrps-13/obj?versionId=1BSwhaY-MjwOOLy73m-3KiPjCrRBsod 2023-05-24T21:11:39.061 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:+6LiexK/HE8W1trFTY0ArmUC4vo= 2023-05-24T21:11:39.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:+6LiexK/HE8W1trFTY0ArmUC4vo=', 'Content-Length': '0'} 2023-05-24T21:11:39.064 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', '1BSwhaY-MjwOOLy73m-3KiPjCrRBsod'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000d5af309e6a8ae3d-00646e7d8b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.065 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:39.065 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:39.066 DEBUG:boto:path=/qskrps-13/obj?versionId=1BSwhaY-MjwOOLy73m-3KiPjCrRBsod 2023-05-24T21:11:39.067 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=1BSwhaY-MjwOOLy73m-3KiPjCrRBsod 2023-05-24T21:11:39.067 DEBUG:boto:Method: GET 2023-05-24T21:11:39.068 DEBUG:boto:Path: /qskrps-13/obj?versionId=1BSwhaY-MjwOOLy73m-3KiPjCrRBsod 2023-05-24T21:11:39.068 DEBUG:boto:Data: 2023-05-24T21:11:39.068 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.069 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:39.069 DEBUG:boto:Port: 8000 2023-05-24T21:11:39.070 DEBUG:boto:Params: {} 2023-05-24T21:11:39.070 DEBUG:boto:Token: None 2023-05-24T21:11:39.070 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:39 GMT /qskrps-13/obj?versionId=1BSwhaY-MjwOOLy73m-3KiPjCrRBsod 2023-05-24T21:11:39.071 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:+6LiexK/HE8W1trFTY0ArmUC4vo= 2023-05-24T21:11:39.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:+6LiexK/HE8W1trFTY0ArmUC4vo=', 'Content-Length': '0'} 2023-05-24T21:11:39.073 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', '1BSwhaY-MjwOOLy73m-3KiPjCrRBsod'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000567806a06a1a3d5-00646e7d8b-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.074 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:39.074 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:39.075 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:39.075 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:39.075 DEBUG:boto:Method: GET 2023-05-24T21:11:39.076 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:39.076 DEBUG:boto:Data: 2023-05-24T21:11:39.076 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.077 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:39.077 DEBUG:boto:Port: 8001 2023-05-24T21:11:39.078 DEBUG:boto:Params: {} 2023-05-24T21:11:39.078 DEBUG:boto:Token: None 2023-05-24T21:11:39.078 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:39 GMT /qskrps-13/ 2023-05-24T21:11:39.079 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:KLzthF2FvsaZLlFQbgC1hHucCR8= 2023-05-24T21:11:39.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:KLzthF2FvsaZLlFQbgC1hHucCR8=', 'Content-Length': '0'} 2023-05-24T21:11:39.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095bae2c66a138601-00646e7d8b-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.082 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:39.083 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:39.083 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:39.084 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:39.084 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:39.085 DEBUG:boto:Method: GET 2023-05-24T21:11:39.085 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:39.085 DEBUG:boto:Data: 2023-05-24T21:11:39.086 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.086 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:39.087 DEBUG:boto:Port: 8000 2023-05-24T21:11:39.087 DEBUG:boto:Params: {} 2023-05-24T21:11:39.087 DEBUG:boto:Token: None 2023-05-24T21:11:39.088 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:39 GMT /qskrps-13/ 2023-05-24T21:11:39.088 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:KLzthF2FvsaZLlFQbgC1hHucCR8= 2023-05-24T21:11:39.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:KLzthF2FvsaZLlFQbgC1hHucCR8=', 'Content-Length': '0'} 2023-05-24T21:11:39.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cff8b6bae369e8de-00646e7d8b-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.092 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:39.092 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:39.093 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:39.093 DEBUG:boto:Method: HEAD 2023-05-24T21:11:39.094 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:11:39.094 DEBUG:boto:Data: 2023-05-24T21:11:39.094 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.095 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:39.095 DEBUG:boto:Port: 8001 2023-05-24T21:11:39.095 DEBUG:boto:Params: {} 2023-05-24T21:11:39.096 DEBUG:boto:Token: None 2023-05-24T21:11:39.096 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:39 GMT /qskrps-13/obj 2023-05-24T21:11:39.097 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:W70bsBNr+FGf5zONb0uhNNkcUOw= 2023-05-24T21:11:39.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:W70bsBNr+FGf5zONb0uhNNkcUOw=', 'Content-Length': '0'} 2023-05-24T21:11:39.101 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f5ded34b988c84a8-00646e7d8b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.101 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:39.102 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:39.102 DEBUG:boto:Method: HEAD 2023-05-24T21:11:39.102 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:11:39.103 DEBUG:boto:Data: 2023-05-24T21:11:39.103 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.103 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:39.104 DEBUG:boto:Port: 8000 2023-05-24T21:11:39.104 DEBUG:boto:Params: {} 2023-05-24T21:11:39.104 DEBUG:boto:Token: None 2023-05-24T21:11:39.105 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:39 GMT /qskrps-13/obj 2023-05-24T21:11:39.105 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:W70bsBNr+FGf5zONb0uhNNkcUOw= 2023-05-24T21:11:39.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:W70bsBNr+FGf5zONb0uhNNkcUOw=', 'Content-Length': '0'} 2023-05-24T21:11:39.109 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fb36b1639a9946b9-00646e7d8b-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:39.110 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:39.110 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:39.111 DEBUG:boto:path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:39.111 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:39.111 DEBUG:boto:Method: GET 2023-05-24T21:11:39.112 DEBUG:boto:Path: /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:39.112 DEBUG:boto:Data: 2023-05-24T21:11:39.113 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.113 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:39.113 DEBUG:boto:Port: 8001 2023-05-24T21:11:39.114 DEBUG:boto:Params: {} 2023-05-24T21:11:39.114 DEBUG:boto:Token: None 2023-05-24T21:11:39.114 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:39 GMT /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:39.115 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:wO031kG+mToKWqIEeiJQuipXtgM= 2023-05-24T21:11:39.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wO031kG+mToKWqIEeiJQuipXtgM=', 'Content-Length': '0'} 2023-05-24T21:11:39.117 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000644fc81752754ac8-00646e7d8b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.164 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:39.165 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:39.165 DEBUG:boto:path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:39.166 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:39.166 DEBUG:boto:Method: GET 2023-05-24T21:11:39.166 DEBUG:boto:Path: /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:39.167 DEBUG:boto:Data: 2023-05-24T21:11:39.167 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.167 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:39.168 DEBUG:boto:Port: 8000 2023-05-24T21:11:39.168 DEBUG:boto:Params: {} 2023-05-24T21:11:39.169 DEBUG:boto:Token: None 2023-05-24T21:11:39.169 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:39 GMT /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:39.170 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:wO031kG+mToKWqIEeiJQuipXtgM= 2023-05-24T21:11:39.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wO031kG+mToKWqIEeiJQuipXtgM=', 'Content-Length': '0'} 2023-05-24T21:11:39.172 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c496567041b3a682-00646e7d8b-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.173 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:39.173 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:39.173 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:39.174 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:39.174 DEBUG:boto:Method: GET 2023-05-24T21:11:39.175 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:39.175 DEBUG:boto:Data: 2023-05-24T21:11:39.175 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.176 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:39.176 DEBUG:boto:Port: 8001 2023-05-24T21:11:39.176 DEBUG:boto:Params: {} 2023-05-24T21:11:39.177 DEBUG:boto:Token: None 2023-05-24T21:11:39.177 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:39 GMT /qskrps-13/ 2023-05-24T21:11:39.178 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:KLzthF2FvsaZLlFQbgC1hHucCR8= 2023-05-24T21:11:39.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:KLzthF2FvsaZLlFQbgC1hHucCR8=', 'Content-Length': '0'} 2023-05-24T21:11:39.181 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f93fde533b3ed070-00646e7d8b-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.181 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:39.182 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:39.182 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:39.183 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:39.183 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:39.183 DEBUG:boto:Method: GET 2023-05-24T21:11:39.184 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:39.184 DEBUG:boto:Data: 2023-05-24T21:11:39.184 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.185 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:39.185 DEBUG:boto:Port: 8000 2023-05-24T21:11:39.185 DEBUG:boto:Params: {} 2023-05-24T21:11:39.186 DEBUG:boto:Token: None 2023-05-24T21:11:39.186 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:39 GMT /qskrps-13/ 2023-05-24T21:11:39.187 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:KLzthF2FvsaZLlFQbgC1hHucCR8= 2023-05-24T21:11:39.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:KLzthF2FvsaZLlFQbgC1hHucCR8=', 'Content-Length': '0'} 2023-05-24T21:11:39.190 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9c555634dbfb711-00646e7d8b-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.190 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:39.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:39.191 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:39.192 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:39.192 DEBUG:boto:path=/qskrps-13/obj?versionId=XBH9.ADrbaEY1JrgZ6O0QvO0TBVsG2v 2023-05-24T21:11:39.192 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=XBH9.ADrbaEY1JrgZ6O0QvO0TBVsG2v 2023-05-24T21:11:39.193 DEBUG:boto:Method: GET 2023-05-24T21:11:39.193 DEBUG:boto:Path: /qskrps-13/obj?versionId=XBH9.ADrbaEY1JrgZ6O0QvO0TBVsG2v 2023-05-24T21:11:39.193 DEBUG:boto:Data: 2023-05-24T21:11:39.194 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.194 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:39.195 DEBUG:boto:Port: 8001 2023-05-24T21:11:39.195 DEBUG:boto:Params: {} 2023-05-24T21:11:39.195 DEBUG:boto:Token: None 2023-05-24T21:11:39.196 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:39 GMT /qskrps-13/obj?versionId=XBH9.ADrbaEY1JrgZ6O0QvO0TBVsG2v 2023-05-24T21:11:39.196 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:RtUNMkSkz94+ZcHqKQW4YUWpI2U= 2023-05-24T21:11:39.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:RtUNMkSkz94+ZcHqKQW4YUWpI2U=', 'Content-Length': '0'} 2023-05-24T21:11:39.199 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'XBH9.ADrbaEY1JrgZ6O0QvO0TBVsG2v'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008da42a9c7344c391-00646e7d8b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.199 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:39.200 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:39.200 DEBUG:boto:path=/qskrps-13/obj?versionId=XBH9.ADrbaEY1JrgZ6O0QvO0TBVsG2v 2023-05-24T21:11:39.201 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=XBH9.ADrbaEY1JrgZ6O0QvO0TBVsG2v 2023-05-24T21:11:39.201 DEBUG:boto:Method: GET 2023-05-24T21:11:39.201 DEBUG:boto:Path: /qskrps-13/obj?versionId=XBH9.ADrbaEY1JrgZ6O0QvO0TBVsG2v 2023-05-24T21:11:39.202 DEBUG:boto:Data: 2023-05-24T21:11:39.202 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.202 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:39.203 DEBUG:boto:Port: 8000 2023-05-24T21:11:39.203 DEBUG:boto:Params: {} 2023-05-24T21:11:39.204 DEBUG:boto:Token: None 2023-05-24T21:11:39.204 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:39 GMT /qskrps-13/obj?versionId=XBH9.ADrbaEY1JrgZ6O0QvO0TBVsG2v 2023-05-24T21:11:39.204 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:RtUNMkSkz94+ZcHqKQW4YUWpI2U= 2023-05-24T21:11:39.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:RtUNMkSkz94+ZcHqKQW4YUWpI2U=', 'Content-Length': '0'} 2023-05-24T21:11:39.207 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'XBH9.ADrbaEY1JrgZ6O0QvO0TBVsG2v'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002aca1aaf4a13f2c5-00646e7d8b-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.248 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:39.249 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:39.249 DEBUG:boto:path=/qskrps-13/obj?versionId=XBH9.ADrbaEY1JrgZ6O0QvO0TBVsG2v 2023-05-24T21:11:39.250 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=XBH9.ADrbaEY1JrgZ6O0QvO0TBVsG2v 2023-05-24T21:11:39.250 DEBUG:boto:Method: HEAD 2023-05-24T21:11:39.250 DEBUG:boto:Path: /qskrps-13/obj?versionId=XBH9.ADrbaEY1JrgZ6O0QvO0TBVsG2v 2023-05-24T21:11:39.251 DEBUG:boto:Data: 2023-05-24T21:11:39.251 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.251 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:39.252 DEBUG:boto:Port: 8001 2023-05-24T21:11:39.252 DEBUG:boto:Params: {} 2023-05-24T21:11:39.252 DEBUG:boto:Token: None 2023-05-24T21:11:39.253 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:39 GMT /qskrps-13/obj?versionId=XBH9.ADrbaEY1JrgZ6O0QvO0TBVsG2v 2023-05-24T21:11:39.253 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:bI90Gj1/WR4r+XT37oXPaEjIuEI= 2023-05-24T21:11:39.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:bI90Gj1/WR4r+XT37oXPaEjIuEI=', 'Content-Length': '0'} 2023-05-24T21:11:39.255 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'XBH9.ADrbaEY1JrgZ6O0QvO0TBVsG2v'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000035b6397ec619561a-00646e7d8b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.256 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:39.256 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:39.257 DEBUG:boto:path=/qskrps-13/obj?versionId=XBH9.ADrbaEY1JrgZ6O0QvO0TBVsG2v 2023-05-24T21:11:39.257 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=XBH9.ADrbaEY1JrgZ6O0QvO0TBVsG2v 2023-05-24T21:11:39.257 DEBUG:boto:Method: HEAD 2023-05-24T21:11:39.258 DEBUG:boto:Path: /qskrps-13/obj?versionId=XBH9.ADrbaEY1JrgZ6O0QvO0TBVsG2v 2023-05-24T21:11:39.258 DEBUG:boto:Data: 2023-05-24T21:11:39.258 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.259 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:39.259 DEBUG:boto:Port: 8000 2023-05-24T21:11:39.259 DEBUG:boto:Params: {} 2023-05-24T21:11:39.260 DEBUG:boto:Token: None 2023-05-24T21:11:39.260 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:39 GMT /qskrps-13/obj?versionId=XBH9.ADrbaEY1JrgZ6O0QvO0TBVsG2v 2023-05-24T21:11:39.261 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:bI90Gj1/WR4r+XT37oXPaEjIuEI= 2023-05-24T21:11:39.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:bI90Gj1/WR4r+XT37oXPaEjIuEI=', 'Content-Length': '0'} 2023-05-24T21:11:39.263 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'XBH9.ADrbaEY1JrgZ6O0QvO0TBVsG2v'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000411f04cb326c8d12-00646e7d8b-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:39.264 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:39.264 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:39.264 DEBUG:boto:path=/qskrps-13/obj?versionId=XBH9.ADrbaEY1JrgZ6O0QvO0TBVsG2v 2023-05-24T21:11:39.265 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=XBH9.ADrbaEY1JrgZ6O0QvO0TBVsG2v 2023-05-24T21:11:39.265 DEBUG:boto:Method: GET 2023-05-24T21:11:39.265 DEBUG:boto:Path: /qskrps-13/obj?versionId=XBH9.ADrbaEY1JrgZ6O0QvO0TBVsG2v 2023-05-24T21:11:39.266 DEBUG:boto:Data: 2023-05-24T21:11:39.266 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.266 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:39.267 DEBUG:boto:Port: 8001 2023-05-24T21:11:39.267 DEBUG:boto:Params: {} 2023-05-24T21:11:39.268 DEBUG:boto:Token: None 2023-05-24T21:11:39.268 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:39 GMT /qskrps-13/obj?versionId=XBH9.ADrbaEY1JrgZ6O0QvO0TBVsG2v 2023-05-24T21:11:39.268 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:RtUNMkSkz94+ZcHqKQW4YUWpI2U= 2023-05-24T21:11:39.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:RtUNMkSkz94+ZcHqKQW4YUWpI2U=', 'Content-Length': '0'} 2023-05-24T21:11:39.271 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'XBH9.ADrbaEY1JrgZ6O0QvO0TBVsG2v'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000052c4ff2c3b86e3dc-00646e7d8b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.271 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:39.272 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:39.272 DEBUG:boto:path=/qskrps-13/obj?versionId=XBH9.ADrbaEY1JrgZ6O0QvO0TBVsG2v 2023-05-24T21:11:39.272 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=XBH9.ADrbaEY1JrgZ6O0QvO0TBVsG2v 2023-05-24T21:11:39.273 DEBUG:boto:Method: GET 2023-05-24T21:11:39.273 DEBUG:boto:Path: /qskrps-13/obj?versionId=XBH9.ADrbaEY1JrgZ6O0QvO0TBVsG2v 2023-05-24T21:11:39.273 DEBUG:boto:Data: 2023-05-24T21:11:39.274 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.274 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:39.275 DEBUG:boto:Port: 8000 2023-05-24T21:11:39.275 DEBUG:boto:Params: {} 2023-05-24T21:11:39.275 DEBUG:boto:Token: None 2023-05-24T21:11:39.276 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:39 GMT /qskrps-13/obj?versionId=XBH9.ADrbaEY1JrgZ6O0QvO0TBVsG2v 2023-05-24T21:11:39.276 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:RtUNMkSkz94+ZcHqKQW4YUWpI2U= 2023-05-24T21:11:39.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:RtUNMkSkz94+ZcHqKQW4YUWpI2U=', 'Content-Length': '0'} 2023-05-24T21:11:39.278 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'XBH9.ADrbaEY1JrgZ6O0QvO0TBVsG2v'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000417ab5e938901a1c-00646e7d8b-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.279 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:39.280 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:39.280 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:39.280 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:39.281 DEBUG:boto:Method: GET 2023-05-24T21:11:39.281 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:39.281 DEBUG:boto:Data: 2023-05-24T21:11:39.282 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.282 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:39.282 DEBUG:boto:Port: 8001 2023-05-24T21:11:39.283 DEBUG:boto:Params: {} 2023-05-24T21:11:39.283 DEBUG:boto:Token: None 2023-05-24T21:11:39.284 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:39 GMT /qskrps-13/ 2023-05-24T21:11:39.284 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:KLzthF2FvsaZLlFQbgC1hHucCR8= 2023-05-24T21:11:39.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:KLzthF2FvsaZLlFQbgC1hHucCR8=', 'Content-Length': '0'} 2023-05-24T21:11:39.287 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f7b0eeaf234e275-00646e7d8b-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.288 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:39.288 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:39.289 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:39.289 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:39.289 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:39.290 DEBUG:boto:Method: GET 2023-05-24T21:11:39.290 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:39.290 DEBUG:boto:Data: 2023-05-24T21:11:39.291 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.291 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:39.292 DEBUG:boto:Port: 8000 2023-05-24T21:11:39.292 DEBUG:boto:Params: {} 2023-05-24T21:11:39.292 DEBUG:boto:Token: None 2023-05-24T21:11:39.293 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:39 GMT /qskrps-13/ 2023-05-24T21:11:39.293 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:KLzthF2FvsaZLlFQbgC1hHucCR8= 2023-05-24T21:11:39.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:KLzthF2FvsaZLlFQbgC1hHucCR8=', 'Content-Length': '0'} 2023-05-24T21:11:39.296 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024f4be5fec2a8f4e-00646e7d8b-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.297 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:39.297 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:39.298 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:39.298 DEBUG:boto:Method: HEAD 2023-05-24T21:11:39.298 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:11:39.299 DEBUG:boto:Data: 2023-05-24T21:11:39.299 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.299 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:39.300 DEBUG:boto:Port: 8001 2023-05-24T21:11:39.300 DEBUG:boto:Params: {} 2023-05-24T21:11:39.300 DEBUG:boto:Token: None 2023-05-24T21:11:39.301 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:39 GMT /qskrps-13/obj 2023-05-24T21:11:39.301 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:W70bsBNr+FGf5zONb0uhNNkcUOw= 2023-05-24T21:11:39.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:W70bsBNr+FGf5zONb0uhNNkcUOw=', 'Content-Length': '0'} 2023-05-24T21:11:39.304 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000644482000a554317-00646e7d8b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.305 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:39.305 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:39.306 DEBUG:boto:Method: HEAD 2023-05-24T21:11:39.306 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:11:39.306 DEBUG:boto:Data: 2023-05-24T21:11:39.307 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.307 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:39.307 DEBUG:boto:Port: 8000 2023-05-24T21:11:39.308 DEBUG:boto:Params: {} 2023-05-24T21:11:39.308 DEBUG:boto:Token: None 2023-05-24T21:11:39.309 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:39 GMT /qskrps-13/obj 2023-05-24T21:11:39.309 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:W70bsBNr+FGf5zONb0uhNNkcUOw= 2023-05-24T21:11:39.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:W70bsBNr+FGf5zONb0uhNNkcUOw=', 'Content-Length': '0'} 2023-05-24T21:11:39.312 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe6c33ab69bb892e-00646e7d8b-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:39.313 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:39.313 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:39.314 DEBUG:boto:path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:39.314 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:39.315 DEBUG:boto:Method: GET 2023-05-24T21:11:39.315 DEBUG:boto:Path: /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:39.315 DEBUG:boto:Data: 2023-05-24T21:11:39.316 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.316 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:39.316 DEBUG:boto:Port: 8001 2023-05-24T21:11:39.317 DEBUG:boto:Params: {} 2023-05-24T21:11:39.317 DEBUG:boto:Token: None 2023-05-24T21:11:39.318 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:39 GMT /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:39.318 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:wO031kG+mToKWqIEeiJQuipXtgM= 2023-05-24T21:11:39.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wO031kG+mToKWqIEeiJQuipXtgM=', 'Content-Length': '0'} 2023-05-24T21:11:39.320 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fedc8c5987967793-00646e7d8b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.364 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:39.365 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:39.365 DEBUG:boto:path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:39.366 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:39.366 DEBUG:boto:Method: GET 2023-05-24T21:11:39.366 DEBUG:boto:Path: /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:39.367 DEBUG:boto:Data: 2023-05-24T21:11:39.367 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.367 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:39.368 DEBUG:boto:Port: 8000 2023-05-24T21:11:39.368 DEBUG:boto:Params: {} 2023-05-24T21:11:39.368 DEBUG:boto:Token: None 2023-05-24T21:11:39.369 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:39 GMT /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:39.369 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:wO031kG+mToKWqIEeiJQuipXtgM= 2023-05-24T21:11:39.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wO031kG+mToKWqIEeiJQuipXtgM=', 'Content-Length': '0'} 2023-05-24T21:11:39.372 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000db9b497ff18330bf-00646e7d8b-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.373 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:39.373 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:39.374 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:39.374 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:39.374 DEBUG:boto:Method: GET 2023-05-24T21:11:39.375 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:39.375 DEBUG:boto:Data: 2023-05-24T21:11:39.375 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.376 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:39.376 DEBUG:boto:Port: 8001 2023-05-24T21:11:39.376 DEBUG:boto:Params: {} 2023-05-24T21:11:39.377 DEBUG:boto:Token: None 2023-05-24T21:11:39.377 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:39 GMT /qskrps-13/ 2023-05-24T21:11:39.378 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:KLzthF2FvsaZLlFQbgC1hHucCR8= 2023-05-24T21:11:39.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:KLzthF2FvsaZLlFQbgC1hHucCR8=', 'Content-Length': '0'} 2023-05-24T21:11:39.381 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a576b3e89262b19e-00646e7d8b-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.381 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:39.382 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:39.383 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:39.383 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:39.383 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:39.384 DEBUG:boto:Method: GET 2023-05-24T21:11:39.384 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:39.384 DEBUG:boto:Data: 2023-05-24T21:11:39.385 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.386 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:39.386 DEBUG:boto:Port: 8000 2023-05-24T21:11:39.386 DEBUG:boto:Params: {} 2023-05-24T21:11:39.387 DEBUG:boto:Token: None 2023-05-24T21:11:39.387 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:39 GMT /qskrps-13/ 2023-05-24T21:11:39.388 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:KLzthF2FvsaZLlFQbgC1hHucCR8= 2023-05-24T21:11:39.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:KLzthF2FvsaZLlFQbgC1hHucCR8=', 'Content-Length': '0'} 2023-05-24T21:11:39.391 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f946c5fa45c83bbd-00646e7d8b-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.391 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:39.392 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:39.392 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:39.393 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:39.393 DEBUG:boto:path=/qskrps-13/obj?versionId=3wYQ13EDow0kiwE6ITIMyTWEPt0A0Hx 2023-05-24T21:11:39.393 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=3wYQ13EDow0kiwE6ITIMyTWEPt0A0Hx 2023-05-24T21:11:39.394 DEBUG:boto:Method: GET 2023-05-24T21:11:39.394 DEBUG:boto:Path: /qskrps-13/obj?versionId=3wYQ13EDow0kiwE6ITIMyTWEPt0A0Hx 2023-05-24T21:11:39.394 DEBUG:boto:Data: 2023-05-24T21:11:39.395 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.395 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:39.396 DEBUG:boto:Port: 8001 2023-05-24T21:11:39.396 DEBUG:boto:Params: {} 2023-05-24T21:11:39.397 DEBUG:boto:Token: None 2023-05-24T21:11:39.398 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:39 GMT /qskrps-13/obj?versionId=3wYQ13EDow0kiwE6ITIMyTWEPt0A0Hx 2023-05-24T21:11:39.398 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:CECbiH3Eo/CrJ82vkJEc3KSwqSw= 2023-05-24T21:11:39.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:CECbiH3Eo/CrJ82vkJEc3KSwqSw=', 'Content-Length': '0'} 2023-05-24T21:11:39.401 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', '3wYQ13EDow0kiwE6ITIMyTWEPt0A0Hx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000572884326d5d6e18-00646e7d8b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.401 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:39.402 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:39.402 DEBUG:boto:path=/qskrps-13/obj?versionId=3wYQ13EDow0kiwE6ITIMyTWEPt0A0Hx 2023-05-24T21:11:39.402 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=3wYQ13EDow0kiwE6ITIMyTWEPt0A0Hx 2023-05-24T21:11:39.403 DEBUG:boto:Method: GET 2023-05-24T21:11:39.403 DEBUG:boto:Path: /qskrps-13/obj?versionId=3wYQ13EDow0kiwE6ITIMyTWEPt0A0Hx 2023-05-24T21:11:39.404 DEBUG:boto:Data: 2023-05-24T21:11:39.404 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.404 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:39.405 DEBUG:boto:Port: 8000 2023-05-24T21:11:39.405 DEBUG:boto:Params: {} 2023-05-24T21:11:39.406 DEBUG:boto:Token: None 2023-05-24T21:11:39.406 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:39 GMT /qskrps-13/obj?versionId=3wYQ13EDow0kiwE6ITIMyTWEPt0A0Hx 2023-05-24T21:11:39.407 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:CECbiH3Eo/CrJ82vkJEc3KSwqSw= 2023-05-24T21:11:39.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:CECbiH3Eo/CrJ82vkJEc3KSwqSw=', 'Content-Length': '0'} 2023-05-24T21:11:39.410 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', '3wYQ13EDow0kiwE6ITIMyTWEPt0A0Hx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b7644e9857398ae2-00646e7d8b-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.412 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:39.412 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:39.413 DEBUG:boto:path=/qskrps-13/obj?versionId=3wYQ13EDow0kiwE6ITIMyTWEPt0A0Hx 2023-05-24T21:11:39.413 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=3wYQ13EDow0kiwE6ITIMyTWEPt0A0Hx 2023-05-24T21:11:39.413 DEBUG:boto:Method: HEAD 2023-05-24T21:11:39.414 DEBUG:boto:Path: /qskrps-13/obj?versionId=3wYQ13EDow0kiwE6ITIMyTWEPt0A0Hx 2023-05-24T21:11:39.414 DEBUG:boto:Data: 2023-05-24T21:11:39.414 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.415 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:39.415 DEBUG:boto:Port: 8001 2023-05-24T21:11:39.415 DEBUG:boto:Params: {} 2023-05-24T21:11:39.416 DEBUG:boto:Token: None 2023-05-24T21:11:39.416 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:39 GMT /qskrps-13/obj?versionId=3wYQ13EDow0kiwE6ITIMyTWEPt0A0Hx 2023-05-24T21:11:39.416 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:3qzP89Rrvcf/Wlt/UAP2liDVyQg= 2023-05-24T21:11:39.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:3qzP89Rrvcf/Wlt/UAP2liDVyQg=', 'Content-Length': '0'} 2023-05-24T21:11:39.419 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', '3wYQ13EDow0kiwE6ITIMyTWEPt0A0Hx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000343a0356487af35f-00646e7d8b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.419 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:39.420 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:39.420 DEBUG:boto:path=/qskrps-13/obj?versionId=3wYQ13EDow0kiwE6ITIMyTWEPt0A0Hx 2023-05-24T21:11:39.420 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=3wYQ13EDow0kiwE6ITIMyTWEPt0A0Hx 2023-05-24T21:11:39.421 DEBUG:boto:Method: HEAD 2023-05-24T21:11:39.422 DEBUG:boto:Path: /qskrps-13/obj?versionId=3wYQ13EDow0kiwE6ITIMyTWEPt0A0Hx 2023-05-24T21:11:39.423 DEBUG:boto:Data: 2023-05-24T21:11:39.424 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.425 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:39.426 DEBUG:boto:Port: 8000 2023-05-24T21:11:39.426 DEBUG:boto:Params: {} 2023-05-24T21:11:39.427 DEBUG:boto:Token: None 2023-05-24T21:11:39.427 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:39 GMT /qskrps-13/obj?versionId=3wYQ13EDow0kiwE6ITIMyTWEPt0A0Hx 2023-05-24T21:11:39.428 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:3qzP89Rrvcf/Wlt/UAP2liDVyQg= 2023-05-24T21:11:39.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:3qzP89Rrvcf/Wlt/UAP2liDVyQg=', 'Content-Length': '0'} 2023-05-24T21:11:39.430 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', '3wYQ13EDow0kiwE6ITIMyTWEPt0A0Hx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000377bfa007ac541da-00646e7d8b-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:39.431 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:39.431 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:39.432 DEBUG:boto:path=/qskrps-13/obj?versionId=3wYQ13EDow0kiwE6ITIMyTWEPt0A0Hx 2023-05-24T21:11:39.432 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=3wYQ13EDow0kiwE6ITIMyTWEPt0A0Hx 2023-05-24T21:11:39.433 DEBUG:boto:Method: GET 2023-05-24T21:11:39.433 DEBUG:boto:Path: /qskrps-13/obj?versionId=3wYQ13EDow0kiwE6ITIMyTWEPt0A0Hx 2023-05-24T21:11:39.433 DEBUG:boto:Data: 2023-05-24T21:11:39.434 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.434 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:39.434 DEBUG:boto:Port: 8001 2023-05-24T21:11:39.435 DEBUG:boto:Params: {} 2023-05-24T21:11:39.435 DEBUG:boto:Token: None 2023-05-24T21:11:39.436 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:39 GMT /qskrps-13/obj?versionId=3wYQ13EDow0kiwE6ITIMyTWEPt0A0Hx 2023-05-24T21:11:39.437 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:CECbiH3Eo/CrJ82vkJEc3KSwqSw= 2023-05-24T21:11:39.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:CECbiH3Eo/CrJ82vkJEc3KSwqSw=', 'Content-Length': '0'} 2023-05-24T21:11:39.441 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', '3wYQ13EDow0kiwE6ITIMyTWEPt0A0Hx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000034ebc0646f99c421-00646e7d8b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.488 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:39.489 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:39.490 DEBUG:boto:path=/qskrps-13/obj?versionId=3wYQ13EDow0kiwE6ITIMyTWEPt0A0Hx 2023-05-24T21:11:39.491 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=3wYQ13EDow0kiwE6ITIMyTWEPt0A0Hx 2023-05-24T21:11:39.492 DEBUG:boto:Method: GET 2023-05-24T21:11:39.493 DEBUG:boto:Path: /qskrps-13/obj?versionId=3wYQ13EDow0kiwE6ITIMyTWEPt0A0Hx 2023-05-24T21:11:39.493 DEBUG:boto:Data: 2023-05-24T21:11:39.494 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.494 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:39.494 DEBUG:boto:Port: 8000 2023-05-24T21:11:39.495 DEBUG:boto:Params: {} 2023-05-24T21:11:39.495 DEBUG:boto:Token: None 2023-05-24T21:11:39.496 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:39 GMT /qskrps-13/obj?versionId=3wYQ13EDow0kiwE6ITIMyTWEPt0A0Hx 2023-05-24T21:11:39.496 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:CECbiH3Eo/CrJ82vkJEc3KSwqSw= 2023-05-24T21:11:39.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:CECbiH3Eo/CrJ82vkJEc3KSwqSw=', 'Content-Length': '0'} 2023-05-24T21:11:39.499 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', '3wYQ13EDow0kiwE6ITIMyTWEPt0A0Hx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000026f41153a76b67e3-00646e7d8b-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.499 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:39.500 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:39.500 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:39.501 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:39.502 DEBUG:boto:Method: GET 2023-05-24T21:11:39.502 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:39.502 DEBUG:boto:Data: 2023-05-24T21:11:39.503 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.503 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:39.504 DEBUG:boto:Port: 8001 2023-05-24T21:11:39.504 DEBUG:boto:Params: {} 2023-05-24T21:11:39.504 DEBUG:boto:Token: None 2023-05-24T21:11:39.505 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:39 GMT /qskrps-13/ 2023-05-24T21:11:39.505 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:KLzthF2FvsaZLlFQbgC1hHucCR8= 2023-05-24T21:11:39.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:KLzthF2FvsaZLlFQbgC1hHucCR8=', 'Content-Length': '0'} 2023-05-24T21:11:39.508 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000472f48cec5ee06cd-00646e7d8b-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.509 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:39.510 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:39.511 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:39.511 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:39.512 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:39.512 DEBUG:boto:Method: GET 2023-05-24T21:11:39.512 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:39.513 DEBUG:boto:Data: 2023-05-24T21:11:39.513 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.514 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:39.514 DEBUG:boto:Port: 8000 2023-05-24T21:11:39.515 DEBUG:boto:Params: {} 2023-05-24T21:11:39.515 DEBUG:boto:Token: None 2023-05-24T21:11:39.515 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:39 GMT /qskrps-13/ 2023-05-24T21:11:39.516 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:KLzthF2FvsaZLlFQbgC1hHucCR8= 2023-05-24T21:11:39.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:KLzthF2FvsaZLlFQbgC1hHucCR8=', 'Content-Length': '0'} 2023-05-24T21:11:39.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9dcbe2b8f22fd17-00646e7d8b-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.520 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:39.521 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:39.521 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:39.522 DEBUG:boto:Method: HEAD 2023-05-24T21:11:39.522 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:11:39.522 DEBUG:boto:Data: 2023-05-24T21:11:39.523 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.523 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:39.523 DEBUG:boto:Port: 8001 2023-05-24T21:11:39.524 DEBUG:boto:Params: {} 2023-05-24T21:11:39.525 DEBUG:boto:Token: None 2023-05-24T21:11:39.525 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:39 GMT /qskrps-13/obj 2023-05-24T21:11:39.525 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:W70bsBNr+FGf5zONb0uhNNkcUOw= 2023-05-24T21:11:39.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:W70bsBNr+FGf5zONb0uhNNkcUOw=', 'Content-Length': '0'} 2023-05-24T21:11:39.529 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bbcc363c48cb70cf-00646e7d8b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.530 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:39.530 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:39.530 DEBUG:boto:Method: HEAD 2023-05-24T21:11:39.531 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:11:39.531 DEBUG:boto:Data: 2023-05-24T21:11:39.531 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.532 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:39.532 DEBUG:boto:Port: 8000 2023-05-24T21:11:39.532 DEBUG:boto:Params: {} 2023-05-24T21:11:39.533 DEBUG:boto:Token: None 2023-05-24T21:11:39.533 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:39 GMT /qskrps-13/obj 2023-05-24T21:11:39.534 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:W70bsBNr+FGf5zONb0uhNNkcUOw= 2023-05-24T21:11:39.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:W70bsBNr+FGf5zONb0uhNNkcUOw=', 'Content-Length': '0'} 2023-05-24T21:11:39.537 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fce76c7507690e5e-00646e7d8b-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:39.538 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:39.538 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:39.539 DEBUG:boto:path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:39.539 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:39.540 DEBUG:boto:Method: GET 2023-05-24T21:11:39.540 DEBUG:boto:Path: /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:39.540 DEBUG:boto:Data: 2023-05-24T21:11:39.541 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.541 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:39.542 DEBUG:boto:Port: 8001 2023-05-24T21:11:39.542 DEBUG:boto:Params: {} 2023-05-24T21:11:39.542 DEBUG:boto:Token: None 2023-05-24T21:11:39.543 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:39 GMT /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:39.543 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:wO031kG+mToKWqIEeiJQuipXtgM= 2023-05-24T21:11:39.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wO031kG+mToKWqIEeiJQuipXtgM=', 'Content-Length': '0'} 2023-05-24T21:11:39.546 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004234ddfbb0e127b4-00646e7d8b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.588 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:39.589 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:39.589 DEBUG:boto:path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:39.590 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:39.590 DEBUG:boto:Method: GET 2023-05-24T21:11:39.591 DEBUG:boto:Path: /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:39.591 DEBUG:boto:Data: 2023-05-24T21:11:39.591 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.592 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:39.592 DEBUG:boto:Port: 8000 2023-05-24T21:11:39.593 DEBUG:boto:Params: {} 2023-05-24T21:11:39.593 DEBUG:boto:Token: None 2023-05-24T21:11:39.593 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:39 GMT /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:39.594 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:wO031kG+mToKWqIEeiJQuipXtgM= 2023-05-24T21:11:39.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wO031kG+mToKWqIEeiJQuipXtgM=', 'Content-Length': '0'} 2023-05-24T21:11:39.596 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005efae4c55798999c-00646e7d8b-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.597 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:39.598 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:39.598 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:39.598 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:39.599 DEBUG:boto:Method: GET 2023-05-24T21:11:39.599 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:39.600 DEBUG:boto:Data: 2023-05-24T21:11:39.600 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.600 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:39.601 DEBUG:boto:Port: 8001 2023-05-24T21:11:39.601 DEBUG:boto:Params: {} 2023-05-24T21:11:39.602 DEBUG:boto:Token: None 2023-05-24T21:11:39.602 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:39 GMT /qskrps-13/ 2023-05-24T21:11:39.603 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:KLzthF2FvsaZLlFQbgC1hHucCR8= 2023-05-24T21:11:39.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:KLzthF2FvsaZLlFQbgC1hHucCR8=', 'Content-Length': '0'} 2023-05-24T21:11:39.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de68ae8f5b99589a-00646e7d8b-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.606 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:39.607 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:39.608 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:39.608 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:39.608 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:39.609 DEBUG:boto:Method: GET 2023-05-24T21:11:39.609 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:39.610 DEBUG:boto:Data: 2023-05-24T21:11:39.610 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.611 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:39.611 DEBUG:boto:Port: 8000 2023-05-24T21:11:39.611 DEBUG:boto:Params: {} 2023-05-24T21:11:39.612 DEBUG:boto:Token: None 2023-05-24T21:11:39.612 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:39 GMT /qskrps-13/ 2023-05-24T21:11:39.613 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:KLzthF2FvsaZLlFQbgC1hHucCR8= 2023-05-24T21:11:39.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:KLzthF2FvsaZLlFQbgC1hHucCR8=', 'Content-Length': '0'} 2023-05-24T21:11:39.616 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003556f1066878b963-00646e7d8b-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.656 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:39.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:39.657 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:39.658 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:39.658 DEBUG:boto:path=/qskrps-13/obj?versionId=vVL.Oc-NasmiTK3cSFNuRAIlXlfwbAC 2023-05-24T21:11:39.659 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=vVL.Oc-NasmiTK3cSFNuRAIlXlfwbAC 2023-05-24T21:11:39.659 DEBUG:boto:Method: GET 2023-05-24T21:11:39.659 DEBUG:boto:Path: /qskrps-13/obj?versionId=vVL.Oc-NasmiTK3cSFNuRAIlXlfwbAC 2023-05-24T21:11:39.660 DEBUG:boto:Data: 2023-05-24T21:11:39.660 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.661 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:39.661 DEBUG:boto:Port: 8001 2023-05-24T21:11:39.661 DEBUG:boto:Params: {} 2023-05-24T21:11:39.662 DEBUG:boto:Token: None 2023-05-24T21:11:39.662 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:39 GMT /qskrps-13/obj?versionId=vVL.Oc-NasmiTK3cSFNuRAIlXlfwbAC 2023-05-24T21:11:39.663 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:wlQ5YEAb4RvYy6hlc/8b/+sILOA= 2023-05-24T21:11:39.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wlQ5YEAb4RvYy6hlc/8b/+sILOA=', 'Content-Length': '0'} 2023-05-24T21:11:39.666 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'vVL.Oc-NasmiTK3cSFNuRAIlXlfwbAC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000057df2057140d3ee0-00646e7d8b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.667 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:39.668 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:39.669 DEBUG:boto:path=/qskrps-13/obj?versionId=vVL.Oc-NasmiTK3cSFNuRAIlXlfwbAC 2023-05-24T21:11:39.670 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=vVL.Oc-NasmiTK3cSFNuRAIlXlfwbAC 2023-05-24T21:11:39.670 DEBUG:boto:Method: GET 2023-05-24T21:11:39.671 DEBUG:boto:Path: /qskrps-13/obj?versionId=vVL.Oc-NasmiTK3cSFNuRAIlXlfwbAC 2023-05-24T21:11:39.672 DEBUG:boto:Data: 2023-05-24T21:11:39.673 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.674 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:39.674 DEBUG:boto:Port: 8000 2023-05-24T21:11:39.675 DEBUG:boto:Params: {} 2023-05-24T21:11:39.676 DEBUG:boto:Token: None 2023-05-24T21:11:39.677 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:39 GMT /qskrps-13/obj?versionId=vVL.Oc-NasmiTK3cSFNuRAIlXlfwbAC 2023-05-24T21:11:39.678 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:wlQ5YEAb4RvYy6hlc/8b/+sILOA= 2023-05-24T21:11:39.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wlQ5YEAb4RvYy6hlc/8b/+sILOA=', 'Content-Length': '0'} 2023-05-24T21:11:39.681 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'vVL.Oc-NasmiTK3cSFNuRAIlXlfwbAC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ca511bf47cc39230-00646e7d8b-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.682 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:39.683 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:39.684 DEBUG:boto:path=/qskrps-13/obj?versionId=vVL.Oc-NasmiTK3cSFNuRAIlXlfwbAC 2023-05-24T21:11:39.685 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=vVL.Oc-NasmiTK3cSFNuRAIlXlfwbAC 2023-05-24T21:11:39.685 DEBUG:boto:Method: HEAD 2023-05-24T21:11:39.686 DEBUG:boto:Path: /qskrps-13/obj?versionId=vVL.Oc-NasmiTK3cSFNuRAIlXlfwbAC 2023-05-24T21:11:39.687 DEBUG:boto:Data: 2023-05-24T21:11:39.688 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.688 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:39.689 DEBUG:boto:Port: 8001 2023-05-24T21:11:39.690 DEBUG:boto:Params: {} 2023-05-24T21:11:39.690 DEBUG:boto:Token: None 2023-05-24T21:11:39.691 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:39 GMT /qskrps-13/obj?versionId=vVL.Oc-NasmiTK3cSFNuRAIlXlfwbAC 2023-05-24T21:11:39.692 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:An+LaT4+kLRsG02dzuLH+twtRpc= 2023-05-24T21:11:39.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:An+LaT4+kLRsG02dzuLH+twtRpc=', 'Content-Length': '0'} 2023-05-24T21:11:39.695 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'vVL.Oc-NasmiTK3cSFNuRAIlXlfwbAC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aaec8ee3758274c6-00646e7d8b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.696 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:39.697 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:39.698 DEBUG:boto:path=/qskrps-13/obj?versionId=vVL.Oc-NasmiTK3cSFNuRAIlXlfwbAC 2023-05-24T21:11:39.698 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=vVL.Oc-NasmiTK3cSFNuRAIlXlfwbAC 2023-05-24T21:11:39.699 DEBUG:boto:Method: HEAD 2023-05-24T21:11:39.700 DEBUG:boto:Path: /qskrps-13/obj?versionId=vVL.Oc-NasmiTK3cSFNuRAIlXlfwbAC 2023-05-24T21:11:39.700 DEBUG:boto:Data: 2023-05-24T21:11:39.701 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.702 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:39.702 DEBUG:boto:Port: 8000 2023-05-24T21:11:39.703 DEBUG:boto:Params: {} 2023-05-24T21:11:39.704 DEBUG:boto:Token: None 2023-05-24T21:11:39.704 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:39 GMT /qskrps-13/obj?versionId=vVL.Oc-NasmiTK3cSFNuRAIlXlfwbAC 2023-05-24T21:11:39.705 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:An+LaT4+kLRsG02dzuLH+twtRpc= 2023-05-24T21:11:39.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:An+LaT4+kLRsG02dzuLH+twtRpc=', 'Content-Length': '0'} 2023-05-24T21:11:39.708 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'vVL.Oc-NasmiTK3cSFNuRAIlXlfwbAC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ecf38067991bd99a-00646e7d8b-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:39.710 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:39.711 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:39.712 DEBUG:boto:path=/qskrps-13/obj?versionId=vVL.Oc-NasmiTK3cSFNuRAIlXlfwbAC 2023-05-24T21:11:39.712 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=vVL.Oc-NasmiTK3cSFNuRAIlXlfwbAC 2023-05-24T21:11:39.713 DEBUG:boto:Method: GET 2023-05-24T21:11:39.714 DEBUG:boto:Path: /qskrps-13/obj?versionId=vVL.Oc-NasmiTK3cSFNuRAIlXlfwbAC 2023-05-24T21:11:39.714 DEBUG:boto:Data: 2023-05-24T21:11:39.715 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.716 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:39.717 DEBUG:boto:Port: 8001 2023-05-24T21:11:39.717 DEBUG:boto:Params: {} 2023-05-24T21:11:39.718 DEBUG:boto:Token: None 2023-05-24T21:11:39.719 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:39 GMT /qskrps-13/obj?versionId=vVL.Oc-NasmiTK3cSFNuRAIlXlfwbAC 2023-05-24T21:11:39.719 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:wlQ5YEAb4RvYy6hlc/8b/+sILOA= 2023-05-24T21:11:39.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wlQ5YEAb4RvYy6hlc/8b/+sILOA=', 'Content-Length': '0'} 2023-05-24T21:11:39.723 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'vVL.Oc-NasmiTK3cSFNuRAIlXlfwbAC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002c2d8f02955a4ed5-00646e7d8b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.724 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:39.724 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:39.725 DEBUG:boto:path=/qskrps-13/obj?versionId=vVL.Oc-NasmiTK3cSFNuRAIlXlfwbAC 2023-05-24T21:11:39.725 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=vVL.Oc-NasmiTK3cSFNuRAIlXlfwbAC 2023-05-24T21:11:39.725 DEBUG:boto:Method: GET 2023-05-24T21:11:39.726 DEBUG:boto:Path: /qskrps-13/obj?versionId=vVL.Oc-NasmiTK3cSFNuRAIlXlfwbAC 2023-05-24T21:11:39.726 DEBUG:boto:Data: 2023-05-24T21:11:39.727 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.727 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:39.727 DEBUG:boto:Port: 8000 2023-05-24T21:11:39.728 DEBUG:boto:Params: {} 2023-05-24T21:11:39.728 DEBUG:boto:Token: None 2023-05-24T21:11:39.729 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:39 GMT /qskrps-13/obj?versionId=vVL.Oc-NasmiTK3cSFNuRAIlXlfwbAC 2023-05-24T21:11:39.729 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:wlQ5YEAb4RvYy6hlc/8b/+sILOA= 2023-05-24T21:11:39.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wlQ5YEAb4RvYy6hlc/8b/+sILOA=', 'Content-Length': '0'} 2023-05-24T21:11:39.731 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'vVL.Oc-NasmiTK3cSFNuRAIlXlfwbAC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000702d437b169f2ddc-00646e7d8b-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.772 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:39.773 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:39.773 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:39.773 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:39.774 DEBUG:boto:Method: GET 2023-05-24T21:11:39.774 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:39.775 DEBUG:boto:Data: 2023-05-24T21:11:39.775 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.775 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:39.776 DEBUG:boto:Port: 8001 2023-05-24T21:11:39.776 DEBUG:boto:Params: {} 2023-05-24T21:11:39.776 DEBUG:boto:Token: None 2023-05-24T21:11:39.777 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:39 GMT /qskrps-13/ 2023-05-24T21:11:39.777 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:KLzthF2FvsaZLlFQbgC1hHucCR8= 2023-05-24T21:11:39.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:KLzthF2FvsaZLlFQbgC1hHucCR8=', 'Content-Length': '0'} 2023-05-24T21:11:39.780 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012bfed0321c360bf-00646e7d8b-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.781 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:39.781 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:39.782 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:39.782 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:39.783 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:39.783 DEBUG:boto:Method: GET 2023-05-24T21:11:39.783 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:39.784 DEBUG:boto:Data: 2023-05-24T21:11:39.784 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.784 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:39.785 DEBUG:boto:Port: 8000 2023-05-24T21:11:39.785 DEBUG:boto:Params: {} 2023-05-24T21:11:39.786 DEBUG:boto:Token: None 2023-05-24T21:11:39.786 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:39 GMT /qskrps-13/ 2023-05-24T21:11:39.786 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:KLzthF2FvsaZLlFQbgC1hHucCR8= 2023-05-24T21:11:39.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:KLzthF2FvsaZLlFQbgC1hHucCR8=', 'Content-Length': '0'} 2023-05-24T21:11:39.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014020300d01eb776-00646e7d8b-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.790 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:39.791 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:39.791 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:39.791 DEBUG:boto:Method: HEAD 2023-05-24T21:11:39.792 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:11:39.792 DEBUG:boto:Data: 2023-05-24T21:11:39.792 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.793 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:39.793 DEBUG:boto:Port: 8001 2023-05-24T21:11:39.793 DEBUG:boto:Params: {} 2023-05-24T21:11:39.794 DEBUG:boto:Token: None 2023-05-24T21:11:39.794 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:39 GMT /qskrps-13/obj 2023-05-24T21:11:39.794 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:W70bsBNr+FGf5zONb0uhNNkcUOw= 2023-05-24T21:11:39.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:W70bsBNr+FGf5zONb0uhNNkcUOw=', 'Content-Length': '0'} 2023-05-24T21:11:39.798 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f80fa378b52ffc8f-00646e7d8b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.798 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:39.799 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:39.799 DEBUG:boto:Method: HEAD 2023-05-24T21:11:39.799 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:11:39.800 DEBUG:boto:Data: 2023-05-24T21:11:39.800 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.800 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:39.801 DEBUG:boto:Port: 8000 2023-05-24T21:11:39.801 DEBUG:boto:Params: {} 2023-05-24T21:11:39.801 DEBUG:boto:Token: None 2023-05-24T21:11:39.802 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:39 GMT /qskrps-13/obj 2023-05-24T21:11:39.802 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:W70bsBNr+FGf5zONb0uhNNkcUOw= 2023-05-24T21:11:39.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:W70bsBNr+FGf5zONb0uhNNkcUOw=', 'Content-Length': '0'} 2023-05-24T21:11:39.805 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008e1ec0245138955f-00646e7d8b-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:39.806 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:39.807 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:39.807 DEBUG:boto:path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:39.807 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:39.808 DEBUG:boto:Method: GET 2023-05-24T21:11:39.808 DEBUG:boto:Path: /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:39.808 DEBUG:boto:Data: 2023-05-24T21:11:39.809 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.809 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:39.810 DEBUG:boto:Port: 8001 2023-05-24T21:11:39.810 DEBUG:boto:Params: {} 2023-05-24T21:11:39.810 DEBUG:boto:Token: None 2023-05-24T21:11:39.811 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:39 GMT /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:39.811 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:wO031kG+mToKWqIEeiJQuipXtgM= 2023-05-24T21:11:39.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wO031kG+mToKWqIEeiJQuipXtgM=', 'Content-Length': '0'} 2023-05-24T21:11:39.813 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008446738b497fda66-00646e7d8b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.860 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:39.861 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:39.861 DEBUG:boto:path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:39.862 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:39.862 DEBUG:boto:Method: GET 2023-05-24T21:11:39.863 DEBUG:boto:Path: /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:39.863 DEBUG:boto:Data: 2023-05-24T21:11:39.864 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.864 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:39.864 DEBUG:boto:Port: 8000 2023-05-24T21:11:39.865 DEBUG:boto:Params: {} 2023-05-24T21:11:39.865 DEBUG:boto:Token: None 2023-05-24T21:11:39.866 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:39 GMT /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:39.866 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:wO031kG+mToKWqIEeiJQuipXtgM= 2023-05-24T21:11:39.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wO031kG+mToKWqIEeiJQuipXtgM=', 'Content-Length': '0'} 2023-05-24T21:11:39.868 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005644a0c58e1ce8ac-00646e7d8b-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.869 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:39.870 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:39.870 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:39.870 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:39.871 DEBUG:boto:Method: GET 2023-05-24T21:11:39.871 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:39.871 DEBUG:boto:Data: 2023-05-24T21:11:39.872 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.872 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:39.873 DEBUG:boto:Port: 8001 2023-05-24T21:11:39.873 DEBUG:boto:Params: {} 2023-05-24T21:11:39.873 DEBUG:boto:Token: None 2023-05-24T21:11:39.874 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:39 GMT /qskrps-13/ 2023-05-24T21:11:39.874 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:KLzthF2FvsaZLlFQbgC1hHucCR8= 2023-05-24T21:11:39.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:KLzthF2FvsaZLlFQbgC1hHucCR8=', 'Content-Length': '0'} 2023-05-24T21:11:39.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c295c0dce112b2d-00646e7d8b-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.878 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:39.879 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:39.879 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:39.879 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:39.880 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:39.880 DEBUG:boto:Method: GET 2023-05-24T21:11:39.880 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:39.881 DEBUG:boto:Data: 2023-05-24T21:11:39.881 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.882 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:39.882 DEBUG:boto:Port: 8000 2023-05-24T21:11:39.882 DEBUG:boto:Params: {} 2023-05-24T21:11:39.883 DEBUG:boto:Token: None 2023-05-24T21:11:39.883 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:39 GMT /qskrps-13/ 2023-05-24T21:11:39.884 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:KLzthF2FvsaZLlFQbgC1hHucCR8= 2023-05-24T21:11:39.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:KLzthF2FvsaZLlFQbgC1hHucCR8=', 'Content-Length': '0'} 2023-05-24T21:11:39.887 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095af398def1cb220-00646e7d8b-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.928 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:39.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:39.929 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:39.930 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:39.930 DEBUG:boto:path=/qskrps-13/obj?versionId=BdBswNggliM4dGAQ3Dgxcgleuyd0Qxo 2023-05-24T21:11:39.931 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=BdBswNggliM4dGAQ3Dgxcgleuyd0Qxo 2023-05-24T21:11:39.931 DEBUG:boto:Method: GET 2023-05-24T21:11:39.931 DEBUG:boto:Path: /qskrps-13/obj?versionId=BdBswNggliM4dGAQ3Dgxcgleuyd0Qxo 2023-05-24T21:11:39.932 DEBUG:boto:Data: 2023-05-24T21:11:39.932 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.933 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:39.933 DEBUG:boto:Port: 8001 2023-05-24T21:11:39.933 DEBUG:boto:Params: {} 2023-05-24T21:11:39.934 DEBUG:boto:Token: None 2023-05-24T21:11:39.934 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:39 GMT /qskrps-13/obj?versionId=BdBswNggliM4dGAQ3Dgxcgleuyd0Qxo 2023-05-24T21:11:39.935 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:KO5sPxxpoGD5dmHHaM/SLVNlCfc= 2023-05-24T21:11:39.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:KO5sPxxpoGD5dmHHaM/SLVNlCfc=', 'Content-Length': '0'} 2023-05-24T21:11:39.937 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'BdBswNggliM4dGAQ3Dgxcgleuyd0Qxo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b49720438934a470-00646e7d8b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.938 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:39.938 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:39.939 DEBUG:boto:path=/qskrps-13/obj?versionId=BdBswNggliM4dGAQ3Dgxcgleuyd0Qxo 2023-05-24T21:11:39.939 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=BdBswNggliM4dGAQ3Dgxcgleuyd0Qxo 2023-05-24T21:11:39.939 DEBUG:boto:Method: GET 2023-05-24T21:11:39.940 DEBUG:boto:Path: /qskrps-13/obj?versionId=BdBswNggliM4dGAQ3Dgxcgleuyd0Qxo 2023-05-24T21:11:39.940 DEBUG:boto:Data: 2023-05-24T21:11:39.940 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.941 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:39.941 DEBUG:boto:Port: 8000 2023-05-24T21:11:39.942 DEBUG:boto:Params: {} 2023-05-24T21:11:39.942 DEBUG:boto:Token: None 2023-05-24T21:11:39.942 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:39 GMT /qskrps-13/obj?versionId=BdBswNggliM4dGAQ3Dgxcgleuyd0Qxo 2023-05-24T21:11:39.943 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:KO5sPxxpoGD5dmHHaM/SLVNlCfc= 2023-05-24T21:11:39.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:KO5sPxxpoGD5dmHHaM/SLVNlCfc=', 'Content-Length': '0'} 2023-05-24T21:11:39.946 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'BdBswNggliM4dGAQ3Dgxcgleuyd0Qxo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000088f8be16612ece0b-00646e7d8b-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.946 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:39.947 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:39.947 DEBUG:boto:path=/qskrps-13/obj?versionId=BdBswNggliM4dGAQ3Dgxcgleuyd0Qxo 2023-05-24T21:11:39.947 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=BdBswNggliM4dGAQ3Dgxcgleuyd0Qxo 2023-05-24T21:11:39.948 DEBUG:boto:Method: HEAD 2023-05-24T21:11:39.948 DEBUG:boto:Path: /qskrps-13/obj?versionId=BdBswNggliM4dGAQ3Dgxcgleuyd0Qxo 2023-05-24T21:11:39.948 DEBUG:boto:Data: 2023-05-24T21:11:39.949 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.949 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:39.949 DEBUG:boto:Port: 8001 2023-05-24T21:11:39.950 DEBUG:boto:Params: {} 2023-05-24T21:11:39.950 DEBUG:boto:Token: None 2023-05-24T21:11:39.951 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:39 GMT /qskrps-13/obj?versionId=BdBswNggliM4dGAQ3Dgxcgleuyd0Qxo 2023-05-24T21:11:39.951 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:3CTfk0DG8MkL2yYfXYyEXrQSprs= 2023-05-24T21:11:39.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:3CTfk0DG8MkL2yYfXYyEXrQSprs=', 'Content-Length': '0'} 2023-05-24T21:11:39.953 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'BdBswNggliM4dGAQ3Dgxcgleuyd0Qxo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e9545f081f0851c8-00646e7d8b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.954 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:39.954 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:39.954 DEBUG:boto:path=/qskrps-13/obj?versionId=BdBswNggliM4dGAQ3Dgxcgleuyd0Qxo 2023-05-24T21:11:39.955 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=BdBswNggliM4dGAQ3Dgxcgleuyd0Qxo 2023-05-24T21:11:39.955 DEBUG:boto:Method: HEAD 2023-05-24T21:11:39.956 DEBUG:boto:Path: /qskrps-13/obj?versionId=BdBswNggliM4dGAQ3Dgxcgleuyd0Qxo 2023-05-24T21:11:39.956 DEBUG:boto:Data: 2023-05-24T21:11:39.956 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.957 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:39.957 DEBUG:boto:Port: 8000 2023-05-24T21:11:39.957 DEBUG:boto:Params: {} 2023-05-24T21:11:39.958 DEBUG:boto:Token: None 2023-05-24T21:11:39.958 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:39 GMT /qskrps-13/obj?versionId=BdBswNggliM4dGAQ3Dgxcgleuyd0Qxo 2023-05-24T21:11:39.959 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:3CTfk0DG8MkL2yYfXYyEXrQSprs= 2023-05-24T21:11:39.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:3CTfk0DG8MkL2yYfXYyEXrQSprs=', 'Content-Length': '0'} 2023-05-24T21:11:39.961 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'BdBswNggliM4dGAQ3Dgxcgleuyd0Qxo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000281353a679650e08-00646e7d8b-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:39.962 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:39.962 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:39.963 DEBUG:boto:path=/qskrps-13/obj?versionId=BdBswNggliM4dGAQ3Dgxcgleuyd0Qxo 2023-05-24T21:11:39.963 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=BdBswNggliM4dGAQ3Dgxcgleuyd0Qxo 2023-05-24T21:11:39.964 DEBUG:boto:Method: GET 2023-05-24T21:11:39.964 DEBUG:boto:Path: /qskrps-13/obj?versionId=BdBswNggliM4dGAQ3Dgxcgleuyd0Qxo 2023-05-24T21:11:39.964 DEBUG:boto:Data: 2023-05-24T21:11:39.965 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.965 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:39.965 DEBUG:boto:Port: 8001 2023-05-24T21:11:39.966 DEBUG:boto:Params: {} 2023-05-24T21:11:39.966 DEBUG:boto:Token: None 2023-05-24T21:11:39.967 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:39 GMT /qskrps-13/obj?versionId=BdBswNggliM4dGAQ3Dgxcgleuyd0Qxo 2023-05-24T21:11:39.967 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:KO5sPxxpoGD5dmHHaM/SLVNlCfc= 2023-05-24T21:11:39.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:KO5sPxxpoGD5dmHHaM/SLVNlCfc=', 'Content-Length': '0'} 2023-05-24T21:11:39.969 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'BdBswNggliM4dGAQ3Dgxcgleuyd0Qxo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a3ff5fef15ebbd7e-00646e7d8b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.970 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:39.971 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:39.971 DEBUG:boto:path=/qskrps-13/obj?versionId=BdBswNggliM4dGAQ3Dgxcgleuyd0Qxo 2023-05-24T21:11:39.971 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=BdBswNggliM4dGAQ3Dgxcgleuyd0Qxo 2023-05-24T21:11:39.972 DEBUG:boto:Method: GET 2023-05-24T21:11:39.972 DEBUG:boto:Path: /qskrps-13/obj?versionId=BdBswNggliM4dGAQ3Dgxcgleuyd0Qxo 2023-05-24T21:11:39.972 DEBUG:boto:Data: 2023-05-24T21:11:39.973 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.973 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:39.974 DEBUG:boto:Port: 8000 2023-05-24T21:11:39.974 DEBUG:boto:Params: {} 2023-05-24T21:11:39.974 DEBUG:boto:Token: None 2023-05-24T21:11:39.975 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:39 GMT /qskrps-13/obj?versionId=BdBswNggliM4dGAQ3Dgxcgleuyd0Qxo 2023-05-24T21:11:39.975 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:KO5sPxxpoGD5dmHHaM/SLVNlCfc= 2023-05-24T21:11:39.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:KO5sPxxpoGD5dmHHaM/SLVNlCfc=', 'Content-Length': '0'} 2023-05-24T21:11:39.977 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'BdBswNggliM4dGAQ3Dgxcgleuyd0Qxo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e6aff419b8c95aa1-00646e7d8b-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.978 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:39.979 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:39.979 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:39.979 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:39.980 DEBUG:boto:Method: GET 2023-05-24T21:11:39.980 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:39.980 DEBUG:boto:Data: 2023-05-24T21:11:39.981 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.981 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:39.982 DEBUG:boto:Port: 8001 2023-05-24T21:11:39.982 DEBUG:boto:Params: {} 2023-05-24T21:11:39.982 DEBUG:boto:Token: None 2023-05-24T21:11:39.983 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:39 GMT /qskrps-13/ 2023-05-24T21:11:39.983 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:KLzthF2FvsaZLlFQbgC1hHucCR8= 2023-05-24T21:11:39.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:KLzthF2FvsaZLlFQbgC1hHucCR8=', 'Content-Length': '0'} 2023-05-24T21:11:39.986 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009be6905f0c88c3ea-00646e7d8b-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.987 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:39.987 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:39.988 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:39.988 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:39.989 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:39.989 DEBUG:boto:Method: GET 2023-05-24T21:11:39.989 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:39.990 DEBUG:boto:Data: 2023-05-24T21:11:39.990 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.990 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:39.991 DEBUG:boto:Port: 8000 2023-05-24T21:11:39.991 DEBUG:boto:Params: {} 2023-05-24T21:11:39.991 DEBUG:boto:Token: None 2023-05-24T21:11:39.992 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:39 GMT /qskrps-13/ 2023-05-24T21:11:39.992 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:KLzthF2FvsaZLlFQbgC1hHucCR8= 2023-05-24T21:11:39.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:KLzthF2FvsaZLlFQbgC1hHucCR8=', 'Content-Length': '0'} 2023-05-24T21:11:39.995 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000670c6a4823fb42c1-00646e7d8b-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:39.996 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:39.996 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:39.997 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:39.997 DEBUG:boto:Method: HEAD 2023-05-24T21:11:39.997 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:11:39.998 DEBUG:boto:Data: 2023-05-24T21:11:39.998 DEBUG:boto:Headers: {} 2023-05-24T21:11:39.998 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:39.999 DEBUG:boto:Port: 8001 2023-05-24T21:11:39.999 DEBUG:boto:Params: {} 2023-05-24T21:11:40.000 DEBUG:boto:Token: None 2023-05-24T21:11:40.000 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:40 GMT /qskrps-13/obj 2023-05-24T21:11:40.000 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:TGmY1bSAjePfJl9pqxcAHyf4WLo= 2023-05-24T21:11:40.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:40 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:TGmY1bSAjePfJl9pqxcAHyf4WLo=', 'Content-Length': '0'} 2023-05-24T21:11:40.004 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000054a42ef4fb7e4f3c-00646e7d8b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:40.004 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:40.005 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:40.005 DEBUG:boto:Method: HEAD 2023-05-24T21:11:40.005 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:11:40.006 DEBUG:boto:Data: 2023-05-24T21:11:40.006 DEBUG:boto:Headers: {} 2023-05-24T21:11:40.006 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:40.007 DEBUG:boto:Port: 8000 2023-05-24T21:11:40.007 DEBUG:boto:Params: {} 2023-05-24T21:11:40.007 DEBUG:boto:Token: None 2023-05-24T21:11:40.008 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:40 GMT /qskrps-13/obj 2023-05-24T21:11:40.008 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:TGmY1bSAjePfJl9pqxcAHyf4WLo= 2023-05-24T21:11:40.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:40 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:TGmY1bSAjePfJl9pqxcAHyf4WLo=', 'Content-Length': '0'} 2023-05-24T21:11:40.012 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a783f58ff2d064a4-00646e7d8c-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:40.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:40.013 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:40.013 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:40.014 DEBUG:boto:path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:40.014 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:40.014 DEBUG:boto:Method: GET 2023-05-24T21:11:40.015 DEBUG:boto:Path: /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:40.015 DEBUG:boto:Data: 2023-05-24T21:11:40.015 DEBUG:boto:Headers: {} 2023-05-24T21:11:40.016 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:40.016 DEBUG:boto:Port: 8001 2023-05-24T21:11:40.016 DEBUG:boto:Params: {} 2023-05-24T21:11:40.017 DEBUG:boto:Token: None 2023-05-24T21:11:40.017 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:40 GMT /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:40.018 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:LeIIS4W0oX58PFFoDXa3s7Vgppo= 2023-05-24T21:11:40.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:40 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:LeIIS4W0oX58PFFoDXa3s7Vgppo=', 'Content-Length': '0'} 2023-05-24T21:11:40.020 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cd9e2f2ae1e006d4-00646e7d8c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:40.021 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:40.021 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:40.021 DEBUG:boto:path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:40.022 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:40.022 DEBUG:boto:Method: GET 2023-05-24T21:11:40.023 DEBUG:boto:Path: /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:40.023 DEBUG:boto:Data: 2023-05-24T21:11:40.023 DEBUG:boto:Headers: {} 2023-05-24T21:11:40.024 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:40.024 DEBUG:boto:Port: 8000 2023-05-24T21:11:40.024 DEBUG:boto:Params: {} 2023-05-24T21:11:40.025 DEBUG:boto:Token: None 2023-05-24T21:11:40.025 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:40 GMT /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:40.026 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:LeIIS4W0oX58PFFoDXa3s7Vgppo= 2023-05-24T21:11:40.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:40 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:LeIIS4W0oX58PFFoDXa3s7Vgppo=', 'Content-Length': '0'} 2023-05-24T21:11:40.028 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005c8251626d525375-00646e7d8c-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:40.029 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:40.029 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:40.029 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:40.030 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:40.030 DEBUG:boto:Method: GET 2023-05-24T21:11:40.030 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:40.031 DEBUG:boto:Data: 2023-05-24T21:11:40.031 DEBUG:boto:Headers: {} 2023-05-24T21:11:40.031 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:40.032 DEBUG:boto:Port: 8001 2023-05-24T21:11:40.032 DEBUG:boto:Params: {} 2023-05-24T21:11:40.033 DEBUG:boto:Token: None 2023-05-24T21:11:40.033 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:40 GMT /qskrps-13/ 2023-05-24T21:11:40.033 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:kuxf0fxcpvtR0VwdEsxHsOrG1CQ= 2023-05-24T21:11:40.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:40 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:kuxf0fxcpvtR0VwdEsxHsOrG1CQ=', 'Content-Length': '0'} 2023-05-24T21:11:40.036 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052d5788cae6619ac-00646e7d8c-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:40.037 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:40.038 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:40.038 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:40.038 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:40.039 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:40.039 DEBUG:boto:Method: GET 2023-05-24T21:11:40.039 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:40.040 DEBUG:boto:Data: 2023-05-24T21:11:40.040 DEBUG:boto:Headers: {} 2023-05-24T21:11:40.040 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:40.041 DEBUG:boto:Port: 8000 2023-05-24T21:11:40.041 DEBUG:boto:Params: {} 2023-05-24T21:11:40.042 DEBUG:boto:Token: None 2023-05-24T21:11:40.042 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:40 GMT /qskrps-13/ 2023-05-24T21:11:40.042 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:kuxf0fxcpvtR0VwdEsxHsOrG1CQ= 2023-05-24T21:11:40.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:40 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:kuxf0fxcpvtR0VwdEsxHsOrG1CQ=', 'Content-Length': '0'} 2023-05-24T21:11:40.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdf057286416c1d1-00646e7d8c-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:40.046 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:40.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:40.047 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:40.047 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:40.047 DEBUG:boto:path=/qskrps-13/obj?versionId=agpDexVocguP.SXhiu6FyO5RbOuqAnw 2023-05-24T21:11:40.048 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=agpDexVocguP.SXhiu6FyO5RbOuqAnw 2023-05-24T21:11:40.048 DEBUG:boto:Method: GET 2023-05-24T21:11:40.048 DEBUG:boto:Path: /qskrps-13/obj?versionId=agpDexVocguP.SXhiu6FyO5RbOuqAnw 2023-05-24T21:11:40.049 DEBUG:boto:Data: 2023-05-24T21:11:40.049 DEBUG:boto:Headers: {} 2023-05-24T21:11:40.049 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:40.050 DEBUG:boto:Port: 8001 2023-05-24T21:11:40.050 DEBUG:boto:Params: {} 2023-05-24T21:11:40.051 DEBUG:boto:Token: None 2023-05-24T21:11:40.051 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:40 GMT /qskrps-13/obj?versionId=agpDexVocguP.SXhiu6FyO5RbOuqAnw 2023-05-24T21:11:40.051 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Hkeg++KuceQfVj/RS4MEoPXw68A= 2023-05-24T21:11:40.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:40 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Hkeg++KuceQfVj/RS4MEoPXw68A=', 'Content-Length': '0'} 2023-05-24T21:11:40.054 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'agpDexVocguP.SXhiu6FyO5RbOuqAnw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d1f40e0693bcc939-00646e7d8c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:40.096 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:40.097 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:40.097 DEBUG:boto:path=/qskrps-13/obj?versionId=agpDexVocguP.SXhiu6FyO5RbOuqAnw 2023-05-24T21:11:40.098 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=agpDexVocguP.SXhiu6FyO5RbOuqAnw 2023-05-24T21:11:40.098 DEBUG:boto:Method: GET 2023-05-24T21:11:40.098 DEBUG:boto:Path: /qskrps-13/obj?versionId=agpDexVocguP.SXhiu6FyO5RbOuqAnw 2023-05-24T21:11:40.099 DEBUG:boto:Data: 2023-05-24T21:11:40.099 DEBUG:boto:Headers: {} 2023-05-24T21:11:40.099 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:40.100 DEBUG:boto:Port: 8000 2023-05-24T21:11:40.100 DEBUG:boto:Params: {} 2023-05-24T21:11:40.100 DEBUG:boto:Token: None 2023-05-24T21:11:40.101 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:40 GMT /qskrps-13/obj?versionId=agpDexVocguP.SXhiu6FyO5RbOuqAnw 2023-05-24T21:11:40.101 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Hkeg++KuceQfVj/RS4MEoPXw68A= 2023-05-24T21:11:40.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:40 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Hkeg++KuceQfVj/RS4MEoPXw68A=', 'Content-Length': '0'} 2023-05-24T21:11:40.104 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'agpDexVocguP.SXhiu6FyO5RbOuqAnw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009d2afdc2ee4a901f-00646e7d8c-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:40.105 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:40.105 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:40.105 DEBUG:boto:path=/qskrps-13/obj?versionId=agpDexVocguP.SXhiu6FyO5RbOuqAnw 2023-05-24T21:11:40.106 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=agpDexVocguP.SXhiu6FyO5RbOuqAnw 2023-05-24T21:11:40.106 DEBUG:boto:Method: HEAD 2023-05-24T21:11:40.106 DEBUG:boto:Path: /qskrps-13/obj?versionId=agpDexVocguP.SXhiu6FyO5RbOuqAnw 2023-05-24T21:11:40.107 DEBUG:boto:Data: 2023-05-24T21:11:40.107 DEBUG:boto:Headers: {} 2023-05-24T21:11:40.107 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:40.108 DEBUG:boto:Port: 8001 2023-05-24T21:11:40.108 DEBUG:boto:Params: {} 2023-05-24T21:11:40.108 DEBUG:boto:Token: None 2023-05-24T21:11:40.109 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:40 GMT /qskrps-13/obj?versionId=agpDexVocguP.SXhiu6FyO5RbOuqAnw 2023-05-24T21:11:40.109 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:vmjBVJIM7GaoiuKGar+QtiZbpXM= 2023-05-24T21:11:40.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:40 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:vmjBVJIM7GaoiuKGar+QtiZbpXM=', 'Content-Length': '0'} 2023-05-24T21:11:40.112 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'agpDexVocguP.SXhiu6FyO5RbOuqAnw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f01f21a1d7396744-00646e7d8c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:40.112 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:40.113 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:40.113 DEBUG:boto:path=/qskrps-13/obj?versionId=agpDexVocguP.SXhiu6FyO5RbOuqAnw 2023-05-24T21:11:40.113 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=agpDexVocguP.SXhiu6FyO5RbOuqAnw 2023-05-24T21:11:40.114 DEBUG:boto:Method: HEAD 2023-05-24T21:11:40.114 DEBUG:boto:Path: /qskrps-13/obj?versionId=agpDexVocguP.SXhiu6FyO5RbOuqAnw 2023-05-24T21:11:40.114 DEBUG:boto:Data: 2023-05-24T21:11:40.115 DEBUG:boto:Headers: {} 2023-05-24T21:11:40.115 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:40.115 DEBUG:boto:Port: 8000 2023-05-24T21:11:40.116 DEBUG:boto:Params: {} 2023-05-24T21:11:40.116 DEBUG:boto:Token: None 2023-05-24T21:11:40.116 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:40 GMT /qskrps-13/obj?versionId=agpDexVocguP.SXhiu6FyO5RbOuqAnw 2023-05-24T21:11:40.117 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:vmjBVJIM7GaoiuKGar+QtiZbpXM= 2023-05-24T21:11:40.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:40 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:vmjBVJIM7GaoiuKGar+QtiZbpXM=', 'Content-Length': '0'} 2023-05-24T21:11:40.119 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'agpDexVocguP.SXhiu6FyO5RbOuqAnw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000018276e9328fdd8c-00646e7d8c-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:40.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:40.120 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:40.120 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:40.121 DEBUG:boto:path=/qskrps-13/obj?versionId=agpDexVocguP.SXhiu6FyO5RbOuqAnw 2023-05-24T21:11:40.121 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=agpDexVocguP.SXhiu6FyO5RbOuqAnw 2023-05-24T21:11:40.121 DEBUG:boto:Method: GET 2023-05-24T21:11:40.122 DEBUG:boto:Path: /qskrps-13/obj?versionId=agpDexVocguP.SXhiu6FyO5RbOuqAnw 2023-05-24T21:11:40.122 DEBUG:boto:Data: 2023-05-24T21:11:40.122 DEBUG:boto:Headers: {} 2023-05-24T21:11:40.123 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:40.123 DEBUG:boto:Port: 8001 2023-05-24T21:11:40.123 DEBUG:boto:Params: {} 2023-05-24T21:11:40.124 DEBUG:boto:Token: None 2023-05-24T21:11:40.124 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:40 GMT /qskrps-13/obj?versionId=agpDexVocguP.SXhiu6FyO5RbOuqAnw 2023-05-24T21:11:40.125 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Hkeg++KuceQfVj/RS4MEoPXw68A= 2023-05-24T21:11:40.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:40 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Hkeg++KuceQfVj/RS4MEoPXw68A=', 'Content-Length': '0'} 2023-05-24T21:11:40.127 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'agpDexVocguP.SXhiu6FyO5RbOuqAnw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f8d0068cdf7fdd1c-00646e7d8c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:40.128 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:40.128 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:40.129 DEBUG:boto:path=/qskrps-13/obj?versionId=agpDexVocguP.SXhiu6FyO5RbOuqAnw 2023-05-24T21:11:40.129 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=agpDexVocguP.SXhiu6FyO5RbOuqAnw 2023-05-24T21:11:40.129 DEBUG:boto:Method: GET 2023-05-24T21:11:40.130 DEBUG:boto:Path: /qskrps-13/obj?versionId=agpDexVocguP.SXhiu6FyO5RbOuqAnw 2023-05-24T21:11:40.130 DEBUG:boto:Data: 2023-05-24T21:11:40.130 DEBUG:boto:Headers: {} 2023-05-24T21:11:40.131 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:40.131 DEBUG:boto:Port: 8000 2023-05-24T21:11:40.131 DEBUG:boto:Params: {} 2023-05-24T21:11:40.132 DEBUG:boto:Token: None 2023-05-24T21:11:40.132 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:40 GMT /qskrps-13/obj?versionId=agpDexVocguP.SXhiu6FyO5RbOuqAnw 2023-05-24T21:11:40.133 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Hkeg++KuceQfVj/RS4MEoPXw68A= 2023-05-24T21:11:40.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:40 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Hkeg++KuceQfVj/RS4MEoPXw68A=', 'Content-Length': '0'} 2023-05-24T21:11:40.135 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'agpDexVocguP.SXhiu6FyO5RbOuqAnw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e4a204317efe4776-00646e7d8c-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:40.176 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:40.177 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:40.177 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:40.178 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:40.178 DEBUG:boto:Method: GET 2023-05-24T21:11:40.178 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:40.179 DEBUG:boto:Data: 2023-05-24T21:11:40.179 DEBUG:boto:Headers: {} 2023-05-24T21:11:40.179 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:40.180 DEBUG:boto:Port: 8001 2023-05-24T21:11:40.180 DEBUG:boto:Params: {} 2023-05-24T21:11:40.181 DEBUG:boto:Token: None 2023-05-24T21:11:40.181 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:40 GMT /qskrps-13/ 2023-05-24T21:11:40.181 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:kuxf0fxcpvtR0VwdEsxHsOrG1CQ= 2023-05-24T21:11:40.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:40 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:kuxf0fxcpvtR0VwdEsxHsOrG1CQ=', 'Content-Length': '0'} 2023-05-24T21:11:40.184 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000467d64a38ac010a9-00646e7d8c-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:40.185 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:40.186 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:40.186 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:40.187 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:40.187 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:40.187 DEBUG:boto:Method: GET 2023-05-24T21:11:40.188 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:40.188 DEBUG:boto:Data: 2023-05-24T21:11:40.188 DEBUG:boto:Headers: {} 2023-05-24T21:11:40.189 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:40.189 DEBUG:boto:Port: 8000 2023-05-24T21:11:40.189 DEBUG:boto:Params: {} 2023-05-24T21:11:40.190 DEBUG:boto:Token: None 2023-05-24T21:11:40.190 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:40 GMT /qskrps-13/ 2023-05-24T21:11:40.191 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:kuxf0fxcpvtR0VwdEsxHsOrG1CQ= 2023-05-24T21:11:40.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:40 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:kuxf0fxcpvtR0VwdEsxHsOrG1CQ=', 'Content-Length': '0'} 2023-05-24T21:11:40.194 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2786c4136389a48-00646e7d8c-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:40.194 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:40.195 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:40.195 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:40.196 DEBUG:boto:Method: HEAD 2023-05-24T21:11:40.196 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:11:40.196 DEBUG:boto:Data: 2023-05-24T21:11:40.197 DEBUG:boto:Headers: {} 2023-05-24T21:11:40.197 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:40.197 DEBUG:boto:Port: 8001 2023-05-24T21:11:40.198 DEBUG:boto:Params: {} 2023-05-24T21:11:40.198 DEBUG:boto:Token: None 2023-05-24T21:11:40.198 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:40 GMT /qskrps-13/obj 2023-05-24T21:11:40.199 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:TGmY1bSAjePfJl9pqxcAHyf4WLo= 2023-05-24T21:11:40.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:40 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:TGmY1bSAjePfJl9pqxcAHyf4WLo=', 'Content-Length': '0'} 2023-05-24T21:11:40.202 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007bc2e0117eff80bf-00646e7d8c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:40.203 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:40.203 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:40.204 DEBUG:boto:Method: HEAD 2023-05-24T21:11:40.204 DEBUG:boto:Path: /qskrps-13/obj 2023-05-24T21:11:40.204 DEBUG:boto:Data: 2023-05-24T21:11:40.205 DEBUG:boto:Headers: {} 2023-05-24T21:11:40.205 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:40.205 DEBUG:boto:Port: 8000 2023-05-24T21:11:40.206 DEBUG:boto:Params: {} 2023-05-24T21:11:40.206 DEBUG:boto:Token: None 2023-05-24T21:11:40.206 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:40 GMT /qskrps-13/obj 2023-05-24T21:11:40.207 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:TGmY1bSAjePfJl9pqxcAHyf4WLo= 2023-05-24T21:11:40.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:40 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:TGmY1bSAjePfJl9pqxcAHyf4WLo=', 'Content-Length': '0'} 2023-05-24T21:11:40.210 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007aa2e5adcea03455-00646e7d8c-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:40.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:11:40.211 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:40.211 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:40.212 DEBUG:boto:path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:40.212 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:40.213 DEBUG:boto:Method: GET 2023-05-24T21:11:40.213 DEBUG:boto:Path: /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:40.213 DEBUG:boto:Data: 2023-05-24T21:11:40.214 DEBUG:boto:Headers: {} 2023-05-24T21:11:40.214 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:40.214 DEBUG:boto:Port: 8001 2023-05-24T21:11:40.215 DEBUG:boto:Params: {} 2023-05-24T21:11:40.215 DEBUG:boto:Token: None 2023-05-24T21:11:40.215 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:40 GMT /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:40.216 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:LeIIS4W0oX58PFFoDXa3s7Vgppo= 2023-05-24T21:11:40.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:40 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:LeIIS4W0oX58PFFoDXa3s7Vgppo=', 'Content-Length': '0'} 2023-05-24T21:11:40.218 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f83413a3651e287d-00646e7d8c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:40.260 DEBUG:boto:path=/qskrps-13/obj 2023-05-24T21:11:40.261 DEBUG:boto:auth_path=/qskrps-13/obj 2023-05-24T21:11:40.261 DEBUG:boto:path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:40.262 DEBUG:boto:auth_path=/qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:40.262 DEBUG:boto:Method: GET 2023-05-24T21:11:40.262 DEBUG:boto:Path: /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:40.263 DEBUG:boto:Data: 2023-05-24T21:11:40.263 DEBUG:boto:Headers: {} 2023-05-24T21:11:40.263 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:40.264 DEBUG:boto:Port: 8000 2023-05-24T21:11:40.264 DEBUG:boto:Params: {} 2023-05-24T21:11:40.265 DEBUG:boto:Token: None 2023-05-24T21:11:40.265 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:40 GMT /qskrps-13/obj?versionId=G.4hkCINEGMtwc87D.MHv0zLLwQSDS7 2023-05-24T21:11:40.266 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:LeIIS4W0oX58PFFoDXa3s7Vgppo= 2023-05-24T21:11:40.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:40 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:LeIIS4W0oX58PFFoDXa3s7Vgppo=', 'Content-Length': '0'} 2023-05-24T21:11:40.268 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:10:35 GMT'), ('x-amz-version-id', 'G.4hkCINEGMtwc87D.MHv0zLLwQSDS7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000007eaffd60ad8bfa1-00646e7d8c-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:40.269 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:40.269 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:40.269 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:40.270 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:40.270 DEBUG:boto:Method: GET 2023-05-24T21:11:40.270 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:40.271 DEBUG:boto:Data: 2023-05-24T21:11:40.271 DEBUG:boto:Headers: {} 2023-05-24T21:11:40.272 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:11:40.272 DEBUG:boto:Port: 8001 2023-05-24T21:11:40.272 DEBUG:boto:Params: {} 2023-05-24T21:11:40.273 DEBUG:boto:Token: None 2023-05-24T21:11:40.273 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:40 GMT /qskrps-13/ 2023-05-24T21:11:40.274 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:kuxf0fxcpvtR0VwdEsxHsOrG1CQ= 2023-05-24T21:11:40.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:40 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:kuxf0fxcpvtR0VwdEsxHsOrG1CQ=', 'Content-Length': '0'} 2023-05-24T21:11:40.276 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000358b610d4cf4b09f-00646e7d8c-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:40.277 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:40.277 DEBUG:boto:path=/qskrps-13/ 2023-05-24T21:11:40.278 DEBUG:boto:auth_path=/qskrps-13/ 2023-05-24T21:11:40.278 DEBUG:boto:path=/qskrps-13/?prefix=obj 2023-05-24T21:11:40.278 DEBUG:boto:auth_path=/qskrps-13/?prefix=obj 2023-05-24T21:11:40.279 DEBUG:boto:Method: GET 2023-05-24T21:11:40.279 DEBUG:boto:Path: /qskrps-13/?prefix=obj 2023-05-24T21:11:40.280 DEBUG:boto:Data: 2023-05-24T21:11:40.280 DEBUG:boto:Headers: {} 2023-05-24T21:11:40.280 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:40.281 DEBUG:boto:Port: 8000 2023-05-24T21:11:40.281 DEBUG:boto:Params: {} 2023-05-24T21:11:40.281 DEBUG:boto:Token: None 2023-05-24T21:11:40.282 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:11:40 GMT /qskrps-13/ 2023-05-24T21:11:40.282 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:kuxf0fxcpvtR0VwdEsxHsOrG1CQ= 2023-05-24T21:11:40.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:40 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:kuxf0fxcpvtR0VwdEsxHsOrG1CQ=', 'Content-Length': '0'} 2023-05-24T21:11:40.285 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf97bf81a8e0055b-00646e7d8c-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:40.286 DEBUG:boto:b'qskrps-13obj1000falseobj2023-05-24T21:10:35.767Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:11:40.286 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qskrps-13 zones={a2, a1} 2023-05-24T21:11:40.287 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-05-24T21:11:40.287 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:11:40.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:11:40.288 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:11:40.343 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:11:40.344 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:11:40.414 INFO:teuthology.orchestra.run.smithi071.stdout:[ 2023-05-24T21:11:40.415 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:11:40.415 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000002627", 2023-05-24T21:11:40.415 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:10:45.893198Z" 2023-05-24T21:11:40.415 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:11:40.416 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:11:40.416 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-05-24T21:11:40.416 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:03:11.196517Z" 2023-05-24T21:11:40.416 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:11:40.416 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:11:40.416 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-05-24T21:11:40.417 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:06:31.177575Z" 2023-05-24T21:11:40.417 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:11:40.417 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:11:40.417 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000002196", 2023-05-24T21:11:40.417 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:11:07.896432Z" 2023-05-24T21:11:40.417 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:11:40.417 INFO:teuthology.orchestra.run.smithi071.stdout:] 2023-05-24T21:11:40.418 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000002627', 1: '00000000000000000000:00000000000000000929', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000002196'} 2023-05-24T21:11:40.419 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-05-24T21:11:40.419 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:11:40.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:11:40.420 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:11:40.472 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:11:40.473 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:11:40.522 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:11:40.523 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:11:40.523 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:11:40.523 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:11:40.523 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:11:40.523 INFO:teuthology.orchestra.run.smithi071.stdout: "instance_id": 13190874283549403954 2023-05-24T21:11:40.524 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:11:40.524 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:11:40.524 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:11:40.524 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:11:40.524 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:11:40.524 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:11:40.525 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000002627", 2023-05-24T21:11:40.525 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:11:40.525 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:11:40.525 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:11:40.525 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:10:45.893198Z" 2023-05-24T21:11:40.525 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:11:40.525 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:11:40.526 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:11:40.526 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:11:40.526 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:11:40.526 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:11:40.526 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-05-24T21:11:40.526 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:11:40.526 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:11:40.527 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:11:40.527 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:03:11.196517Z" 2023-05-24T21:11:40.527 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:11:40.527 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:11:40.527 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:11:40.527 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:11:40.527 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:11:40.528 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:11:40.528 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-05-24T21:11:40.528 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:11:40.528 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:11:40.528 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:11:40.528 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:31.177575Z" 2023-05-24T21:11:40.529 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:11:40.529 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:11:40.529 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:11:40.529 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:11:40.529 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:11:40.529 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:11:40.529 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000002196", 2023-05-24T21:11:40.530 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:11:40.530 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:11:40.530 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:11:40.530 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:11:07.896432Z" 2023-05-24T21:11:40.530 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:11:40.530 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:11:40.531 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:11:40.531 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:11:40.531 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:11:40.531 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 0, 2023-05-24T21:11:40.531 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 0 2023-05-24T21:11:40.531 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:11:40.532 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:11:40Z" 2023-05-24T21:11:40.532 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:11:40.533 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13190874283549403954 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002627", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:10:45.893198Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000929", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:03:11.196517Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:31.177575Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002196", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:11:07.896432Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-05-24T21:11:40Z" } 2023-05-24T21:11:40.533 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002627', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:10:45.893198Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000929', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:03:11.196517Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:31.177575Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002196', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:11:07.896432Z'}}] 2023-05-24T21:11:40.534 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002627', 1: '00000000000000000000:00000000000000000929', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000002196'} 2023-05-24T21:11:40.534 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002627', 1: '00000000000000000000:00000000000000000929', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000002196'} 2023-05-24T21:11:40.534 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-05-24T21:11:40.535 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-05-24T21:11:40.535 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:11:40.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:11:40.536 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:11:40.583 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:11:40.584 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:11:40.660 INFO:teuthology.orchestra.run.smithi071.stdout:[ 2023-05-24T21:11:40.661 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:11:40.661 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000002345", 2023-05-24T21:11:40.661 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:10:23.920015Z" 2023-05-24T21:11:40.661 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:11:40.661 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:11:40.662 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-05-24T21:11:40.662 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:03:51.210005Z" 2023-05-24T21:11:40.662 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:11:40.662 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:11:40.662 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-05-24T21:11:40.662 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:03:11.225401Z" 2023-05-24T21:11:40.662 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:11:40.663 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:11:40.663 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000002337", 2023-05-24T21:11:40.663 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:11:29.925322Z" 2023-05-24T21:11:40.663 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:11:40.663 INFO:teuthology.orchestra.run.smithi071.stdout:] 2023-05-24T21:11:40.664 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000002345', 1: '00000000000000000000:00000000000000000929', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000002337'} 2023-05-24T21:11:40.665 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-05-24T21:11:40.665 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:11:40.665 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:11:40.666 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:11:40.716 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:11:40.717 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:11:40.783 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:11:40.783 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:11:40.783 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:11:40.784 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:11:40.784 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:11:40.784 INFO:teuthology.orchestra.run.smithi071.stdout: "instance_id": 6585121019900467362 2023-05-24T21:11:40.784 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:11:40.784 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:11:40.784 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:11:40.784 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:11:40.785 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:11:40.785 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:11:40.785 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000002345", 2023-05-24T21:11:40.785 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:11:40.785 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:11:40.785 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:11:40.786 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:10:23.920015Z" 2023-05-24T21:11:40.786 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:11:40.786 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:11:40.786 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:11:40.786 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:11:40.786 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:11:40.786 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:11:40.787 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-05-24T21:11:40.787 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:11:40.787 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:11:40.787 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:11:40.787 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:03:51.210005Z" 2023-05-24T21:11:40.787 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:11:40.787 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:11:40.788 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:11:40.788 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:11:40.788 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:11:40.788 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:11:40.788 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-05-24T21:11:40.788 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:11:40.788 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:11:40.789 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:11:40.789 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:03:11.225401Z" 2023-05-24T21:11:40.789 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:11:40.789 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:11:40.789 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:11:40.789 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:11:40.789 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:11:40.790 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:11:40.790 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000002337", 2023-05-24T21:11:40.790 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:11:40.790 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:11:40.790 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:11:40.790 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:11:29.925322Z" 2023-05-24T21:11:40.790 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:11:40.791 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:11:40.791 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:11:40.791 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:11:40.791 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:11:40.791 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 0, 2023-05-24T21:11:40.791 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 0 2023-05-24T21:11:40.791 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:11:40.792 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:11:40Z" 2023-05-24T21:11:40.792 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:11:40.793 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6585121019900467362 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002345", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:10:23.920015Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000929", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:03:51.210005Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:03:11.225401Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002337", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:11:29.925322Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-05-24T21:11:40Z" } 2023-05-24T21:11:40.793 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002345', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:10:23.920015Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000929', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:03:51.210005Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:03:11.225401Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002337', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:11:29.925322Z'}}] 2023-05-24T21:11:40.794 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002345', 1: '00000000000000000000:00000000000000000929', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000002337'} 2023-05-24T21:11:40.794 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002345', 1: '00000000000000000000:00000000000000000929', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000002337'} 2023-05-24T21:11:40.794 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-05-24T21:11:40.795 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2023-05-24T21:11:40.796 DEBUG:boto:path=/qskrps-14/ 2023-05-24T21:11:40.796 DEBUG:boto:auth_path=/qskrps-14/ 2023-05-24T21:11:40.797 DEBUG:boto:Method: PUT 2023-05-24T21:11:40.797 DEBUG:boto:Path: /qskrps-14/ 2023-05-24T21:11:40.797 DEBUG:boto:Data: 2023-05-24T21:11:40.798 DEBUG:boto:Headers: {} 2023-05-24T21:11:40.798 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:40.798 DEBUG:boto:Port: 8000 2023-05-24T21:11:40.799 DEBUG:boto:Params: {} 2023-05-24T21:11:40.799 DEBUG:boto:Token: None 2023-05-24T21:11:40.799 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:11:40 GMT /qskrps-14/ 2023-05-24T21:11:40.800 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:B3uJ+rZHydWmG5yhitJbcB3rJPs= 2023-05-24T21:11:40.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:40 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:B3uJ+rZHydWmG5yhitJbcB3rJPs=', 'Content-Length': '0'} 2023-05-24T21:11:40.821 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003a1cad87f9178037-00646e7d8c-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:40.821 DEBUG:rgw_multi.tests:created bucket=qskrps-14 2023-05-24T21:11:40.822 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:11:40.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:11:40.822 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:11:40.870 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:11:40.871 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:11:40.901 INFO:teuthology.orchestra.run.smithi071.stderr:No --period given, using current period=5c04936b-70ea-4415-b4eb-3aad3dc612e5 2023-05-24T21:11:40.919 INFO:teuthology.orchestra.run.smithi071.stdout:[ 2023-05-24T21:11:40.920 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:11:40.920 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962700.813147_143.1", 2023-05-24T21:11:40.920 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:11:40.813147Z" 2023-05-24T21:11:40.920 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:11:40.920 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:11:40.921 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:11:40.921 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:06:32.333844Z" 2023-05-24T21:11:40.921 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:11:40.921 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:11:40.921 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:11:40.921 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:11:40.921 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:11:40.922 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:11:40.922 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:11:40.922 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:11:40.922 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:11:40.922 INFO:teuthology.orchestra.run.smithi071.stdout:] 2023-05-24T21:11:40.923 DEBUG:rgw_multi.tests:master meta markers={0: '1_1684962700.813147_143.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:11:40.923 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-24T21:11:40.924 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:11:40.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:11:40.925 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:11:40.974 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:11:40.975 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:11:41.022 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:11:41.015+0000 7f0d0664e780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:11:41.023 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:11:41.016+0000 7f0d0664e780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:11:41.041 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:11:41.041 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:11:41.041 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:11:41.041 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:11:41.041 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:11:41.042 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:11:41.042 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:11:41.042 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:11:41.042 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:11:41.042 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:11:41.042 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:11:41.043 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:11:41.043 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:11:41.043 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962700.813147_143.1", 2023-05-24T21:11:41.043 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:11:41.043 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:11:41.043 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:11:41.043 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:11:40.813147Z", 2023-05-24T21:11:41.044 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:11:41.044 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:11:41.044 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:11:41.044 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:11:41.044 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:11:41.044 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:11:41.045 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:11:41.045 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:11:41.045 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:11:41.045 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:11:41.045 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:11:41.045 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:11:41.045 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:11:41.046 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:11:41.046 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:11:41.046 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:11:41.046 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:11:41.046 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:11:41.046 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:11:41.046 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:11:41.047 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:11:41.047 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:11:41.047 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:11:41.047 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:11:41.047 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:11:41.047 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:11:41.047 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:11:41.048 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:11:41.048 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:11:41.048 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:11:41.048 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:11:41.048 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:11:41.048 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:11:41.048 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:11:41.049 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:11:41.049 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:11:41.049 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:11:41.049 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:11:41.049 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:11:41.049 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:11:41.049 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:11:41.050 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:11:41.050 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:11:41.050 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:11:41.050 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:11:41.050 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:11:41Z" 2023-05-24T21:11:41.050 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:11:41.051 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962700.813147_143.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:11:40.813147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:11:41Z" } 2023-05-24T21:11:41.052 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962700.813147_143.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:11:40.813147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:11:41.052 DEBUG:rgw_multi.tests:log_status={0: '1_1684962700.813147_143.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:11:41.052 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962700.813147_143.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:11:41.053 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-24T21:11:41.053 DEBUG:boto:path=/qskrps-14/ 2023-05-24T21:11:41.053 DEBUG:boto:auth_path=/qskrps-14/ 2023-05-24T21:11:41.054 DEBUG:boto:Method: HEAD 2023-05-24T21:11:41.054 DEBUG:boto:Path: /qskrps-14/ 2023-05-24T21:11:41.054 DEBUG:boto:Data: 2023-05-24T21:11:41.055 DEBUG:boto:Headers: {} 2023-05-24T21:11:41.055 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:41.055 DEBUG:boto:Port: 8000 2023-05-24T21:11:41.056 DEBUG:boto:Params: {} 2023-05-24T21:11:41.056 DEBUG:boto:Token: None 2023-05-24T21:11:41.056 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:11:41 GMT /qskrps-14/ 2023-05-24T21:11:41.057 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5Mv1WH25Sxq05ugzijQ9YeppWFI= 2023-05-24T21:11:41.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:41 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5Mv1WH25Sxq05ugzijQ9YeppWFI=', 'Content-Length': '0'} 2023-05-24T21:11:41.059 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f87f5650fe1fcf6-00646e7d8d-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:41.060 DEBUG:boto:path=/qskrps-14/obj 2023-05-24T21:11:41.060 DEBUG:boto:auth_path=/qskrps-14/obj 2023-05-24T21:11:41.061 DEBUG:boto:Method: PUT 2023-05-24T21:11:41.061 DEBUG:boto:Path: /qskrps-14/obj 2023-05-24T21:11:41.061 DEBUG:boto:Data: 2023-05-24T21:11:41.062 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-05-24T21:11:41.062 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:11:41.062 DEBUG:boto:Port: 8000 2023-05-24T21:11:41.063 DEBUG:boto:Params: {} 2023-05-24T21:11:41.063 DEBUG:boto:Token: None 2023-05-24T21:11:41.064 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Wed, 24 May 2023 21:11:41 GMT /qskrps-14/obj 2023-05-24T21:11:41.064 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:6440LndfcooM5MH7mE7QEYqinK4= 2023-05-24T21:11:41.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:11:41 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:6440LndfcooM5MH7mE7QEYqinK4='} 2023-05-24T21:11:41.070 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b25652dab95cc86c-00646e7d8d-110d-a1'), ('Date', 'Wed, 24 May 2023 21:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:11:41.071 DEBUG:rgw_multi.tests:created initial version id=None 2023-05-24T21:11:41.071 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=qskrps-14 2023-05-24T21:11:41.072 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:11:41.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:11:41.073 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-14 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:11:41.128 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:11:41.128 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:11:41.173 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:11:41.166+0000 7f9bb2287780 1 waiting to reach incremental sync.. 2023-05-24T21:11:46.173 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:11:46.167+0000 7f9bb2287780 1 waiting to reach incremental sync.. 2023-05-24T21:11:51.174 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:11:51.168+0000 7f9bb2287780 1 waiting to reach incremental sync.. 2023-05-24T21:11:56.176 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:11:56.170+0000 7f9bb2287780 1 bucket sync caught up with source: 2023-05-24T21:11:56.176 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000001.6.6, , ] 2023-05-24T21:11:56.176 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000001.6.6, , ] 2023-05-24T21:11:56.177 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:11:56.170+0000 7f9bb2287780 0 bucket checkpoint complete 2023-05-24T21:11:56.197 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=qskrps-14 2023-05-24T21:11:56.197 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:11:56.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:11:56.198 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:11:56.260 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:11:56.260 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:11:56.314 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:11:56.308+0000 7faebd007780 1 waiting to reach incremental sync.. 2023-05-24T21:12:01.314 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:12:01.308+0000 7faebd007780 1 waiting to reach incremental sync.. 2023-05-24T21:12:06.315 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:12:06.309+0000 7faebd007780 1 waiting to reach incremental sync.. 2023-05-24T21:12:11.317 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:12:11.311+0000 7faebd007780 1 bucket sync caught up with source: 2023-05-24T21:12:11.317 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000001.14.6, , ] 2023-05-24T21:12:11.317 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000001.14.6, , ] 2023-05-24T21:12:11.317 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:12:11.311+0000 7faebd007780 0 bucket checkpoint complete 2023-05-24T21:12:11.336 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qskrps-14 zones={a2, a1} 2023-05-24T21:12:11.337 DEBUG:boto:path=/qskrps-14/ 2023-05-24T21:12:11.337 DEBUG:boto:auth_path=/qskrps-14/ 2023-05-24T21:12:11.338 DEBUG:boto:Method: HEAD 2023-05-24T21:12:11.338 DEBUG:boto:Path: /qskrps-14/ 2023-05-24T21:12:11.338 DEBUG:boto:Data: 2023-05-24T21:12:11.339 DEBUG:boto:Headers: {} 2023-05-24T21:12:11.339 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:12:11.339 DEBUG:boto:Port: 8001 2023-05-24T21:12:11.340 DEBUG:boto:Params: {} 2023-05-24T21:12:11.340 DEBUG:boto:Token: None 2023-05-24T21:12:11.341 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:12:11 GMT /qskrps-14/ 2023-05-24T21:12:11.341 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:MzwBoA7no3UPTBSmsVhvqJO7UXI= 2023-05-24T21:12:11.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:MzwBoA7no3UPTBSmsVhvqJO7UXI=', 'Content-Length': '0'} 2023-05-24T21:12:11.344 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cd975f8963a85d9e-00646e7dab-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:11.344 DEBUG:boto:path=/qskrps-14/ 2023-05-24T21:12:11.345 DEBUG:boto:auth_path=/qskrps-14/ 2023-05-24T21:12:11.345 DEBUG:boto:Method: HEAD 2023-05-24T21:12:11.345 DEBUG:boto:Path: /qskrps-14/ 2023-05-24T21:12:11.346 DEBUG:boto:Data: 2023-05-24T21:12:11.346 DEBUG:boto:Headers: {} 2023-05-24T21:12:11.346 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:12:11.347 DEBUG:boto:Port: 8000 2023-05-24T21:12:11.347 DEBUG:boto:Params: {} 2023-05-24T21:12:11.347 DEBUG:boto:Token: None 2023-05-24T21:12:11.348 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:12:11 GMT /qskrps-14/ 2023-05-24T21:12:11.348 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:MzwBoA7no3UPTBSmsVhvqJO7UXI= 2023-05-24T21:12:11.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:MzwBoA7no3UPTBSmsVhvqJO7UXI=', 'Content-Length': '0'} 2023-05-24T21:12:11.351 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003de92953b58cb115-00646e7dab-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:11.352 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-24T21:12:11.352 DEBUG:boto:path=/qskrps-14/ 2023-05-24T21:12:11.352 DEBUG:boto:auth_path=/qskrps-14/ 2023-05-24T21:12:11.353 DEBUG:boto:path=/qskrps-14/?versions&max-keys=999 2023-05-24T21:12:11.353 DEBUG:boto:auth_path=/qskrps-14/?versions&max-keys=999 2023-05-24T21:12:11.353 DEBUG:boto:Method: GET 2023-05-24T21:12:11.354 DEBUG:boto:Path: /qskrps-14/?versions&max-keys=999 2023-05-24T21:12:11.354 DEBUG:boto:Data: 2023-05-24T21:12:11.354 DEBUG:boto:Headers: {} 2023-05-24T21:12:11.355 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:12:11.355 DEBUG:boto:Port: 8001 2023-05-24T21:12:11.355 DEBUG:boto:Params: {} 2023-05-24T21:12:11.356 DEBUG:boto:Token: None 2023-05-24T21:12:11.356 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:12:11 GMT /qskrps-14/?versions 2023-05-24T21:12:11.357 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:rWahvQXcFcgDsm20fzKP07CRdZs= 2023-05-24T21:12:11.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:rWahvQXcFcgDsm20fzKP07CRdZs=', 'Content-Length': '0'} 2023-05-24T21:12:11.360 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb34778a565a30ee-00646e7dab-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:11.360 DEBUG:boto:b'qskrps-14999falseobjnulltrue2023-05-24T21:11:41.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:12:11.361 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:12:11.361 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-24T21:12:11.362 DEBUG:boto:path=/qskrps-14/ 2023-05-24T21:12:11.362 DEBUG:boto:auth_path=/qskrps-14/ 2023-05-24T21:12:11.362 DEBUG:boto:path=/qskrps-14/?versions&max-keys=999 2023-05-24T21:12:11.363 DEBUG:boto:auth_path=/qskrps-14/?versions&max-keys=999 2023-05-24T21:12:11.363 DEBUG:boto:Method: GET 2023-05-24T21:12:11.363 DEBUG:boto:Path: /qskrps-14/?versions&max-keys=999 2023-05-24T21:12:11.364 DEBUG:boto:Data: 2023-05-24T21:12:11.364 DEBUG:boto:Headers: {} 2023-05-24T21:12:11.364 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:12:11.365 DEBUG:boto:Port: 8000 2023-05-24T21:12:11.365 DEBUG:boto:Params: {} 2023-05-24T21:12:11.365 DEBUG:boto:Token: None 2023-05-24T21:12:11.366 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:12:11 GMT /qskrps-14/?versions 2023-05-24T21:12:11.366 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:rWahvQXcFcgDsm20fzKP07CRdZs= 2023-05-24T21:12:11.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:rWahvQXcFcgDsm20fzKP07CRdZs=', 'Content-Length': '0'} 2023-05-24T21:12:11.369 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a4d57d7ce506012-00646e7dab-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:11.369 DEBUG:boto:b'qskrps-14999falseobjnulltrue2023-05-24T21:11:41.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:12:11.370 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:12:11.370 DEBUG:boto:path=/qskrps-14/ 2023-05-24T21:12:11.371 DEBUG:boto:auth_path=/qskrps-14/ 2023-05-24T21:12:11.371 DEBUG:boto:path=/qskrps-14/?versions&max-keys=999 2023-05-24T21:12:11.371 DEBUG:boto:auth_path=/qskrps-14/?versions&max-keys=999 2023-05-24T21:12:11.372 DEBUG:boto:Method: GET 2023-05-24T21:12:11.372 DEBUG:boto:Path: /qskrps-14/?versions&max-keys=999 2023-05-24T21:12:11.372 DEBUG:boto:Data: 2023-05-24T21:12:11.373 DEBUG:boto:Headers: {} 2023-05-24T21:12:11.373 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:12:11.373 DEBUG:boto:Port: 8001 2023-05-24T21:12:11.374 DEBUG:boto:Params: {} 2023-05-24T21:12:11.374 DEBUG:boto:Token: None 2023-05-24T21:12:11.375 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:12:11 GMT /qskrps-14/?versions 2023-05-24T21:12:11.375 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:rWahvQXcFcgDsm20fzKP07CRdZs= 2023-05-24T21:12:11.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:rWahvQXcFcgDsm20fzKP07CRdZs=', 'Content-Length': '0'} 2023-05-24T21:12:11.378 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfe391492b2411e3-00646e7dab-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:11.378 DEBUG:boto:b'qskrps-14999falseobjnulltrue2023-05-24T21:11:41.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:12:11.379 DEBUG:boto:path=/qskrps-14/ 2023-05-24T21:12:11.379 DEBUG:boto:auth_path=/qskrps-14/ 2023-05-24T21:12:11.380 DEBUG:boto:path=/qskrps-14/?versions&max-keys=999 2023-05-24T21:12:11.380 DEBUG:boto:auth_path=/qskrps-14/?versions&max-keys=999 2023-05-24T21:12:11.380 DEBUG:boto:Method: GET 2023-05-24T21:12:11.381 DEBUG:boto:Path: /qskrps-14/?versions&max-keys=999 2023-05-24T21:12:11.381 DEBUG:boto:Data: 2023-05-24T21:12:11.381 DEBUG:boto:Headers: {} 2023-05-24T21:12:11.382 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:12:11.382 DEBUG:boto:Port: 8000 2023-05-24T21:12:11.382 DEBUG:boto:Params: {} 2023-05-24T21:12:11.383 DEBUG:boto:Token: None 2023-05-24T21:12:11.383 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:12:11 GMT /qskrps-14/?versions 2023-05-24T21:12:11.384 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:rWahvQXcFcgDsm20fzKP07CRdZs= 2023-05-24T21:12:11.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:rWahvQXcFcgDsm20fzKP07CRdZs=', 'Content-Length': '0'} 2023-05-24T21:12:11.387 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022293414f59031c2-00646e7dab-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:11.428 DEBUG:boto:b'qskrps-14999falseobjnulltrue2023-05-24T21:11:41.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:12:11.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:12:11.430 DEBUG:boto:path=/qskrps-14/obj 2023-05-24T21:12:11.430 DEBUG:boto:auth_path=/qskrps-14/obj 2023-05-24T21:12:11.430 DEBUG:boto:path=/qskrps-14/obj?versionId=null 2023-05-24T21:12:11.431 DEBUG:boto:auth_path=/qskrps-14/obj?versionId=null 2023-05-24T21:12:11.431 DEBUG:boto:Method: GET 2023-05-24T21:12:11.431 DEBUG:boto:Path: /qskrps-14/obj?versionId=null 2023-05-24T21:12:11.432 DEBUG:boto:Data: 2023-05-24T21:12:11.432 DEBUG:boto:Headers: {} 2023-05-24T21:12:11.433 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:12:11.433 DEBUG:boto:Port: 8001 2023-05-24T21:12:11.433 DEBUG:boto:Params: {} 2023-05-24T21:12:11.434 DEBUG:boto:Token: None 2023-05-24T21:12:11.434 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:12:11 GMT /qskrps-14/obj?versionId=null 2023-05-24T21:12:11.435 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:qe78chEEAKZ29B8x9kmpdOcJ1zw= 2023-05-24T21:12:11.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qe78chEEAKZ29B8x9kmpdOcJ1zw=', 'Content-Length': '0'} 2023-05-24T21:12:11.437 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:11:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b16d896956fd782e-00646e7dab-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:11.438 DEBUG:boto:path=/qskrps-14/obj 2023-05-24T21:12:11.438 DEBUG:boto:auth_path=/qskrps-14/obj 2023-05-24T21:12:11.438 DEBUG:boto:path=/qskrps-14/obj?versionId=null 2023-05-24T21:12:11.439 DEBUG:boto:auth_path=/qskrps-14/obj?versionId=null 2023-05-24T21:12:11.439 DEBUG:boto:Method: GET 2023-05-24T21:12:11.439 DEBUG:boto:Path: /qskrps-14/obj?versionId=null 2023-05-24T21:12:11.440 DEBUG:boto:Data: 2023-05-24T21:12:11.440 DEBUG:boto:Headers: {} 2023-05-24T21:12:11.440 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:12:11.441 DEBUG:boto:Port: 8000 2023-05-24T21:12:11.441 DEBUG:boto:Params: {} 2023-05-24T21:12:11.442 DEBUG:boto:Token: None 2023-05-24T21:12:11.442 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:12:11 GMT /qskrps-14/obj?versionId=null 2023-05-24T21:12:11.443 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:qe78chEEAKZ29B8x9kmpdOcJ1zw= 2023-05-24T21:12:11.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qe78chEEAKZ29B8x9kmpdOcJ1zw=', 'Content-Length': '0'} 2023-05-24T21:12:11.445 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:11:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000016712e6ae7788499-00646e7dab-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:11.446 DEBUG:boto:path=/qskrps-14/obj 2023-05-24T21:12:11.446 DEBUG:boto:auth_path=/qskrps-14/obj 2023-05-24T21:12:11.446 DEBUG:boto:path=/qskrps-14/obj?versionId=null 2023-05-24T21:12:11.447 DEBUG:boto:auth_path=/qskrps-14/obj?versionId=null 2023-05-24T21:12:11.447 DEBUG:boto:Method: HEAD 2023-05-24T21:12:11.447 DEBUG:boto:Path: /qskrps-14/obj?versionId=null 2023-05-24T21:12:11.448 DEBUG:boto:Data: 2023-05-24T21:12:11.448 DEBUG:boto:Headers: {} 2023-05-24T21:12:11.449 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:12:11.449 DEBUG:boto:Port: 8001 2023-05-24T21:12:11.449 DEBUG:boto:Params: {} 2023-05-24T21:12:11.450 DEBUG:boto:Token: None 2023-05-24T21:12:11.450 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:12:11 GMT /qskrps-14/obj?versionId=null 2023-05-24T21:12:11.451 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:pJ4J/0fPJ9kmeb1IxabNPyGN3Cg= 2023-05-24T21:12:11.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:pJ4J/0fPJ9kmeb1IxabNPyGN3Cg=', 'Content-Length': '0'} 2023-05-24T21:12:11.453 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:11:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004662871a439eaa4e-00646e7dab-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:11.454 DEBUG:boto:path=/qskrps-14/obj 2023-05-24T21:12:11.454 DEBUG:boto:auth_path=/qskrps-14/obj 2023-05-24T21:12:11.454 DEBUG:boto:path=/qskrps-14/obj?versionId=null 2023-05-24T21:12:11.455 DEBUG:boto:auth_path=/qskrps-14/obj?versionId=null 2023-05-24T21:12:11.455 DEBUG:boto:Method: HEAD 2023-05-24T21:12:11.455 DEBUG:boto:Path: /qskrps-14/obj?versionId=null 2023-05-24T21:12:11.456 DEBUG:boto:Data: 2023-05-24T21:12:11.456 DEBUG:boto:Headers: {} 2023-05-24T21:12:11.456 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:12:11.457 DEBUG:boto:Port: 8000 2023-05-24T21:12:11.457 DEBUG:boto:Params: {} 2023-05-24T21:12:11.457 DEBUG:boto:Token: None 2023-05-24T21:12:11.458 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:12:11 GMT /qskrps-14/obj?versionId=null 2023-05-24T21:12:11.458 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:pJ4J/0fPJ9kmeb1IxabNPyGN3Cg= 2023-05-24T21:12:11.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:pJ4J/0fPJ9kmeb1IxabNPyGN3Cg=', 'Content-Length': '0'} 2023-05-24T21:12:11.461 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:11:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b9241e9a801cb341-00646e7dab-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:11.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:12:11.462 DEBUG:boto:path=/qskrps-14/obj 2023-05-24T21:12:11.462 DEBUG:boto:auth_path=/qskrps-14/obj 2023-05-24T21:12:11.462 DEBUG:boto:path=/qskrps-14/obj?versionId=null 2023-05-24T21:12:11.463 DEBUG:boto:auth_path=/qskrps-14/obj?versionId=null 2023-05-24T21:12:11.463 DEBUG:boto:Method: GET 2023-05-24T21:12:11.464 DEBUG:boto:Path: /qskrps-14/obj?versionId=null 2023-05-24T21:12:11.464 DEBUG:boto:Data: 2023-05-24T21:12:11.464 DEBUG:boto:Headers: {} 2023-05-24T21:12:11.465 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:12:11.465 DEBUG:boto:Port: 8001 2023-05-24T21:12:11.466 DEBUG:boto:Params: {} 2023-05-24T21:12:11.466 DEBUG:boto:Token: None 2023-05-24T21:12:11.466 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:12:11 GMT /qskrps-14/obj?versionId=null 2023-05-24T21:12:11.467 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:qe78chEEAKZ29B8x9kmpdOcJ1zw= 2023-05-24T21:12:11.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qe78chEEAKZ29B8x9kmpdOcJ1zw=', 'Content-Length': '0'} 2023-05-24T21:12:11.469 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:11:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004ed791d419f0fabf-00646e7dab-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:11.470 DEBUG:boto:path=/qskrps-14/obj 2023-05-24T21:12:11.470 DEBUG:boto:auth_path=/qskrps-14/obj 2023-05-24T21:12:11.471 DEBUG:boto:path=/qskrps-14/obj?versionId=null 2023-05-24T21:12:11.471 DEBUG:boto:auth_path=/qskrps-14/obj?versionId=null 2023-05-24T21:12:11.471 DEBUG:boto:Method: GET 2023-05-24T21:12:11.472 DEBUG:boto:Path: /qskrps-14/obj?versionId=null 2023-05-24T21:12:11.472 DEBUG:boto:Data: 2023-05-24T21:12:11.472 DEBUG:boto:Headers: {} 2023-05-24T21:12:11.473 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:12:11.473 DEBUG:boto:Port: 8000 2023-05-24T21:12:11.473 DEBUG:boto:Params: {} 2023-05-24T21:12:11.474 DEBUG:boto:Token: None 2023-05-24T21:12:11.474 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:12:11 GMT /qskrps-14/obj?versionId=null 2023-05-24T21:12:11.475 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:qe78chEEAKZ29B8x9kmpdOcJ1zw= 2023-05-24T21:12:11.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qe78chEEAKZ29B8x9kmpdOcJ1zw=', 'Content-Length': '0'} 2023-05-24T21:12:11.477 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:11:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006fa04035c52fec5a-00646e7dab-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:11.478 DEBUG:boto:path=/qskrps-14/ 2023-05-24T21:12:11.478 DEBUG:boto:auth_path=/qskrps-14/ 2023-05-24T21:12:11.478 DEBUG:boto:path=/qskrps-14/?prefix=obj 2023-05-24T21:12:11.479 DEBUG:boto:auth_path=/qskrps-14/?prefix=obj 2023-05-24T21:12:11.479 DEBUG:boto:Method: GET 2023-05-24T21:12:11.480 DEBUG:boto:Path: /qskrps-14/?prefix=obj 2023-05-24T21:12:11.480 DEBUG:boto:Data: 2023-05-24T21:12:11.481 DEBUG:boto:Headers: {} 2023-05-24T21:12:11.481 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:12:11.481 DEBUG:boto:Port: 8001 2023-05-24T21:12:11.482 DEBUG:boto:Params: {} 2023-05-24T21:12:11.482 DEBUG:boto:Token: None 2023-05-24T21:12:11.482 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:12:11 GMT /qskrps-14/ 2023-05-24T21:12:11.483 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:hMEQm9ksDbEDTkkWUFU5Xqblnyg= 2023-05-24T21:12:11.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:hMEQm9ksDbEDTkkWUFU5Xqblnyg=', 'Content-Length': '0'} 2023-05-24T21:12:11.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a13f9fe12b370ca0-00646e7dab-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:11.528 DEBUG:boto:b'qskrps-14obj1000falseobj2023-05-24T21:11:41.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:12:11.529 DEBUG:boto:path=/qskrps-14/ 2023-05-24T21:12:11.530 DEBUG:boto:auth_path=/qskrps-14/ 2023-05-24T21:12:11.530 DEBUG:boto:path=/qskrps-14/?prefix=obj 2023-05-24T21:12:11.531 DEBUG:boto:auth_path=/qskrps-14/?prefix=obj 2023-05-24T21:12:11.531 DEBUG:boto:Method: GET 2023-05-24T21:12:11.531 DEBUG:boto:Path: /qskrps-14/?prefix=obj 2023-05-24T21:12:11.532 DEBUG:boto:Data: 2023-05-24T21:12:11.532 DEBUG:boto:Headers: {} 2023-05-24T21:12:11.532 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:12:11.533 DEBUG:boto:Port: 8000 2023-05-24T21:12:11.533 DEBUG:boto:Params: {} 2023-05-24T21:12:11.534 DEBUG:boto:Token: None 2023-05-24T21:12:11.534 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:12:11 GMT /qskrps-14/ 2023-05-24T21:12:11.535 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:hMEQm9ksDbEDTkkWUFU5Xqblnyg= 2023-05-24T21:12:11.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:hMEQm9ksDbEDTkkWUFU5Xqblnyg=', 'Content-Length': '0'} 2023-05-24T21:12:11.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a512af78c73418d7-00646e7dab-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:11.538 DEBUG:boto:b'qskrps-14obj1000falseobj2023-05-24T21:11:41.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:12:11.539 DEBUG:boto:path=/qskrps-14/obj 2023-05-24T21:12:11.539 DEBUG:boto:auth_path=/qskrps-14/obj 2023-05-24T21:12:11.539 DEBUG:boto:Method: HEAD 2023-05-24T21:12:11.540 DEBUG:boto:Path: /qskrps-14/obj 2023-05-24T21:12:11.540 DEBUG:boto:Data: 2023-05-24T21:12:11.541 DEBUG:boto:Headers: {} 2023-05-24T21:12:11.541 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:12:11.541 DEBUG:boto:Port: 8001 2023-05-24T21:12:11.542 DEBUG:boto:Params: {} 2023-05-24T21:12:11.542 DEBUG:boto:Token: None 2023-05-24T21:12:11.543 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:12:11 GMT /qskrps-14/obj 2023-05-24T21:12:11.543 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Ajtt4n/GcpLB+i9gdbtAJWPJCr0= 2023-05-24T21:12:11.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Ajtt4n/GcpLB+i9gdbtAJWPJCr0=', 'Content-Length': '0'} 2023-05-24T21:12:11.545 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:11:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c39e473d0a3827c6-00646e7dab-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:11.546 DEBUG:boto:path=/qskrps-14/obj 2023-05-24T21:12:11.546 DEBUG:boto:auth_path=/qskrps-14/obj 2023-05-24T21:12:11.547 DEBUG:boto:Method: HEAD 2023-05-24T21:12:11.547 DEBUG:boto:Path: /qskrps-14/obj 2023-05-24T21:12:11.547 DEBUG:boto:Data: 2023-05-24T21:12:11.548 DEBUG:boto:Headers: {} 2023-05-24T21:12:11.548 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:12:11.548 DEBUG:boto:Port: 8000 2023-05-24T21:12:11.549 DEBUG:boto:Params: {} 2023-05-24T21:12:11.549 DEBUG:boto:Token: None 2023-05-24T21:12:11.549 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:12:11 GMT /qskrps-14/obj 2023-05-24T21:12:11.550 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Ajtt4n/GcpLB+i9gdbtAJWPJCr0= 2023-05-24T21:12:11.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Ajtt4n/GcpLB+i9gdbtAJWPJCr0=', 'Content-Length': '0'} 2023-05-24T21:12:11.552 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:11:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fea591ddd5849b82-00646e7dab-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:11.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:12:11.553 DEBUG:boto:path=/qskrps-14/obj 2023-05-24T21:12:11.554 DEBUG:boto:auth_path=/qskrps-14/obj 2023-05-24T21:12:11.554 DEBUG:boto:Method: GET 2023-05-24T21:12:11.554 DEBUG:boto:Path: /qskrps-14/obj 2023-05-24T21:12:11.555 DEBUG:boto:Data: 2023-05-24T21:12:11.555 DEBUG:boto:Headers: {} 2023-05-24T21:12:11.555 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:12:11.556 DEBUG:boto:Port: 8001 2023-05-24T21:12:11.556 DEBUG:boto:Params: {} 2023-05-24T21:12:11.557 DEBUG:boto:Token: None 2023-05-24T21:12:11.557 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:12:11 GMT /qskrps-14/obj 2023-05-24T21:12:11.557 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:nGEdrgj9zbOzOyVatPa3ZZoOjf4= 2023-05-24T21:12:11.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:nGEdrgj9zbOzOyVatPa3ZZoOjf4=', 'Content-Length': '0'} 2023-05-24T21:12:11.559 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:11:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002df7a91b05206540-00646e7dab-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:11.560 DEBUG:boto:path=/qskrps-14/obj 2023-05-24T21:12:11.561 DEBUG:boto:auth_path=/qskrps-14/obj 2023-05-24T21:12:11.561 DEBUG:boto:Method: GET 2023-05-24T21:12:11.562 DEBUG:boto:Path: /qskrps-14/obj 2023-05-24T21:12:11.562 DEBUG:boto:Data: 2023-05-24T21:12:11.562 DEBUG:boto:Headers: {} 2023-05-24T21:12:11.563 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:12:11.563 DEBUG:boto:Port: 8000 2023-05-24T21:12:11.563 DEBUG:boto:Params: {} 2023-05-24T21:12:11.564 DEBUG:boto:Token: None 2023-05-24T21:12:11.564 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:12:11 GMT /qskrps-14/obj 2023-05-24T21:12:11.565 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:nGEdrgj9zbOzOyVatPa3ZZoOjf4= 2023-05-24T21:12:11.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:nGEdrgj9zbOzOyVatPa3ZZoOjf4=', 'Content-Length': '0'} 2023-05-24T21:12:11.567 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:11:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b67a4a19d34781e1-00646e7dab-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:11.567 DEBUG:boto:path=/qskrps-14/ 2023-05-24T21:12:11.568 DEBUG:boto:auth_path=/qskrps-14/ 2023-05-24T21:12:11.568 DEBUG:boto:path=/qskrps-14/?prefix=obj 2023-05-24T21:12:11.569 DEBUG:boto:auth_path=/qskrps-14/?prefix=obj 2023-05-24T21:12:11.569 DEBUG:boto:Method: GET 2023-05-24T21:12:11.569 DEBUG:boto:Path: /qskrps-14/?prefix=obj 2023-05-24T21:12:11.570 DEBUG:boto:Data: 2023-05-24T21:12:11.570 DEBUG:boto:Headers: {} 2023-05-24T21:12:11.570 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:12:11.571 DEBUG:boto:Port: 8001 2023-05-24T21:12:11.571 DEBUG:boto:Params: {} 2023-05-24T21:12:11.572 DEBUG:boto:Token: None 2023-05-24T21:12:11.572 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:12:11 GMT /qskrps-14/ 2023-05-24T21:12:11.573 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:hMEQm9ksDbEDTkkWUFU5Xqblnyg= 2023-05-24T21:12:11.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:hMEQm9ksDbEDTkkWUFU5Xqblnyg=', 'Content-Length': '0'} 2023-05-24T21:12:11.575 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed362bb94a0c13ed-00646e7dab-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:11.576 DEBUG:boto:b'qskrps-14obj1000falseobj2023-05-24T21:11:41.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:12:11.577 DEBUG:boto:path=/qskrps-14/ 2023-05-24T21:12:11.577 DEBUG:boto:auth_path=/qskrps-14/ 2023-05-24T21:12:11.577 DEBUG:boto:path=/qskrps-14/?prefix=obj 2023-05-24T21:12:11.578 DEBUG:boto:auth_path=/qskrps-14/?prefix=obj 2023-05-24T21:12:11.578 DEBUG:boto:Method: GET 2023-05-24T21:12:11.578 DEBUG:boto:Path: /qskrps-14/?prefix=obj 2023-05-24T21:12:11.579 DEBUG:boto:Data: 2023-05-24T21:12:11.579 DEBUG:boto:Headers: {} 2023-05-24T21:12:11.579 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:12:11.580 DEBUG:boto:Port: 8000 2023-05-24T21:12:11.580 DEBUG:boto:Params: {} 2023-05-24T21:12:11.581 DEBUG:boto:Token: None 2023-05-24T21:12:11.581 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:12:11 GMT /qskrps-14/ 2023-05-24T21:12:11.582 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:hMEQm9ksDbEDTkkWUFU5Xqblnyg= 2023-05-24T21:12:11.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:hMEQm9ksDbEDTkkWUFU5Xqblnyg=', 'Content-Length': '0'} 2023-05-24T21:12:11.585 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055a8b54a978ae57e-00646e7dab-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:11.586 DEBUG:boto:b'qskrps-14obj1000falseobj2023-05-24T21:11:41.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:12:11.586 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qskrps-14 zones={a2, a1} 2023-05-24T21:12:11.587 DEBUG:boto:path=/qskrps-14/ 2023-05-24T21:12:11.587 DEBUG:boto:auth_path=/qskrps-14/ 2023-05-24T21:12:11.587 DEBUG:boto:path=/qskrps-14/?versioning 2023-05-24T21:12:11.588 DEBUG:boto:auth_path=/qskrps-14/?versioning 2023-05-24T21:12:11.588 DEBUG:boto:Method: PUT 2023-05-24T21:12:11.588 DEBUG:boto:Path: /qskrps-14/?versioning 2023-05-24T21:12:11.589 DEBUG:boto:Data: Enabled Disabled 2023-05-24T21:12:11.589 DEBUG:boto:Headers: {} 2023-05-24T21:12:11.589 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:12:11.590 DEBUG:boto:Port: 8000 2023-05-24T21:12:11.590 DEBUG:boto:Params: {} 2023-05-24T21:12:11.590 DEBUG:boto:Token: None 2023-05-24T21:12:11.591 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:12:11 GMT /qskrps-14/?versioning 2023-05-24T21:12:11.591 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:+EPPPAfh130nDTh0RSPNdv4Prnw= 2023-05-24T21:12:11.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:+EPPPAfh130nDTh0RSPNdv4Prnw=', 'Content-Length': '228'} 2023-05-24T21:12:11.598 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fa48085c351d9319-00646e7dab-110d-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:11.598 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:12:11.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:12:11.599 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:12:11.655 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:12:11.656 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:12:11.705 INFO:teuthology.orchestra.run.smithi071.stderr:No --period given, using current period=5c04936b-70ea-4415-b4eb-3aad3dc612e5 2023-05-24T21:12:11.718 INFO:teuthology.orchestra.run.smithi071.stdout:[ 2023-05-24T21:12:11.718 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:12:11.718 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962731.592071_145.1", 2023-05-24T21:12:11.719 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:12:11.592071Z" 2023-05-24T21:12:11.719 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:12:11.719 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:12:11.720 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:12:11.720 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:06:32.333844Z" 2023-05-24T21:12:11.720 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:12:11.721 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:12:11.721 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:12:11.721 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:12:11.722 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:12:11.722 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:12:11.722 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:12:11.722 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:12:11.723 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:12:11.723 INFO:teuthology.orchestra.run.smithi071.stdout:] 2023-05-24T21:12:11.725 DEBUG:rgw_multi.tests:master meta markers={0: '1_1684962731.592071_145.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:12:11.726 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-24T21:12:11.726 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:12:11.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:12:11.727 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:12:11.779 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:12:11.779 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:12:11.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:12:11.809+0000 7fbdc0be4780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:12:11.816 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:12:11.810+0000 7fbdc0be4780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:12:11.827 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:12:11.827 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:12:11.828 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:12:11.828 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:12:11.828 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:12:11.828 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:12:11.829 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:12:11.829 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:12:11.829 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:12:11.829 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:12:11.829 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:12:11.829 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:12:11.829 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:12:11.830 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962731.592071_145.1", 2023-05-24T21:12:11.830 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:12:11.830 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:12:11.830 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:12:11.831 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:12:11.592071Z", 2023-05-24T21:12:11.831 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:12:11.831 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:12:11.832 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:12:11.832 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:12:11.832 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:12:11.832 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:12:11.833 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:12:11.833 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:12:11.833 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:12:11.833 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:12:11.833 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:12:11.834 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:12:11.834 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:12:11.834 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:12:11.834 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:12:11.834 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:12:11.834 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:12:11.835 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:12:11.835 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:12:11.835 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:12:11.835 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:12:11.835 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:12:11.835 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:12:11.835 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:12:11.836 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:12:11.836 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:12:11.836 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:12:11.836 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:12:11.836 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:12:11.836 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:12:11.836 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:12:11.837 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:12:11.837 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:12:11.837 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:12:11.837 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:12:11.837 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:12:11.837 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:12:11.838 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:12:11.838 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:12:11.838 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:12:11.838 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:12:11.838 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:12:11.838 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:12:11.838 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:12:11.839 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:12:11.839 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:12:11Z" 2023-05-24T21:12:11.839 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:12:11.840 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962731.592071_145.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:12:11.592071Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:12:11Z" } 2023-05-24T21:12:11.840 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962731.592071_145.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:12:11.592071Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:12:11.841 DEBUG:rgw_multi.tests:log_status={0: '1_1684962731.592071_145.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:12:11.841 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962731.592071_145.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:12:11.842 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-24T21:12:11.842 DEBUG:boto:path=/qskrps-14/ 2023-05-24T21:12:11.843 DEBUG:boto:auth_path=/qskrps-14/ 2023-05-24T21:12:11.844 DEBUG:boto:Method: HEAD 2023-05-24T21:12:11.844 DEBUG:boto:Path: /qskrps-14/ 2023-05-24T21:12:11.844 DEBUG:boto:Data: 2023-05-24T21:12:11.845 DEBUG:boto:Headers: {} 2023-05-24T21:12:11.845 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:12:11.846 DEBUG:boto:Port: 8000 2023-05-24T21:12:11.846 DEBUG:boto:Params: {} 2023-05-24T21:12:11.846 DEBUG:boto:Token: None 2023-05-24T21:12:11.847 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:12:11 GMT /qskrps-14/ 2023-05-24T21:12:11.847 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:MzwBoA7no3UPTBSmsVhvqJO7UXI= 2023-05-24T21:12:11.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:MzwBoA7no3UPTBSmsVhvqJO7UXI=', 'Content-Length': '0'} 2023-05-24T21:12:11.850 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d303e78ffea3088f-00646e7dab-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:11.850 DEBUG:boto:path=/qskrps-14/obj 2023-05-24T21:12:11.851 DEBUG:boto:auth_path=/qskrps-14/obj 2023-05-24T21:12:11.851 DEBUG:boto:Method: PUT 2023-05-24T21:12:11.852 DEBUG:boto:Path: /qskrps-14/obj 2023-05-24T21:12:11.852 DEBUG:boto:Data: 2023-05-24T21:12:11.852 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-05-24T21:12:11.853 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:12:11.853 DEBUG:boto:Port: 8000 2023-05-24T21:12:11.853 DEBUG:boto:Params: {} 2023-05-24T21:12:11.854 DEBUG:boto:Token: None 2023-05-24T21:12:11.854 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Wed, 24 May 2023 21:12:11 GMT /qskrps-14/obj 2023-05-24T21:12:11.855 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:98Kmj+pdBFIbUoQLH8CtlwXJNcU= 2023-05-24T21:12:11.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:12:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:98Kmj+pdBFIbUoQLH8CtlwXJNcU='} 2023-05-24T21:12:11.865 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ'), ('x-amz-request-id', 'tx00000984d0741f0fd46d7-00646e7dab-110d-a1'), ('Date', 'Wed, 24 May 2023 21:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:11.866 DEBUG:rgw_multi.tests:created new version id=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-24T21:12:11.867 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=qskrps-14 2023-05-24T21:12:11.867 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:12:11.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:12:11.869 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-14 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:12:11.926 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:12:11.927 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:12:11.977 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:12:11.971+0000 7f4b6b563780 1 waiting for incremental sync to catch up: 2023-05-24T21:12:11.978 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000001.6.6, , ] 2023-05-24T21:12:11.978 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000003.9.13, , ] 2023-05-24T21:12:16.978 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:12:16.973+0000 7f4b6b563780 1 waiting for incremental sync to catch up: 2023-05-24T21:12:16.979 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000001.6.6, , ] 2023-05-24T21:12:16.979 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000003.9.13, , ] 2023-05-24T21:12:21.981 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:12:21.974+0000 7f4b6b563780 1 waiting for incremental sync to catch up: 2023-05-24T21:12:21.982 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000001.6.6, , ] 2023-05-24T21:12:21.982 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000003.9.13, , ] 2023-05-24T21:12:26.981 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:12:26.975+0000 7f4b6b563780 1 waiting for incremental sync to catch up: 2023-05-24T21:12:26.981 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000001.6.6, , ] 2023-05-24T21:12:26.982 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000003.9.13, , ] 2023-05-24T21:12:31.983 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:12:31.976+0000 7f4b6b563780 1 bucket sync caught up with source: 2023-05-24T21:12:31.983 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000003.9.13, , ] 2023-05-24T21:12:31.983 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000003.9.13, , ] 2023-05-24T21:12:31.984 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:12:31.976+0000 7f4b6b563780 0 bucket checkpoint complete 2023-05-24T21:12:32.003 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=qskrps-14 2023-05-24T21:12:32.004 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:12:32.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:12:32.005 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:12:32.068 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:12:32.068 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:12:32.117 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:12:32.110+0000 7f9050f0f780 1 waiting for incremental sync to catch up: 2023-05-24T21:12:32.117 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000001.14.6, , ] 2023-05-24T21:12:32.117 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000003.19.13, , ] 2023-05-24T21:12:37.118 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:12:37.111+0000 7f9050f0f780 1 waiting for incremental sync to catch up: 2023-05-24T21:12:37.118 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000001.14.6, , ] 2023-05-24T21:12:37.118 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000003.19.13, , ] 2023-05-24T21:12:42.119 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:12:42.112+0000 7f9050f0f780 1 waiting for incremental sync to catch up: 2023-05-24T21:12:42.120 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000001.14.6, , ] 2023-05-24T21:12:42.120 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000003.19.13, , ] 2023-05-24T21:12:47.120 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:12:47.113+0000 7f9050f0f780 1 waiting for incremental sync to catch up: 2023-05-24T21:12:47.121 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000001.14.6, , ] 2023-05-24T21:12:47.121 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000003.19.13, , ] 2023-05-24T21:12:52.122 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:12:52.116+0000 7f9050f0f780 1 bucket sync caught up with source: 2023-05-24T21:12:52.122 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000003.19.13, , ] 2023-05-24T21:12:52.122 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000003.19.13, , ] 2023-05-24T21:12:52.122 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:12:52.116+0000 7f9050f0f780 0 bucket checkpoint complete 2023-05-24T21:12:52.142 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qskrps-14 zones={a2, a1} 2023-05-24T21:12:52.143 DEBUG:boto:path=/qskrps-14/ 2023-05-24T21:12:52.143 DEBUG:boto:auth_path=/qskrps-14/ 2023-05-24T21:12:52.144 DEBUG:boto:Method: HEAD 2023-05-24T21:12:52.144 DEBUG:boto:Path: /qskrps-14/ 2023-05-24T21:12:52.144 DEBUG:boto:Data: 2023-05-24T21:12:52.145 DEBUG:boto:Headers: {} 2023-05-24T21:12:52.145 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:12:52.146 DEBUG:boto:Port: 8001 2023-05-24T21:12:52.146 DEBUG:boto:Params: {} 2023-05-24T21:12:52.147 DEBUG:boto:Token: None 2023-05-24T21:12:52.147 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:12:52 GMT /qskrps-14/ 2023-05-24T21:12:52.148 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:0Cy+rIDC9q1jv0yqhIaIOr9Zt7A= 2023-05-24T21:12:52.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:0Cy+rIDC9q1jv0yqhIaIOr9Zt7A=', 'Content-Length': '0'} 2023-05-24T21:12:52.151 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008de07933e0eb021a-00646e7dd4-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:52.152 DEBUG:boto:path=/qskrps-14/ 2023-05-24T21:12:52.152 DEBUG:boto:auth_path=/qskrps-14/ 2023-05-24T21:12:52.153 DEBUG:boto:Method: HEAD 2023-05-24T21:12:52.153 DEBUG:boto:Path: /qskrps-14/ 2023-05-24T21:12:52.153 DEBUG:boto:Data: 2023-05-24T21:12:52.154 DEBUG:boto:Headers: {} 2023-05-24T21:12:52.154 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:12:52.154 DEBUG:boto:Port: 8000 2023-05-24T21:12:52.155 DEBUG:boto:Params: {} 2023-05-24T21:12:52.155 DEBUG:boto:Token: None 2023-05-24T21:12:52.155 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:12:52 GMT /qskrps-14/ 2023-05-24T21:12:52.156 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:0Cy+rIDC9q1jv0yqhIaIOr9Zt7A= 2023-05-24T21:12:52.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:0Cy+rIDC9q1jv0yqhIaIOr9Zt7A=', 'Content-Length': '0'} 2023-05-24T21:12:52.158 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ec67a3315e81f04-00646e7dd4-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:52.159 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-24T21:12:52.159 DEBUG:boto:path=/qskrps-14/ 2023-05-24T21:12:52.160 DEBUG:boto:auth_path=/qskrps-14/ 2023-05-24T21:12:52.160 DEBUG:boto:path=/qskrps-14/?versions&max-keys=999 2023-05-24T21:12:52.160 DEBUG:boto:auth_path=/qskrps-14/?versions&max-keys=999 2023-05-24T21:12:52.161 DEBUG:boto:Method: GET 2023-05-24T21:12:52.161 DEBUG:boto:Path: /qskrps-14/?versions&max-keys=999 2023-05-24T21:12:52.161 DEBUG:boto:Data: 2023-05-24T21:12:52.162 DEBUG:boto:Headers: {} 2023-05-24T21:12:52.162 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:12:52.163 DEBUG:boto:Port: 8001 2023-05-24T21:12:52.163 DEBUG:boto:Params: {} 2023-05-24T21:12:52.163 DEBUG:boto:Token: None 2023-05-24T21:12:52.164 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:12:52 GMT /qskrps-14/?versions 2023-05-24T21:12:52.164 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:CWB4xyHRVKNVc7Wqh3Tn8xuA6Yo= 2023-05-24T21:12:52.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:CWB4xyHRVKNVc7Wqh3Tn8xuA6Yo=', 'Content-Length': '0'} 2023-05-24T21:12:52.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001098095c98bd5ce8-00646e7dd4-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:52.167 DEBUG:boto:b'qskrps-14999falseobjpgpY3GcrHqwBxmj52IgrlMvqJeqfWWQtrue2023-05-24T21:12:11.852Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-05-24T21:11:41.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:12:52.168 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:12:52.168 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:12:52.169 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-24T21:12:52.169 DEBUG:boto:path=/qskrps-14/ 2023-05-24T21:12:52.169 DEBUG:boto:auth_path=/qskrps-14/ 2023-05-24T21:12:52.170 DEBUG:boto:path=/qskrps-14/?versions&max-keys=999 2023-05-24T21:12:52.170 DEBUG:boto:auth_path=/qskrps-14/?versions&max-keys=999 2023-05-24T21:12:52.170 DEBUG:boto:Method: GET 2023-05-24T21:12:52.171 DEBUG:boto:Path: /qskrps-14/?versions&max-keys=999 2023-05-24T21:12:52.171 DEBUG:boto:Data: 2023-05-24T21:12:52.172 DEBUG:boto:Headers: {} 2023-05-24T21:12:52.172 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:12:52.172 DEBUG:boto:Port: 8000 2023-05-24T21:12:52.173 DEBUG:boto:Params: {} 2023-05-24T21:12:52.173 DEBUG:boto:Token: None 2023-05-24T21:12:52.173 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:12:52 GMT /qskrps-14/?versions 2023-05-24T21:12:52.174 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:CWB4xyHRVKNVc7Wqh3Tn8xuA6Yo= 2023-05-24T21:12:52.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:CWB4xyHRVKNVc7Wqh3Tn8xuA6Yo=', 'Content-Length': '0'} 2023-05-24T21:12:52.176 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000952e6182db509d6e-00646e7dd4-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:52.177 DEBUG:boto:b'qskrps-14999falseobjpgpY3GcrHqwBxmj52IgrlMvqJeqfWWQtrue2023-05-24T21:12:11.852Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-05-24T21:11:41.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:12:52.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:12:52.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:12:52.178 DEBUG:boto:path=/qskrps-14/ 2023-05-24T21:12:52.179 DEBUG:boto:auth_path=/qskrps-14/ 2023-05-24T21:12:52.179 DEBUG:boto:path=/qskrps-14/?versions&max-keys=999 2023-05-24T21:12:52.179 DEBUG:boto:auth_path=/qskrps-14/?versions&max-keys=999 2023-05-24T21:12:52.180 DEBUG:boto:Method: GET 2023-05-24T21:12:52.180 DEBUG:boto:Path: /qskrps-14/?versions&max-keys=999 2023-05-24T21:12:52.180 DEBUG:boto:Data: 2023-05-24T21:12:52.181 DEBUG:boto:Headers: {} 2023-05-24T21:12:52.181 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:12:52.181 DEBUG:boto:Port: 8001 2023-05-24T21:12:52.182 DEBUG:boto:Params: {} 2023-05-24T21:12:52.182 DEBUG:boto:Token: None 2023-05-24T21:12:52.182 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:12:52 GMT /qskrps-14/?versions 2023-05-24T21:12:52.183 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:CWB4xyHRVKNVc7Wqh3Tn8xuA6Yo= 2023-05-24T21:12:52.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:CWB4xyHRVKNVc7Wqh3Tn8xuA6Yo=', 'Content-Length': '0'} 2023-05-24T21:12:52.185 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f88e9da453e89024-00646e7dd4-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:52.186 DEBUG:boto:b'qskrps-14999falseobjpgpY3GcrHqwBxmj52IgrlMvqJeqfWWQtrue2023-05-24T21:12:11.852Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-05-24T21:11:41.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:12:52.187 DEBUG:boto:path=/qskrps-14/ 2023-05-24T21:12:52.187 DEBUG:boto:auth_path=/qskrps-14/ 2023-05-24T21:12:52.187 DEBUG:boto:path=/qskrps-14/?versions&max-keys=999 2023-05-24T21:12:52.188 DEBUG:boto:auth_path=/qskrps-14/?versions&max-keys=999 2023-05-24T21:12:52.188 DEBUG:boto:Method: GET 2023-05-24T21:12:52.188 DEBUG:boto:Path: /qskrps-14/?versions&max-keys=999 2023-05-24T21:12:52.189 DEBUG:boto:Data: 2023-05-24T21:12:52.189 DEBUG:boto:Headers: {} 2023-05-24T21:12:52.189 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:12:52.190 DEBUG:boto:Port: 8000 2023-05-24T21:12:52.190 DEBUG:boto:Params: {} 2023-05-24T21:12:52.191 DEBUG:boto:Token: None 2023-05-24T21:12:52.191 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:12:52 GMT /qskrps-14/?versions 2023-05-24T21:12:52.191 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:CWB4xyHRVKNVc7Wqh3Tn8xuA6Yo= 2023-05-24T21:12:52.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:CWB4xyHRVKNVc7Wqh3Tn8xuA6Yo=', 'Content-Length': '0'} 2023-05-24T21:12:52.194 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fa150a230e5df29-00646e7dd4-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:52.195 DEBUG:boto:b'qskrps-14999falseobjpgpY3GcrHqwBxmj52IgrlMvqJeqfWWQtrue2023-05-24T21:12:11.852Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-05-24T21:11:41.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:12:52.196 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:12:52.197 DEBUG:boto:path=/qskrps-14/obj 2023-05-24T21:12:52.197 DEBUG:boto:auth_path=/qskrps-14/obj 2023-05-24T21:12:52.198 DEBUG:boto:path=/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-24T21:12:52.198 DEBUG:boto:auth_path=/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-24T21:12:52.199 DEBUG:boto:Method: GET 2023-05-24T21:12:52.199 DEBUG:boto:Path: /qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-24T21:12:52.199 DEBUG:boto:Data: 2023-05-24T21:12:52.200 DEBUG:boto:Headers: {} 2023-05-24T21:12:52.200 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:12:52.200 DEBUG:boto:Port: 8001 2023-05-24T21:12:52.201 DEBUG:boto:Params: {} 2023-05-24T21:12:52.201 DEBUG:boto:Token: None 2023-05-24T21:12:52.202 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:12:52 GMT /qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-24T21:12:52.202 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:wsFZcAhU2JLBSCRMGFMrN1vubfI= 2023-05-24T21:12:52.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wsFZcAhU2JLBSCRMGFMrN1vubfI=', 'Content-Length': '0'} 2023-05-24T21:12:52.204 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:12:11 GMT'), ('x-amz-version-id', 'pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f020e489be7834b0-00646e7dd4-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:52.205 DEBUG:boto:path=/qskrps-14/obj 2023-05-24T21:12:52.205 DEBUG:boto:auth_path=/qskrps-14/obj 2023-05-24T21:12:52.205 DEBUG:boto:path=/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-24T21:12:52.206 DEBUG:boto:auth_path=/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-24T21:12:52.206 DEBUG:boto:Method: GET 2023-05-24T21:12:52.207 DEBUG:boto:Path: /qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-24T21:12:52.207 DEBUG:boto:Data: 2023-05-24T21:12:52.207 DEBUG:boto:Headers: {} 2023-05-24T21:12:52.208 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:12:52.208 DEBUG:boto:Port: 8000 2023-05-24T21:12:52.208 DEBUG:boto:Params: {} 2023-05-24T21:12:52.209 DEBUG:boto:Token: None 2023-05-24T21:12:52.209 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:12:52 GMT /qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-24T21:12:52.210 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:wsFZcAhU2JLBSCRMGFMrN1vubfI= 2023-05-24T21:12:52.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wsFZcAhU2JLBSCRMGFMrN1vubfI=', 'Content-Length': '0'} 2023-05-24T21:12:52.212 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:12:11 GMT'), ('x-amz-version-id', 'pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000045b3a8eb15606305-00646e7dd4-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:52.213 DEBUG:boto:path=/qskrps-14/obj 2023-05-24T21:12:52.213 DEBUG:boto:auth_path=/qskrps-14/obj 2023-05-24T21:12:52.213 DEBUG:boto:path=/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-24T21:12:52.214 DEBUG:boto:auth_path=/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-24T21:12:52.214 DEBUG:boto:Method: HEAD 2023-05-24T21:12:52.214 DEBUG:boto:Path: /qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-24T21:12:52.215 DEBUG:boto:Data: 2023-05-24T21:12:52.215 DEBUG:boto:Headers: {} 2023-05-24T21:12:52.215 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:12:52.216 DEBUG:boto:Port: 8001 2023-05-24T21:12:52.216 DEBUG:boto:Params: {} 2023-05-24T21:12:52.216 DEBUG:boto:Token: None 2023-05-24T21:12:52.217 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:12:52 GMT /qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-24T21:12:52.217 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ArfXbmIiadu1cLfI1lKemiMrfRI= 2023-05-24T21:12:52.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ArfXbmIiadu1cLfI1lKemiMrfRI=', 'Content-Length': '0'} 2023-05-24T21:12:52.219 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:12:11 GMT'), ('x-amz-version-id', 'pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d278122f4806f89b-00646e7dd4-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:52.220 DEBUG:boto:path=/qskrps-14/obj 2023-05-24T21:12:52.220 DEBUG:boto:auth_path=/qskrps-14/obj 2023-05-24T21:12:52.221 DEBUG:boto:path=/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-24T21:12:52.221 DEBUG:boto:auth_path=/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-24T21:12:52.222 DEBUG:boto:Method: HEAD 2023-05-24T21:12:52.222 DEBUG:boto:Path: /qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-24T21:12:52.222 DEBUG:boto:Data: 2023-05-24T21:12:52.223 DEBUG:boto:Headers: {} 2023-05-24T21:12:52.223 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:12:52.223 DEBUG:boto:Port: 8000 2023-05-24T21:12:52.224 DEBUG:boto:Params: {} 2023-05-24T21:12:52.224 DEBUG:boto:Token: None 2023-05-24T21:12:52.225 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:12:52 GMT /qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-24T21:12:52.225 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ArfXbmIiadu1cLfI1lKemiMrfRI= 2023-05-24T21:12:52.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ArfXbmIiadu1cLfI1lKemiMrfRI=', 'Content-Length': '0'} 2023-05-24T21:12:52.227 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:12:11 GMT'), ('x-amz-version-id', 'pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f3ecb63a23b0271a-00646e7dd4-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:52.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:12:52.228 DEBUG:boto:path=/qskrps-14/obj 2023-05-24T21:12:52.229 DEBUG:boto:auth_path=/qskrps-14/obj 2023-05-24T21:12:52.229 DEBUG:boto:path=/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-24T21:12:52.229 DEBUG:boto:auth_path=/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-24T21:12:52.230 DEBUG:boto:Method: GET 2023-05-24T21:12:52.230 DEBUG:boto:Path: /qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-24T21:12:52.230 DEBUG:boto:Data: 2023-05-24T21:12:52.231 DEBUG:boto:Headers: {} 2023-05-24T21:12:52.231 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:12:52.231 DEBUG:boto:Port: 8001 2023-05-24T21:12:52.232 DEBUG:boto:Params: {} 2023-05-24T21:12:52.232 DEBUG:boto:Token: None 2023-05-24T21:12:52.233 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:12:52 GMT /qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-24T21:12:52.233 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:wsFZcAhU2JLBSCRMGFMrN1vubfI= 2023-05-24T21:12:52.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wsFZcAhU2JLBSCRMGFMrN1vubfI=', 'Content-Length': '0'} 2023-05-24T21:12:52.235 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:12:11 GMT'), ('x-amz-version-id', 'pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e2144d34bd63e8b5-00646e7dd4-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:52.236 DEBUG:boto:path=/qskrps-14/obj 2023-05-24T21:12:52.236 DEBUG:boto:auth_path=/qskrps-14/obj 2023-05-24T21:12:52.236 DEBUG:boto:path=/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-24T21:12:52.237 DEBUG:boto:auth_path=/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-24T21:12:52.237 DEBUG:boto:Method: GET 2023-05-24T21:12:52.238 DEBUG:boto:Path: /qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-24T21:12:52.238 DEBUG:boto:Data: 2023-05-24T21:12:52.238 DEBUG:boto:Headers: {} 2023-05-24T21:12:52.239 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:12:52.239 DEBUG:boto:Port: 8000 2023-05-24T21:12:52.239 DEBUG:boto:Params: {} 2023-05-24T21:12:52.240 DEBUG:boto:Token: None 2023-05-24T21:12:52.240 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:12:52 GMT /qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-24T21:12:52.240 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:wsFZcAhU2JLBSCRMGFMrN1vubfI= 2023-05-24T21:12:52.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wsFZcAhU2JLBSCRMGFMrN1vubfI=', 'Content-Length': '0'} 2023-05-24T21:12:52.242 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:12:11 GMT'), ('x-amz-version-id', 'pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004a3e8d1e784cb911-00646e7dd4-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:52.243 DEBUG:boto:path=/qskrps-14/ 2023-05-24T21:12:52.244 DEBUG:boto:auth_path=/qskrps-14/ 2023-05-24T21:12:52.244 DEBUG:boto:path=/qskrps-14/?prefix=obj 2023-05-24T21:12:52.244 DEBUG:boto:auth_path=/qskrps-14/?prefix=obj 2023-05-24T21:12:52.245 DEBUG:boto:Method: GET 2023-05-24T21:12:52.245 DEBUG:boto:Path: /qskrps-14/?prefix=obj 2023-05-24T21:12:52.246 DEBUG:boto:Data: 2023-05-24T21:12:52.246 DEBUG:boto:Headers: {} 2023-05-24T21:12:52.246 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:12:52.247 DEBUG:boto:Port: 8001 2023-05-24T21:12:52.247 DEBUG:boto:Params: {} 2023-05-24T21:12:52.248 DEBUG:boto:Token: None 2023-05-24T21:12:52.248 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:12:52 GMT /qskrps-14/ 2023-05-24T21:12:52.248 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:MX4R/2vj3cFapNs0cBkZB8d2G/E= 2023-05-24T21:12:52.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:MX4R/2vj3cFapNs0cBkZB8d2G/E=', 'Content-Length': '0'} 2023-05-24T21:12:52.251 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a952da7ad7c291c7-00646e7dd4-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:52.292 DEBUG:boto:b'qskrps-14obj1000falseobj2023-05-24T21:12:11.852Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:12:52.293 DEBUG:boto:path=/qskrps-14/ 2023-05-24T21:12:52.294 DEBUG:boto:auth_path=/qskrps-14/ 2023-05-24T21:12:52.294 DEBUG:boto:path=/qskrps-14/?prefix=obj 2023-05-24T21:12:52.295 DEBUG:boto:auth_path=/qskrps-14/?prefix=obj 2023-05-24T21:12:52.295 DEBUG:boto:Method: GET 2023-05-24T21:12:52.295 DEBUG:boto:Path: /qskrps-14/?prefix=obj 2023-05-24T21:12:52.296 DEBUG:boto:Data: 2023-05-24T21:12:52.296 DEBUG:boto:Headers: {} 2023-05-24T21:12:52.296 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:12:52.297 DEBUG:boto:Port: 8000 2023-05-24T21:12:52.297 DEBUG:boto:Params: {} 2023-05-24T21:12:52.298 DEBUG:boto:Token: None 2023-05-24T21:12:52.298 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:12:52 GMT /qskrps-14/ 2023-05-24T21:12:52.299 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:MX4R/2vj3cFapNs0cBkZB8d2G/E= 2023-05-24T21:12:52.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:MX4R/2vj3cFapNs0cBkZB8d2G/E=', 'Content-Length': '0'} 2023-05-24T21:12:52.301 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005885a6b6faf3374b-00646e7dd4-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:52.302 DEBUG:boto:b'qskrps-14obj1000falseobj2023-05-24T21:12:11.852Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:12:52.302 DEBUG:boto:path=/qskrps-14/obj 2023-05-24T21:12:52.303 DEBUG:boto:auth_path=/qskrps-14/obj 2023-05-24T21:12:52.303 DEBUG:boto:Method: HEAD 2023-05-24T21:12:52.304 DEBUG:boto:Path: /qskrps-14/obj 2023-05-24T21:12:52.304 DEBUG:boto:Data: 2023-05-24T21:12:52.304 DEBUG:boto:Headers: {} 2023-05-24T21:12:52.305 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:12:52.305 DEBUG:boto:Port: 8001 2023-05-24T21:12:52.305 DEBUG:boto:Params: {} 2023-05-24T21:12:52.306 DEBUG:boto:Token: None 2023-05-24T21:12:52.306 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:12:52 GMT /qskrps-14/obj 2023-05-24T21:12:52.306 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:u2r/WGlaLQ93C4498vxuOA54DW0= 2023-05-24T21:12:52.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:u2r/WGlaLQ93C4498vxuOA54DW0=', 'Content-Length': '0'} 2023-05-24T21:12:52.309 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:12:11 GMT'), ('x-amz-version-id', 'pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ed591ba734ec38dd-00646e7dd4-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:52.310 DEBUG:boto:path=/qskrps-14/obj 2023-05-24T21:12:52.310 DEBUG:boto:auth_path=/qskrps-14/obj 2023-05-24T21:12:52.310 DEBUG:boto:Method: HEAD 2023-05-24T21:12:52.311 DEBUG:boto:Path: /qskrps-14/obj 2023-05-24T21:12:52.311 DEBUG:boto:Data: 2023-05-24T21:12:52.311 DEBUG:boto:Headers: {} 2023-05-24T21:12:52.312 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:12:52.312 DEBUG:boto:Port: 8000 2023-05-24T21:12:52.312 DEBUG:boto:Params: {} 2023-05-24T21:12:52.313 DEBUG:boto:Token: None 2023-05-24T21:12:52.313 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:12:52 GMT /qskrps-14/obj 2023-05-24T21:12:52.313 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:u2r/WGlaLQ93C4498vxuOA54DW0= 2023-05-24T21:12:52.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:u2r/WGlaLQ93C4498vxuOA54DW0=', 'Content-Length': '0'} 2023-05-24T21:12:52.316 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:12:11 GMT'), ('x-amz-version-id', 'pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f8f7196d697433a9-00646e7dd4-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:52.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:12:52.317 DEBUG:boto:path=/qskrps-14/obj 2023-05-24T21:12:52.317 DEBUG:boto:auth_path=/qskrps-14/obj 2023-05-24T21:12:52.318 DEBUG:boto:path=/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-24T21:12:52.318 DEBUG:boto:auth_path=/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-24T21:12:52.318 DEBUG:boto:Method: GET 2023-05-24T21:12:52.319 DEBUG:boto:Path: /qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-24T21:12:52.319 DEBUG:boto:Data: 2023-05-24T21:12:52.319 DEBUG:boto:Headers: {} 2023-05-24T21:12:52.320 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:12:52.320 DEBUG:boto:Port: 8001 2023-05-24T21:12:52.320 DEBUG:boto:Params: {} 2023-05-24T21:12:52.321 DEBUG:boto:Token: None 2023-05-24T21:12:52.321 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:12:52 GMT /qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-24T21:12:52.322 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:wsFZcAhU2JLBSCRMGFMrN1vubfI= 2023-05-24T21:12:52.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wsFZcAhU2JLBSCRMGFMrN1vubfI=', 'Content-Length': '0'} 2023-05-24T21:12:52.324 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:12:11 GMT'), ('x-amz-version-id', 'pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000021e3fbdf75ca9184-00646e7dd4-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:52.324 DEBUG:boto:path=/qskrps-14/obj 2023-05-24T21:12:52.325 DEBUG:boto:auth_path=/qskrps-14/obj 2023-05-24T21:12:52.325 DEBUG:boto:path=/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-24T21:12:52.326 DEBUG:boto:auth_path=/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-24T21:12:52.326 DEBUG:boto:Method: GET 2023-05-24T21:12:52.326 DEBUG:boto:Path: /qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-24T21:12:52.327 DEBUG:boto:Data: 2023-05-24T21:12:52.327 DEBUG:boto:Headers: {} 2023-05-24T21:12:52.327 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:12:52.328 DEBUG:boto:Port: 8000 2023-05-24T21:12:52.328 DEBUG:boto:Params: {} 2023-05-24T21:12:52.328 DEBUG:boto:Token: None 2023-05-24T21:12:52.329 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:12:52 GMT /qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-24T21:12:52.329 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:wsFZcAhU2JLBSCRMGFMrN1vubfI= 2023-05-24T21:12:52.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wsFZcAhU2JLBSCRMGFMrN1vubfI=', 'Content-Length': '0'} 2023-05-24T21:12:52.332 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:12:11 GMT'), ('x-amz-version-id', 'pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009022ac476830c28e-00646e7dd4-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:52.332 DEBUG:boto:path=/qskrps-14/ 2023-05-24T21:12:52.333 DEBUG:boto:auth_path=/qskrps-14/ 2023-05-24T21:12:52.333 DEBUG:boto:path=/qskrps-14/?prefix=obj 2023-05-24T21:12:52.333 DEBUG:boto:auth_path=/qskrps-14/?prefix=obj 2023-05-24T21:12:52.334 DEBUG:boto:Method: GET 2023-05-24T21:12:52.334 DEBUG:boto:Path: /qskrps-14/?prefix=obj 2023-05-24T21:12:52.334 DEBUG:boto:Data: 2023-05-24T21:12:52.335 DEBUG:boto:Headers: {} 2023-05-24T21:12:52.335 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:12:52.335 DEBUG:boto:Port: 8001 2023-05-24T21:12:52.336 DEBUG:boto:Params: {} 2023-05-24T21:12:52.336 DEBUG:boto:Token: None 2023-05-24T21:12:52.337 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:12:52 GMT /qskrps-14/ 2023-05-24T21:12:52.337 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:MX4R/2vj3cFapNs0cBkZB8d2G/E= 2023-05-24T21:12:52.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:MX4R/2vj3cFapNs0cBkZB8d2G/E=', 'Content-Length': '0'} 2023-05-24T21:12:52.339 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fcdd6b4c65f7410-00646e7dd4-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:52.340 DEBUG:boto:b'qskrps-14obj1000falseobj2023-05-24T21:12:11.852Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:12:52.341 DEBUG:boto:path=/qskrps-14/ 2023-05-24T21:12:52.341 DEBUG:boto:auth_path=/qskrps-14/ 2023-05-24T21:12:52.341 DEBUG:boto:path=/qskrps-14/?prefix=obj 2023-05-24T21:12:52.342 DEBUG:boto:auth_path=/qskrps-14/?prefix=obj 2023-05-24T21:12:52.342 DEBUG:boto:Method: GET 2023-05-24T21:12:52.343 DEBUG:boto:Path: /qskrps-14/?prefix=obj 2023-05-24T21:12:52.343 DEBUG:boto:Data: 2023-05-24T21:12:52.343 DEBUG:boto:Headers: {} 2023-05-24T21:12:52.344 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:12:52.344 DEBUG:boto:Port: 8000 2023-05-24T21:12:52.344 DEBUG:boto:Params: {} 2023-05-24T21:12:52.345 DEBUG:boto:Token: None 2023-05-24T21:12:52.345 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:12:52 GMT /qskrps-14/ 2023-05-24T21:12:52.345 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:MX4R/2vj3cFapNs0cBkZB8d2G/E= 2023-05-24T21:12:52.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:MX4R/2vj3cFapNs0cBkZB8d2G/E=', 'Content-Length': '0'} 2023-05-24T21:12:52.348 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055089a2e7b63bac2-00646e7dd4-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:52.349 DEBUG:boto:b'qskrps-14obj1000falseobj2023-05-24T21:12:11.852Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:12:52.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:12:52.349 DEBUG:boto:path=/qskrps-14/obj 2023-05-24T21:12:52.350 DEBUG:boto:auth_path=/qskrps-14/obj 2023-05-24T21:12:52.350 DEBUG:boto:path=/qskrps-14/obj?versionId=null 2023-05-24T21:12:52.351 DEBUG:boto:auth_path=/qskrps-14/obj?versionId=null 2023-05-24T21:12:52.351 DEBUG:boto:Method: GET 2023-05-24T21:12:52.351 DEBUG:boto:Path: /qskrps-14/obj?versionId=null 2023-05-24T21:12:52.352 DEBUG:boto:Data: 2023-05-24T21:12:52.352 DEBUG:boto:Headers: {} 2023-05-24T21:12:52.352 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:12:52.353 DEBUG:boto:Port: 8001 2023-05-24T21:12:52.353 DEBUG:boto:Params: {} 2023-05-24T21:12:52.353 DEBUG:boto:Token: None 2023-05-24T21:12:52.354 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:12:52 GMT /qskrps-14/obj?versionId=null 2023-05-24T21:12:52.354 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5mIWh1Gf07Go8j8HH2/IZVW9MfM= 2023-05-24T21:12:52.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5mIWh1Gf07Go8j8HH2/IZVW9MfM=', 'Content-Length': '0'} 2023-05-24T21:12:52.356 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:11:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f8b6f315834a15f7-00646e7dd4-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:52.357 DEBUG:boto:path=/qskrps-14/obj 2023-05-24T21:12:52.357 DEBUG:boto:auth_path=/qskrps-14/obj 2023-05-24T21:12:52.358 DEBUG:boto:path=/qskrps-14/obj?versionId=null 2023-05-24T21:12:52.358 DEBUG:boto:auth_path=/qskrps-14/obj?versionId=null 2023-05-24T21:12:52.358 DEBUG:boto:Method: GET 2023-05-24T21:12:52.359 DEBUG:boto:Path: /qskrps-14/obj?versionId=null 2023-05-24T21:12:52.359 DEBUG:boto:Data: 2023-05-24T21:12:52.360 DEBUG:boto:Headers: {} 2023-05-24T21:12:52.360 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:12:52.360 DEBUG:boto:Port: 8000 2023-05-24T21:12:52.361 DEBUG:boto:Params: {} 2023-05-24T21:12:52.361 DEBUG:boto:Token: None 2023-05-24T21:12:52.361 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:12:52 GMT /qskrps-14/obj?versionId=null 2023-05-24T21:12:52.362 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5mIWh1Gf07Go8j8HH2/IZVW9MfM= 2023-05-24T21:12:52.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5mIWh1Gf07Go8j8HH2/IZVW9MfM=', 'Content-Length': '0'} 2023-05-24T21:12:52.364 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:11:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000017a3ec15b731da42-00646e7dd4-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:52.365 DEBUG:boto:path=/qskrps-14/obj 2023-05-24T21:12:52.365 DEBUG:boto:auth_path=/qskrps-14/obj 2023-05-24T21:12:52.365 DEBUG:boto:path=/qskrps-14/obj?versionId=null 2023-05-24T21:12:52.366 DEBUG:boto:auth_path=/qskrps-14/obj?versionId=null 2023-05-24T21:12:52.366 DEBUG:boto:Method: HEAD 2023-05-24T21:12:52.366 DEBUG:boto:Path: /qskrps-14/obj?versionId=null 2023-05-24T21:12:52.367 DEBUG:boto:Data: 2023-05-24T21:12:52.367 DEBUG:boto:Headers: {} 2023-05-24T21:12:52.367 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:12:52.368 DEBUG:boto:Port: 8001 2023-05-24T21:12:52.368 DEBUG:boto:Params: {} 2023-05-24T21:12:52.368 DEBUG:boto:Token: None 2023-05-24T21:12:52.369 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:12:52 GMT /qskrps-14/obj?versionId=null 2023-05-24T21:12:52.369 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:L/ZM2TaoP3N0cMpaQNyQawz+D78= 2023-05-24T21:12:52.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:L/ZM2TaoP3N0cMpaQNyQawz+D78=', 'Content-Length': '0'} 2023-05-24T21:12:52.371 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:11:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f7afac507df8d0f7-00646e7dd4-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:52.372 DEBUG:boto:path=/qskrps-14/obj 2023-05-24T21:12:52.372 DEBUG:boto:auth_path=/qskrps-14/obj 2023-05-24T21:12:52.372 DEBUG:boto:path=/qskrps-14/obj?versionId=null 2023-05-24T21:12:52.373 DEBUG:boto:auth_path=/qskrps-14/obj?versionId=null 2023-05-24T21:12:52.373 DEBUG:boto:Method: HEAD 2023-05-24T21:12:52.373 DEBUG:boto:Path: /qskrps-14/obj?versionId=null 2023-05-24T21:12:52.374 DEBUG:boto:Data: 2023-05-24T21:12:52.374 DEBUG:boto:Headers: {} 2023-05-24T21:12:52.374 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:12:52.375 DEBUG:boto:Port: 8000 2023-05-24T21:12:52.375 DEBUG:boto:Params: {} 2023-05-24T21:12:52.375 DEBUG:boto:Token: None 2023-05-24T21:12:52.376 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:12:52 GMT /qskrps-14/obj?versionId=null 2023-05-24T21:12:52.376 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:L/ZM2TaoP3N0cMpaQNyQawz+D78= 2023-05-24T21:12:52.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:L/ZM2TaoP3N0cMpaQNyQawz+D78=', 'Content-Length': '0'} 2023-05-24T21:12:52.378 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:11:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c2800c1d3f39af60-00646e7dd4-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:52.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:12:52.379 DEBUG:boto:path=/qskrps-14/obj 2023-05-24T21:12:52.380 DEBUG:boto:auth_path=/qskrps-14/obj 2023-05-24T21:12:52.380 DEBUG:boto:path=/qskrps-14/obj?versionId=null 2023-05-24T21:12:52.380 DEBUG:boto:auth_path=/qskrps-14/obj?versionId=null 2023-05-24T21:12:52.381 DEBUG:boto:Method: GET 2023-05-24T21:12:52.381 DEBUG:boto:Path: /qskrps-14/obj?versionId=null 2023-05-24T21:12:52.382 DEBUG:boto:Data: 2023-05-24T21:12:52.382 DEBUG:boto:Headers: {} 2023-05-24T21:12:52.382 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:12:52.383 DEBUG:boto:Port: 8001 2023-05-24T21:12:52.383 DEBUG:boto:Params: {} 2023-05-24T21:12:52.383 DEBUG:boto:Token: None 2023-05-24T21:12:52.384 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:12:52 GMT /qskrps-14/obj?versionId=null 2023-05-24T21:12:52.384 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5mIWh1Gf07Go8j8HH2/IZVW9MfM= 2023-05-24T21:12:52.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5mIWh1Gf07Go8j8HH2/IZVW9MfM=', 'Content-Length': '0'} 2023-05-24T21:12:52.386 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:11:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000126a7f7051ffbc8f-00646e7dd4-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:52.387 DEBUG:boto:path=/qskrps-14/obj 2023-05-24T21:12:52.387 DEBUG:boto:auth_path=/qskrps-14/obj 2023-05-24T21:12:52.388 DEBUG:boto:path=/qskrps-14/obj?versionId=null 2023-05-24T21:12:52.388 DEBUG:boto:auth_path=/qskrps-14/obj?versionId=null 2023-05-24T21:12:52.388 DEBUG:boto:Method: GET 2023-05-24T21:12:52.389 DEBUG:boto:Path: /qskrps-14/obj?versionId=null 2023-05-24T21:12:52.389 DEBUG:boto:Data: 2023-05-24T21:12:52.389 DEBUG:boto:Headers: {} 2023-05-24T21:12:52.390 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:12:52.390 DEBUG:boto:Port: 8000 2023-05-24T21:12:52.390 DEBUG:boto:Params: {} 2023-05-24T21:12:52.391 DEBUG:boto:Token: None 2023-05-24T21:12:52.391 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:12:52 GMT /qskrps-14/obj?versionId=null 2023-05-24T21:12:52.392 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5mIWh1Gf07Go8j8HH2/IZVW9MfM= 2023-05-24T21:12:52.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5mIWh1Gf07Go8j8HH2/IZVW9MfM=', 'Content-Length': '0'} 2023-05-24T21:12:52.394 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:11:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000029fd533e44d37849-00646e7dd4-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:52.394 DEBUG:boto:path=/qskrps-14/ 2023-05-24T21:12:52.395 DEBUG:boto:auth_path=/qskrps-14/ 2023-05-24T21:12:52.395 DEBUG:boto:path=/qskrps-14/?prefix=obj 2023-05-24T21:12:52.396 DEBUG:boto:auth_path=/qskrps-14/?prefix=obj 2023-05-24T21:12:52.396 DEBUG:boto:Method: GET 2023-05-24T21:12:52.396 DEBUG:boto:Path: /qskrps-14/?prefix=obj 2023-05-24T21:12:52.397 DEBUG:boto:Data: 2023-05-24T21:12:52.397 DEBUG:boto:Headers: {} 2023-05-24T21:12:52.397 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:12:52.398 DEBUG:boto:Port: 8001 2023-05-24T21:12:52.398 DEBUG:boto:Params: {} 2023-05-24T21:12:52.398 DEBUG:boto:Token: None 2023-05-24T21:12:52.399 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:12:52 GMT /qskrps-14/ 2023-05-24T21:12:52.399 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:MX4R/2vj3cFapNs0cBkZB8d2G/E= 2023-05-24T21:12:52.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:MX4R/2vj3cFapNs0cBkZB8d2G/E=', 'Content-Length': '0'} 2023-05-24T21:12:52.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015b0f6efd3848946-00646e7dd4-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:52.444 DEBUG:boto:b'qskrps-14obj1000falseobj2023-05-24T21:12:11.852Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:12:52.445 DEBUG:boto:path=/qskrps-14/ 2023-05-24T21:12:52.446 DEBUG:boto:auth_path=/qskrps-14/ 2023-05-24T21:12:52.446 DEBUG:boto:path=/qskrps-14/?prefix=obj 2023-05-24T21:12:52.446 DEBUG:boto:auth_path=/qskrps-14/?prefix=obj 2023-05-24T21:12:52.447 DEBUG:boto:Method: GET 2023-05-24T21:12:52.447 DEBUG:boto:Path: /qskrps-14/?prefix=obj 2023-05-24T21:12:52.448 DEBUG:boto:Data: 2023-05-24T21:12:52.448 DEBUG:boto:Headers: {} 2023-05-24T21:12:52.448 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:12:52.449 DEBUG:boto:Port: 8000 2023-05-24T21:12:52.449 DEBUG:boto:Params: {} 2023-05-24T21:12:52.449 DEBUG:boto:Token: None 2023-05-24T21:12:52.450 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:12:52 GMT /qskrps-14/ 2023-05-24T21:12:52.450 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:MX4R/2vj3cFapNs0cBkZB8d2G/E= 2023-05-24T21:12:52.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:MX4R/2vj3cFapNs0cBkZB8d2G/E=', 'Content-Length': '0'} 2023-05-24T21:12:52.453 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037f8ee460fe36ef1-00646e7dd4-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:52.454 DEBUG:boto:b'qskrps-14obj1000falseobj2023-05-24T21:12:11.852Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:12:52.454 DEBUG:boto:path=/qskrps-14/obj 2023-05-24T21:12:52.455 DEBUG:boto:auth_path=/qskrps-14/obj 2023-05-24T21:12:52.455 DEBUG:boto:Method: HEAD 2023-05-24T21:12:52.455 DEBUG:boto:Path: /qskrps-14/obj 2023-05-24T21:12:52.456 DEBUG:boto:Data: 2023-05-24T21:12:52.456 DEBUG:boto:Headers: {} 2023-05-24T21:12:52.456 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:12:52.457 DEBUG:boto:Port: 8001 2023-05-24T21:12:52.457 DEBUG:boto:Params: {} 2023-05-24T21:12:52.457 DEBUG:boto:Token: None 2023-05-24T21:12:52.458 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:12:52 GMT /qskrps-14/obj 2023-05-24T21:12:52.458 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:u2r/WGlaLQ93C4498vxuOA54DW0= 2023-05-24T21:12:52.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:u2r/WGlaLQ93C4498vxuOA54DW0=', 'Content-Length': '0'} 2023-05-24T21:12:52.461 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:12:11 GMT'), ('x-amz-version-id', 'pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006144a267bcf3946e-00646e7dd4-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:52.461 DEBUG:boto:path=/qskrps-14/obj 2023-05-24T21:12:52.461 DEBUG:boto:auth_path=/qskrps-14/obj 2023-05-24T21:12:52.462 DEBUG:boto:Method: HEAD 2023-05-24T21:12:52.462 DEBUG:boto:Path: /qskrps-14/obj 2023-05-24T21:12:52.463 DEBUG:boto:Data: 2023-05-24T21:12:52.463 DEBUG:boto:Headers: {} 2023-05-24T21:12:52.463 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:12:52.463 DEBUG:boto:Port: 8000 2023-05-24T21:12:52.464 DEBUG:boto:Params: {} 2023-05-24T21:12:52.464 DEBUG:boto:Token: None 2023-05-24T21:12:52.465 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:12:52 GMT /qskrps-14/obj 2023-05-24T21:12:52.465 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:u2r/WGlaLQ93C4498vxuOA54DW0= 2023-05-24T21:12:52.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:u2r/WGlaLQ93C4498vxuOA54DW0=', 'Content-Length': '0'} 2023-05-24T21:12:52.467 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:12:11 GMT'), ('x-amz-version-id', 'pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000a78f4ceec3cfd3a-00646e7dd4-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:52.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:12:52.468 DEBUG:boto:path=/qskrps-14/obj 2023-05-24T21:12:52.469 DEBUG:boto:auth_path=/qskrps-14/obj 2023-05-24T21:12:52.469 DEBUG:boto:path=/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-24T21:12:52.469 DEBUG:boto:auth_path=/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-24T21:12:52.470 DEBUG:boto:Method: GET 2023-05-24T21:12:52.470 DEBUG:boto:Path: /qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-24T21:12:52.471 DEBUG:boto:Data: 2023-05-24T21:12:52.471 DEBUG:boto:Headers: {} 2023-05-24T21:12:52.471 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:12:52.472 DEBUG:boto:Port: 8001 2023-05-24T21:12:52.472 DEBUG:boto:Params: {} 2023-05-24T21:12:52.472 DEBUG:boto:Token: None 2023-05-24T21:12:52.473 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:12:52 GMT /qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-24T21:12:52.473 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:wsFZcAhU2JLBSCRMGFMrN1vubfI= 2023-05-24T21:12:52.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wsFZcAhU2JLBSCRMGFMrN1vubfI=', 'Content-Length': '0'} 2023-05-24T21:12:52.475 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:12:11 GMT'), ('x-amz-version-id', 'pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000016204aa5eb88de54-00646e7dd4-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:52.476 DEBUG:boto:path=/qskrps-14/obj 2023-05-24T21:12:52.476 DEBUG:boto:auth_path=/qskrps-14/obj 2023-05-24T21:12:52.477 DEBUG:boto:path=/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-24T21:12:52.477 DEBUG:boto:auth_path=/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-24T21:12:52.477 DEBUG:boto:Method: GET 2023-05-24T21:12:52.478 DEBUG:boto:Path: /qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-24T21:12:52.478 DEBUG:boto:Data: 2023-05-24T21:12:52.479 DEBUG:boto:Headers: {} 2023-05-24T21:12:52.479 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:12:52.479 DEBUG:boto:Port: 8000 2023-05-24T21:12:52.480 DEBUG:boto:Params: {} 2023-05-24T21:12:52.480 DEBUG:boto:Token: None 2023-05-24T21:12:52.480 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:12:52 GMT /qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-24T21:12:52.481 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:wsFZcAhU2JLBSCRMGFMrN1vubfI= 2023-05-24T21:12:52.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wsFZcAhU2JLBSCRMGFMrN1vubfI=', 'Content-Length': '0'} 2023-05-24T21:12:52.483 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:12:11 GMT'), ('x-amz-version-id', 'pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ec8961372c7903a8-00646e7dd4-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:52.484 DEBUG:boto:path=/qskrps-14/ 2023-05-24T21:12:52.484 DEBUG:boto:auth_path=/qskrps-14/ 2023-05-24T21:12:52.485 DEBUG:boto:path=/qskrps-14/?prefix=obj 2023-05-24T21:12:52.485 DEBUG:boto:auth_path=/qskrps-14/?prefix=obj 2023-05-24T21:12:52.485 DEBUG:boto:Method: GET 2023-05-24T21:12:52.486 DEBUG:boto:Path: /qskrps-14/?prefix=obj 2023-05-24T21:12:52.486 DEBUG:boto:Data: 2023-05-24T21:12:52.486 DEBUG:boto:Headers: {} 2023-05-24T21:12:52.487 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:12:52.487 DEBUG:boto:Port: 8001 2023-05-24T21:12:52.487 DEBUG:boto:Params: {} 2023-05-24T21:12:52.488 DEBUG:boto:Token: None 2023-05-24T21:12:52.488 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:12:52 GMT /qskrps-14/ 2023-05-24T21:12:52.489 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:MX4R/2vj3cFapNs0cBkZB8d2G/E= 2023-05-24T21:12:52.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:MX4R/2vj3cFapNs0cBkZB8d2G/E=', 'Content-Length': '0'} 2023-05-24T21:12:52.491 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098dde42f3beaac93-00646e7dd4-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:52.492 DEBUG:boto:b'qskrps-14obj1000falseobj2023-05-24T21:12:11.852Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:12:52.492 DEBUG:boto:path=/qskrps-14/ 2023-05-24T21:12:52.493 DEBUG:boto:auth_path=/qskrps-14/ 2023-05-24T21:12:52.493 DEBUG:boto:path=/qskrps-14/?prefix=obj 2023-05-24T21:12:52.493 DEBUG:boto:auth_path=/qskrps-14/?prefix=obj 2023-05-24T21:12:52.494 DEBUG:boto:Method: GET 2023-05-24T21:12:52.494 DEBUG:boto:Path: /qskrps-14/?prefix=obj 2023-05-24T21:12:52.495 DEBUG:boto:Data: 2023-05-24T21:12:52.495 DEBUG:boto:Headers: {} 2023-05-24T21:12:52.495 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:12:52.496 DEBUG:boto:Port: 8000 2023-05-24T21:12:52.496 DEBUG:boto:Params: {} 2023-05-24T21:12:52.496 DEBUG:boto:Token: None 2023-05-24T21:12:52.497 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:12:52 GMT /qskrps-14/ 2023-05-24T21:12:52.497 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:MX4R/2vj3cFapNs0cBkZB8d2G/E= 2023-05-24T21:12:52.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:MX4R/2vj3cFapNs0cBkZB8d2G/E=', 'Content-Length': '0'} 2023-05-24T21:12:52.500 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067da7d2ad336a08d-00646e7dd4-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:52.500 DEBUG:boto:b'qskrps-14obj1000falseobj2023-05-24T21:12:11.852Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:12:52.501 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qskrps-14 zones={a2, a1} 2023-05-24T21:12:52.501 DEBUG:boto:path=/qskrps-14/ 2023-05-24T21:12:52.502 DEBUG:boto:auth_path=/qskrps-14/ 2023-05-24T21:12:52.502 DEBUG:boto:path=/qskrps-14/?versioning 2023-05-24T21:12:52.502 DEBUG:boto:auth_path=/qskrps-14/?versioning 2023-05-24T21:12:52.503 DEBUG:boto:Method: PUT 2023-05-24T21:12:52.503 DEBUG:boto:Path: /qskrps-14/?versioning 2023-05-24T21:12:52.503 DEBUG:boto:Data: Suspended Disabled 2023-05-24T21:12:52.504 DEBUG:boto:Headers: {} 2023-05-24T21:12:52.504 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:12:52.504 DEBUG:boto:Port: 8000 2023-05-24T21:12:52.505 DEBUG:boto:Params: {} 2023-05-24T21:12:52.505 DEBUG:boto:Token: None 2023-05-24T21:12:52.505 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:12:52 GMT /qskrps-14/?versioning 2023-05-24T21:12:52.506 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:GjPPJWLd7kDpY5V9WUOh14mt04c= 2023-05-24T21:12:52.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:GjPPJWLd7kDpY5V9WUOh14mt04c=', 'Content-Length': '230'} 2023-05-24T21:12:52.512 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007b8d5c7b8d31b52c-00646e7dd4-110d-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:52.512 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:12:52.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:12:52.513 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:12:52.568 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:12:52.569 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:12:52.612 INFO:teuthology.orchestra.run.smithi071.stderr:No --period given, using current period=5c04936b-70ea-4415-b4eb-3aad3dc612e5 2023-05-24T21:12:52.624 INFO:teuthology.orchestra.run.smithi071.stdout:[ 2023-05-24T21:12:52.625 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:12:52.625 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962772.506122_148.1", 2023-05-24T21:12:52.625 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:12:52.506122Z" 2023-05-24T21:12:52.625 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:12:52.625 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:12:52.625 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:12:52.626 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:06:32.333844Z" 2023-05-24T21:12:52.626 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:12:52.626 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:12:52.626 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:12:52.626 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:12:52.626 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:12:52.626 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:12:52.627 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:12:52.627 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:12:52.627 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:12:52.627 INFO:teuthology.orchestra.run.smithi071.stdout:] 2023-05-24T21:12:52.628 DEBUG:rgw_multi.tests:master meta markers={0: '1_1684962772.506122_148.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:12:52.628 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-24T21:12:52.629 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:12:52.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:12:52.630 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:12:52.679 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:12:52.679 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:12:52.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:12:52.704+0000 7fb3a5737780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:12:52.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:12:52.705+0000 7fb3a5737780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:12:52.726 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:12:52.726 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:12:52.726 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:12:52.727 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:12:52.727 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:12:52.727 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:12:52.727 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:12:52.727 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:12:52.727 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:12:52.728 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:12:52.728 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:12:52.728 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:12:52.728 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:12:52.728 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962772.506122_148.1", 2023-05-24T21:12:52.728 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:12:52.729 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:12:52.729 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:12:52.729 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:12:52.506122Z", 2023-05-24T21:12:52.729 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:12:52.729 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:12:52.729 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:12:52.729 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:12:52.730 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:12:52.730 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:12:52.730 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:12:52.730 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:12:52.730 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:12:52.730 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:12:52.731 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:12:52.731 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:12:52.731 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:12:52.731 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:12:52.731 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:12:52.731 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:12:52.732 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:12:52.732 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:12:52.732 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:12:52.732 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:12:52.732 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:12:52.732 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:12:52.732 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:12:52.733 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:12:52.733 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:12:52.733 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:12:52.733 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:12:52.733 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:12:52.734 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:12:52.734 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:12:52.734 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:12:52.734 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:12:52.734 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:12:52.734 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:12:52.734 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:12:52.735 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:12:52.735 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:12:52.735 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:12:52.735 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:12:52.735 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:12:52.735 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:12:52.735 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:12:52.736 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:12:52.736 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:12:52.736 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:12:52.736 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:12:52Z" 2023-05-24T21:12:52.736 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:12:52.737 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962772.506122_148.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:12:52.506122Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:12:52Z" } 2023-05-24T21:12:52.738 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962772.506122_148.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:12:52.506122Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:12:52.738 DEBUG:rgw_multi.tests:log_status={0: '1_1684962772.506122_148.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:12:52.739 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962772.506122_148.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:12:52.739 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-24T21:12:52.740 DEBUG:boto:path=/qskrps-14/ 2023-05-24T21:12:52.740 DEBUG:boto:auth_path=/qskrps-14/ 2023-05-24T21:12:52.740 DEBUG:boto:Method: HEAD 2023-05-24T21:12:52.741 DEBUG:boto:Path: /qskrps-14/ 2023-05-24T21:12:52.741 DEBUG:boto:Data: 2023-05-24T21:12:52.742 DEBUG:boto:Headers: {} 2023-05-24T21:12:52.742 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:12:52.742 DEBUG:boto:Port: 8000 2023-05-24T21:12:52.743 DEBUG:boto:Params: {} 2023-05-24T21:12:52.743 DEBUG:boto:Token: None 2023-05-24T21:12:52.744 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:12:52 GMT /qskrps-14/ 2023-05-24T21:12:52.744 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:0Cy+rIDC9q1jv0yqhIaIOr9Zt7A= 2023-05-24T21:12:52.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:0Cy+rIDC9q1jv0yqhIaIOr9Zt7A=', 'Content-Length': '0'} 2023-05-24T21:12:52.747 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000054d4ce6feb2859fc-00646e7dd4-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:52.747 DEBUG:boto:path=/qskrps-14/obj 2023-05-24T21:12:52.748 DEBUG:boto:auth_path=/qskrps-14/obj 2023-05-24T21:12:52.748 DEBUG:boto:Method: PUT 2023-05-24T21:12:52.749 DEBUG:boto:Path: /qskrps-14/obj 2023-05-24T21:12:52.749 DEBUG:boto:Data: 2023-05-24T21:12:52.749 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-05-24T21:12:52.750 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:12:52.750 DEBUG:boto:Port: 8000 2023-05-24T21:12:52.750 DEBUG:boto:Params: {} 2023-05-24T21:12:52.751 DEBUG:boto:Token: None 2023-05-24T21:12:52.751 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Wed, 24 May 2023 21:12:52 GMT /qskrps-14/obj 2023-05-24T21:12:52.751 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5LN+6rfDBoXovo1GkJhlcj/BHhk= 2023-05-24T21:12:52.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5LN+6rfDBoXovo1GkJhlcj/BHhk='} 2023-05-24T21:12:52.767 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000051e347def97ed52-00646e7dd4-110d-a1'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:12:52.768 DEBUG:rgw_multi.tests:created null version id=None 2023-05-24T21:12:52.768 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=qskrps-14 2023-05-24T21:12:52.769 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:12:52.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:12:52.770 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-14 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:12:52.825 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:12:52.826 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:12:52.872 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:12:52.866+0000 7f3ad87e1780 1 waiting for incremental sync to catch up: 2023-05-24T21:12:52.873 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000003.9.13, , ] 2023-05-24T21:12:52.873 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000005.13.18, , ] 2023-05-24T21:12:57.873 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:12:57.868+0000 7f3ad87e1780 1 waiting for incremental sync to catch up: 2023-05-24T21:12:57.874 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000003.9.13, , ] 2023-05-24T21:12:57.874 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000005.13.18, , ] 2023-05-24T21:13:02.875 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:13:02.869+0000 7f3ad87e1780 1 waiting for incremental sync to catch up: 2023-05-24T21:13:02.875 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000003.9.13, , ] 2023-05-24T21:13:02.875 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000005.13.18, , ] 2023-05-24T21:13:07.876 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:13:07.870+0000 7f3ad87e1780 1 waiting for incremental sync to catch up: 2023-05-24T21:13:07.876 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000003.9.13, , ] 2023-05-24T21:13:07.877 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000005.13.18, , ] 2023-05-24T21:13:12.877 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:13:12.871+0000 7f3ad87e1780 1 bucket sync caught up with source: 2023-05-24T21:13:12.878 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000005.13.18, , ] 2023-05-24T21:13:12.878 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000005.13.18, , ] 2023-05-24T21:13:12.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:13:12.871+0000 7f3ad87e1780 0 bucket checkpoint complete 2023-05-24T21:13:12.894 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=qskrps-14 2023-05-24T21:13:12.895 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:13:12.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:13:12.896 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:13:12.958 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:13:12.958 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:13:13.008 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:13:13.002+0000 7f1121e81780 1 bucket sync caught up with source: 2023-05-24T21:13:13.008 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000005.25.18, , ] 2023-05-24T21:13:13.009 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000005.25.18, , ] 2023-05-24T21:13:13.009 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:13:13.002+0000 7f1121e81780 0 bucket checkpoint complete 2023-05-24T21:13:13.021 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qskrps-14 zones={a2, a1} 2023-05-24T21:13:13.022 DEBUG:boto:path=/qskrps-14/ 2023-05-24T21:13:13.022 DEBUG:boto:auth_path=/qskrps-14/ 2023-05-24T21:13:13.023 DEBUG:boto:Method: HEAD 2023-05-24T21:13:13.023 DEBUG:boto:Path: /qskrps-14/ 2023-05-24T21:13:13.023 DEBUG:boto:Data: 2023-05-24T21:13:13.024 DEBUG:boto:Headers: {} 2023-05-24T21:13:13.024 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:13:13.025 DEBUG:boto:Port: 8001 2023-05-24T21:13:13.025 DEBUG:boto:Params: {} 2023-05-24T21:13:13.026 DEBUG:boto:Token: None 2023-05-24T21:13:13.026 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:13:13 GMT /qskrps-14/ 2023-05-24T21:13:13.026 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:+G9FIAuwt8Cy3G5h79fJJ8pZS4Q= 2023-05-24T21:13:13.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:13 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:+G9FIAuwt8Cy3G5h79fJJ8pZS4Q=', 'Content-Length': '0'} 2023-05-24T21:13:13.029 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000106d672a3b6aa2a3-00646e7de9-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:13:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:13.030 DEBUG:boto:path=/qskrps-14/ 2023-05-24T21:13:13.030 DEBUG:boto:auth_path=/qskrps-14/ 2023-05-24T21:13:13.030 DEBUG:boto:Method: HEAD 2023-05-24T21:13:13.031 DEBUG:boto:Path: /qskrps-14/ 2023-05-24T21:13:13.031 DEBUG:boto:Data: 2023-05-24T21:13:13.032 DEBUG:boto:Headers: {} 2023-05-24T21:13:13.032 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:13:13.032 DEBUG:boto:Port: 8000 2023-05-24T21:13:13.033 DEBUG:boto:Params: {} 2023-05-24T21:13:13.033 DEBUG:boto:Token: None 2023-05-24T21:13:13.034 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:13:13 GMT /qskrps-14/ 2023-05-24T21:13:13.034 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:+G9FIAuwt8Cy3G5h79fJJ8pZS4Q= 2023-05-24T21:13:13.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:13 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:+G9FIAuwt8Cy3G5h79fJJ8pZS4Q=', 'Content-Length': '0'} 2023-05-24T21:13:13.036 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000054e055d6d4c732ce-00646e7de9-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:13:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:13.037 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-24T21:13:13.037 DEBUG:boto:path=/qskrps-14/ 2023-05-24T21:13:13.038 DEBUG:boto:auth_path=/qskrps-14/ 2023-05-24T21:13:13.038 DEBUG:boto:path=/qskrps-14/?versions&max-keys=999 2023-05-24T21:13:13.038 DEBUG:boto:auth_path=/qskrps-14/?versions&max-keys=999 2023-05-24T21:13:13.039 DEBUG:boto:Method: GET 2023-05-24T21:13:13.039 DEBUG:boto:Path: /qskrps-14/?versions&max-keys=999 2023-05-24T21:13:13.039 DEBUG:boto:Data: 2023-05-24T21:13:13.040 DEBUG:boto:Headers: {} 2023-05-24T21:13:13.040 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:13:13.041 DEBUG:boto:Port: 8001 2023-05-24T21:13:13.041 DEBUG:boto:Params: {} 2023-05-24T21:13:13.041 DEBUG:boto:Token: None 2023-05-24T21:13:13.042 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:13:13 GMT /qskrps-14/?versions 2023-05-24T21:13:13.042 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:7Wr1QbYXljXJIwCH3+iek4lhITc= 2023-05-24T21:13:13.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:13 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:7Wr1QbYXljXJIwCH3+iek4lhITc=', 'Content-Length': '0'} 2023-05-24T21:13:13.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f40f2d14507851e7-00646e7de9-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:13:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:13.088 DEBUG:boto:b'qskrps-14999falseobjnulltrue2023-05-24T21:12:52.749Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpgpY3GcrHqwBxmj52IgrlMvqJeqfWWQfalse2023-05-24T21:12:11.852Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:13:13.089 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:13:13.090 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:13:13.090 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-24T21:13:13.090 DEBUG:boto:path=/qskrps-14/ 2023-05-24T21:13:13.091 DEBUG:boto:auth_path=/qskrps-14/ 2023-05-24T21:13:13.091 DEBUG:boto:path=/qskrps-14/?versions&max-keys=999 2023-05-24T21:13:13.091 DEBUG:boto:auth_path=/qskrps-14/?versions&max-keys=999 2023-05-24T21:13:13.092 DEBUG:boto:Method: GET 2023-05-24T21:13:13.092 DEBUG:boto:Path: /qskrps-14/?versions&max-keys=999 2023-05-24T21:13:13.093 DEBUG:boto:Data: 2023-05-24T21:13:13.093 DEBUG:boto:Headers: {} 2023-05-24T21:13:13.093 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:13:13.094 DEBUG:boto:Port: 8000 2023-05-24T21:13:13.094 DEBUG:boto:Params: {} 2023-05-24T21:13:13.094 DEBUG:boto:Token: None 2023-05-24T21:13:13.095 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:13:13 GMT /qskrps-14/?versions 2023-05-24T21:13:13.095 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:7Wr1QbYXljXJIwCH3+iek4lhITc= 2023-05-24T21:13:13.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:13 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:7Wr1QbYXljXJIwCH3+iek4lhITc=', 'Content-Length': '0'} 2023-05-24T21:13:13.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065da53aa29eddddb-00646e7de9-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:13:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:13.099 DEBUG:boto:b'qskrps-14999falseobjnulltrue2023-05-24T21:12:52.749Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpgpY3GcrHqwBxmj52IgrlMvqJeqfWWQfalse2023-05-24T21:12:11.852Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:13:13.099 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:13:13.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:13:13.100 DEBUG:boto:path=/qskrps-14/ 2023-05-24T21:13:13.100 DEBUG:boto:auth_path=/qskrps-14/ 2023-05-24T21:13:13.101 DEBUG:boto:path=/qskrps-14/?versions&max-keys=999 2023-05-24T21:13:13.101 DEBUG:boto:auth_path=/qskrps-14/?versions&max-keys=999 2023-05-24T21:13:13.102 DEBUG:boto:Method: GET 2023-05-24T21:13:13.102 DEBUG:boto:Path: /qskrps-14/?versions&max-keys=999 2023-05-24T21:13:13.102 DEBUG:boto:Data: 2023-05-24T21:13:13.103 DEBUG:boto:Headers: {} 2023-05-24T21:13:13.103 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:13:13.103 DEBUG:boto:Port: 8001 2023-05-24T21:13:13.104 DEBUG:boto:Params: {} 2023-05-24T21:13:13.104 DEBUG:boto:Token: None 2023-05-24T21:13:13.105 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:13:13 GMT /qskrps-14/?versions 2023-05-24T21:13:13.105 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:7Wr1QbYXljXJIwCH3+iek4lhITc= 2023-05-24T21:13:13.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:13 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:7Wr1QbYXljXJIwCH3+iek4lhITc=', 'Content-Length': '0'} 2023-05-24T21:13:13.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abd29942ef5053cd-00646e7de9-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:13:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:13.108 DEBUG:boto:b'qskrps-14999falseobjnulltrue2023-05-24T21:12:52.749Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpgpY3GcrHqwBxmj52IgrlMvqJeqfWWQfalse2023-05-24T21:12:11.852Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:13:13.109 DEBUG:boto:path=/qskrps-14/ 2023-05-24T21:13:13.109 DEBUG:boto:auth_path=/qskrps-14/ 2023-05-24T21:13:13.110 DEBUG:boto:path=/qskrps-14/?versions&max-keys=999 2023-05-24T21:13:13.110 DEBUG:boto:auth_path=/qskrps-14/?versions&max-keys=999 2023-05-24T21:13:13.110 DEBUG:boto:Method: GET 2023-05-24T21:13:13.111 DEBUG:boto:Path: /qskrps-14/?versions&max-keys=999 2023-05-24T21:13:13.111 DEBUG:boto:Data: 2023-05-24T21:13:13.112 DEBUG:boto:Headers: {} 2023-05-24T21:13:13.112 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:13:13.113 DEBUG:boto:Port: 8000 2023-05-24T21:13:13.113 DEBUG:boto:Params: {} 2023-05-24T21:13:13.113 DEBUG:boto:Token: None 2023-05-24T21:13:13.114 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:13:13 GMT /qskrps-14/?versions 2023-05-24T21:13:13.114 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:7Wr1QbYXljXJIwCH3+iek4lhITc= 2023-05-24T21:13:13.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:13 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:7Wr1QbYXljXJIwCH3+iek4lhITc=', 'Content-Length': '0'} 2023-05-24T21:13:13.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbe3d0766389b3ab-00646e7de9-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:13:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:13.118 DEBUG:boto:b'qskrps-14999falseobjnulltrue2023-05-24T21:12:52.749Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpgpY3GcrHqwBxmj52IgrlMvqJeqfWWQfalse2023-05-24T21:12:11.852Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:13:13.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:13:13.120 DEBUG:boto:path=/qskrps-14/obj 2023-05-24T21:13:13.121 DEBUG:boto:auth_path=/qskrps-14/obj 2023-05-24T21:13:13.122 DEBUG:boto:path=/qskrps-14/obj?versionId=null 2023-05-24T21:13:13.123 DEBUG:boto:auth_path=/qskrps-14/obj?versionId=null 2023-05-24T21:13:13.124 DEBUG:boto:Method: GET 2023-05-24T21:13:13.125 DEBUG:boto:Path: /qskrps-14/obj?versionId=null 2023-05-24T21:13:13.125 DEBUG:boto:Data: 2023-05-24T21:13:13.126 DEBUG:boto:Headers: {} 2023-05-24T21:13:13.126 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:13:13.127 DEBUG:boto:Port: 8001 2023-05-24T21:13:13.128 DEBUG:boto:Params: {} 2023-05-24T21:13:13.129 DEBUG:boto:Token: None 2023-05-24T21:13:13.129 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:13:13 GMT /qskrps-14/obj?versionId=null 2023-05-24T21:13:13.130 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:geJrMH2kqjS8vWuG3CWZnSBL9No= 2023-05-24T21:13:13.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:13 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:geJrMH2kqjS8vWuG3CWZnSBL9No=', 'Content-Length': '0'} 2023-05-24T21:13:13.132 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:11:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f40d4ef35b43395b-00646e7de9-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:13:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:13.133 DEBUG:boto:path=/qskrps-14/obj 2023-05-24T21:13:13.133 DEBUG:boto:auth_path=/qskrps-14/obj 2023-05-24T21:13:13.134 DEBUG:boto:path=/qskrps-14/obj?versionId=null 2023-05-24T21:13:13.134 DEBUG:boto:auth_path=/qskrps-14/obj?versionId=null 2023-05-24T21:13:13.134 DEBUG:boto:Method: GET 2023-05-24T21:13:13.135 DEBUG:boto:Path: /qskrps-14/obj?versionId=null 2023-05-24T21:13:13.135 DEBUG:boto:Data: 2023-05-24T21:13:13.135 DEBUG:boto:Headers: {} 2023-05-24T21:13:13.136 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:13:13.136 DEBUG:boto:Port: 8000 2023-05-24T21:13:13.136 DEBUG:boto:Params: {} 2023-05-24T21:13:13.137 DEBUG:boto:Token: None 2023-05-24T21:13:13.139 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:13:13 GMT /qskrps-14/obj?versionId=null 2023-05-24T21:13:13.140 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:geJrMH2kqjS8vWuG3CWZnSBL9No= 2023-05-24T21:13:13.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:13 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:geJrMH2kqjS8vWuG3CWZnSBL9No=', 'Content-Length': '0'} 2023-05-24T21:13:13.142 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:12:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000008ac4d1dad44ad92-00646e7de9-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:13:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:13.143 DEBUG:boto:path=/qskrps-14/obj 2023-05-24T21:13:13.143 DEBUG:boto:auth_path=/qskrps-14/obj 2023-05-24T21:13:13.144 DEBUG:boto:path=/qskrps-14/obj?versionId=null 2023-05-24T21:13:13.144 DEBUG:boto:auth_path=/qskrps-14/obj?versionId=null 2023-05-24T21:13:13.144 DEBUG:boto:Method: HEAD 2023-05-24T21:13:13.145 DEBUG:boto:Path: /qskrps-14/obj?versionId=null 2023-05-24T21:13:13.145 DEBUG:boto:Data: 2023-05-24T21:13:13.145 DEBUG:boto:Headers: {} 2023-05-24T21:13:13.146 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:13:13.146 DEBUG:boto:Port: 8001 2023-05-24T21:13:13.146 DEBUG:boto:Params: {} 2023-05-24T21:13:13.147 DEBUG:boto:Token: None 2023-05-24T21:13:13.147 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:13:13 GMT /qskrps-14/obj?versionId=null 2023-05-24T21:13:13.148 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:CKNiyvqSSzAv/wUysbKBZN47cdg= 2023-05-24T21:13:13.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:13 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:CKNiyvqSSzAv/wUysbKBZN47cdg=', 'Content-Length': '0'} 2023-05-24T21:13:13.150 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:11:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001f0adb49075a613d-00646e7de9-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:13:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:13.151 DEBUG:boto:path=/qskrps-14/obj 2023-05-24T21:13:13.152 DEBUG:boto:auth_path=/qskrps-14/obj 2023-05-24T21:13:13.152 DEBUG:boto:path=/qskrps-14/obj?versionId=null 2023-05-24T21:13:13.153 DEBUG:boto:auth_path=/qskrps-14/obj?versionId=null 2023-05-24T21:13:13.153 DEBUG:boto:Method: HEAD 2023-05-24T21:13:13.154 DEBUG:boto:Path: /qskrps-14/obj?versionId=null 2023-05-24T21:13:13.155 DEBUG:boto:Data: 2023-05-24T21:13:13.155 DEBUG:boto:Headers: {} 2023-05-24T21:13:13.155 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:13:13.156 DEBUG:boto:Port: 8000 2023-05-24T21:13:13.156 DEBUG:boto:Params: {} 2023-05-24T21:13:13.157 DEBUG:boto:Token: None 2023-05-24T21:13:13.157 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:13:13 GMT /qskrps-14/obj?versionId=null 2023-05-24T21:13:13.157 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:CKNiyvqSSzAv/wUysbKBZN47cdg= 2023-05-24T21:13:13.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:13 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:CKNiyvqSSzAv/wUysbKBZN47cdg=', 'Content-Length': '0'} 2023-05-24T21:13:13.160 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:12:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000fdf299548c9d7a5-00646e7de9-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:13:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:13.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:13:13.162 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_version_suspended_incremental_sync ... FAIL 2023-05-24T21:13:13.163 INFO:rgw_multi.tests:create bucket zone=a1 name=qskrps-15 2023-05-24T21:13:13.164 DEBUG:boto:path=/qskrps-15/ 2023-05-24T21:13:13.164 DEBUG:boto:auth_path=/qskrps-15/ 2023-05-24T21:13:13.164 DEBUG:boto:Method: PUT 2023-05-24T21:13:13.165 DEBUG:boto:Path: /qskrps-15/ 2023-05-24T21:13:13.165 DEBUG:boto:Data: 2023-05-24T21:13:13.165 DEBUG:boto:Headers: {} 2023-05-24T21:13:13.166 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:13:13.167 DEBUG:boto:Port: 8000 2023-05-24T21:13:13.167 DEBUG:boto:Params: {} 2023-05-24T21:13:13.168 DEBUG:boto:Token: None 2023-05-24T21:13:13.169 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:13:13 GMT /qskrps-15/ 2023-05-24T21:13:13.170 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Jnsq7YA5IdjVDlNPxva2coaxMnQ= 2023-05-24T21:13:13.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:13 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Jnsq7YA5IdjVDlNPxva2coaxMnQ=', 'Content-Length': '0'} 2023-05-24T21:13:13.184 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000887aaa24b8883395-00646e7de9-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:13:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:13.185 INFO:rgw_multi.tests:create bucket zone=a2 name=qskrps-16 2023-05-24T21:13:13.185 DEBUG:boto:path=/qskrps-16/ 2023-05-24T21:13:13.185 DEBUG:boto:auth_path=/qskrps-16/ 2023-05-24T21:13:13.186 DEBUG:boto:Method: PUT 2023-05-24T21:13:13.186 DEBUG:boto:Path: /qskrps-16/ 2023-05-24T21:13:13.186 DEBUG:boto:Data: 2023-05-24T21:13:13.187 DEBUG:boto:Headers: {} 2023-05-24T21:13:13.187 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:13:13.187 DEBUG:boto:Port: 8001 2023-05-24T21:13:13.188 DEBUG:boto:Params: {} 2023-05-24T21:13:13.188 DEBUG:boto:Token: None 2023-05-24T21:13:13.188 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:13:13 GMT /qskrps-16/ 2023-05-24T21:13:13.189 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:sdemf8EAJZBfINyAHI97g6DLtOU= 2023-05-24T21:13:13.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:13 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:sdemf8EAJZBfINyAHI97g6DLtOU=', 'Content-Length': '0'} 2023-05-24T21:13:13.219 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a643366b7449754e-00646e7de9-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:13:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:13.220 DEBUG:boto:path=/qskrps-15/ 2023-05-24T21:13:13.221 DEBUG:boto:auth_path=/qskrps-15/ 2023-05-24T21:13:13.221 DEBUG:boto:path=/qskrps-15/?versioning 2023-05-24T21:13:13.221 DEBUG:boto:auth_path=/qskrps-15/?versioning 2023-05-24T21:13:13.222 DEBUG:boto:Method: PUT 2023-05-24T21:13:13.222 DEBUG:boto:Path: /qskrps-15/?versioning 2023-05-24T21:13:13.222 DEBUG:boto:Data: Enabled Disabled 2023-05-24T21:13:13.223 DEBUG:boto:Headers: {} 2023-05-24T21:13:13.223 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:13:13.223 DEBUG:boto:Port: 8000 2023-05-24T21:13:13.223 DEBUG:boto:Params: {} 2023-05-24T21:13:13.224 DEBUG:boto:Token: None 2023-05-24T21:13:13.224 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:13:13 GMT /qskrps-15/?versioning 2023-05-24T21:13:13.225 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:wSYOU1Gcb6+K47FWgTytdWwgx0I= 2023-05-24T21:13:13.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:13 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wSYOU1Gcb6+K47FWgTytdWwgx0I=', 'Content-Length': '228'} 2023-05-24T21:13:13.230 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a548fd0b20987ce8-00646e7de9-110d-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:13:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:13.231 DEBUG:boto:path=/qskrps-16/ 2023-05-24T21:13:13.231 DEBUG:boto:auth_path=/qskrps-16/ 2023-05-24T21:13:13.232 DEBUG:boto:path=/qskrps-16/?versioning 2023-05-24T21:13:13.232 DEBUG:boto:auth_path=/qskrps-16/?versioning 2023-05-24T21:13:13.232 DEBUG:boto:Method: PUT 2023-05-24T21:13:13.233 DEBUG:boto:Path: /qskrps-16/?versioning 2023-05-24T21:13:13.233 DEBUG:boto:Data: Enabled Disabled 2023-05-24T21:13:13.234 DEBUG:boto:Headers: {} 2023-05-24T21:13:13.235 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:13:13.235 DEBUG:boto:Port: 8001 2023-05-24T21:13:13.236 DEBUG:boto:Params: {} 2023-05-24T21:13:13.237 DEBUG:boto:Token: None 2023-05-24T21:13:13.238 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:13:13 GMT /qskrps-16/?versioning 2023-05-24T21:13:13.239 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5OAaPAbs8rL8ho/Y2OaROsq8oJE= 2023-05-24T21:13:13.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:13 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5OAaPAbs8rL8ho/Y2OaROsq8oJE=', 'Content-Length': '228'} 2023-05-24T21:13:13.247 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b4a6111b32c4caf0-00646e7de9-1111-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:13:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:13.248 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:13:13.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:13:13.249 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:13:13.298 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:13:13.298 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:13:13.334 INFO:teuthology.orchestra.run.smithi071.stderr:No --period given, using current period=5c04936b-70ea-4415-b4eb-3aad3dc612e5 2023-05-24T21:13:13.347 INFO:teuthology.orchestra.run.smithi071.stdout:[ 2023-05-24T21:13:13.348 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:13:13.348 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962793.239814_160.1", 2023-05-24T21:13:13.348 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:13:13.239814Z" 2023-05-24T21:13:13.348 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:13:13.348 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:13:13.349 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:13:13.349 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:06:32.333844Z" 2023-05-24T21:13:13.349 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:13:13.349 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:13:13.349 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:13:13.349 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:13:13.349 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:13:13.350 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:13:13.350 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:13:13.350 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:13:13.350 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:13:13.350 INFO:teuthology.orchestra.run.smithi071.stdout:] 2023-05-24T21:13:13.351 DEBUG:rgw_multi.tests:master meta markers={0: '1_1684962793.239814_160.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:13:13.352 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-24T21:13:13.352 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:13:13.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:13:13.353 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:13:13.406 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:13:13.406 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:13:13.460 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:13:13.455+0000 7f305b23f780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:13:13.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:13:13.455+0000 7f305b23f780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:13:13.481 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:13:13.481 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:13:13.481 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:13:13.482 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:13:13.482 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:13:13.483 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:13:13.483 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:13:13.483 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:13:13.483 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:13:13.483 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:13:13.483 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:13:13.483 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:13:13.484 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:13:13.484 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962793.239814_160.1", 2023-05-24T21:13:13.484 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:13:13.485 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:13:13.485 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:13:13.485 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:13:13.239814Z", 2023-05-24T21:13:13.486 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:13:13.486 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:13:13.486 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:13:13.486 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:13:13.486 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:13:13.486 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:13:13.486 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:13:13.487 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:13:13.487 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:13:13.487 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:13:13.487 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:13:13.487 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:13:13.487 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:13:13.488 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:13:13.488 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:13:13.488 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:13:13.488 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:13:13.488 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:13:13.488 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:13:13.488 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:13:13.489 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:13:13.489 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:13:13.489 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:13:13.489 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:13:13.489 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:13:13.489 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:13:13.489 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:13:13.490 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:13:13.490 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:13:13.490 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:13:13.490 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:13:13.490 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:13:13.490 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:13:13.490 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:13:13.491 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:13:13.491 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:13:13.491 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:13:13.491 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:13:13.491 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:13:13.491 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:13:13.492 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:13:13.492 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:13:13.492 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:13:13.492 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:13:13.492 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:13:13.492 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:13:13Z" 2023-05-24T21:13:13.492 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:13:13.493 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962793.239814_160.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:13:13.239814Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:13:13Z" } 2023-05-24T21:13:13.494 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962793.239814_160.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:13:13.239814Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:13:13.494 DEBUG:rgw_multi.tests:log_status={0: '1_1684962793.239814_160.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:13:13.495 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962793.239814_160.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:13:13.495 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-24T21:13:13.497 DEBUG:boto:path=/qskrps-15/ 2023-05-24T21:13:13.498 DEBUG:boto:auth_path=/qskrps-15/ 2023-05-24T21:13:13.499 DEBUG:boto:Method: HEAD 2023-05-24T21:13:13.500 DEBUG:boto:Path: /qskrps-15/ 2023-05-24T21:13:13.501 DEBUG:boto:Data: 2023-05-24T21:13:13.502 DEBUG:boto:Headers: {} 2023-05-24T21:13:13.502 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:13:13.503 DEBUG:boto:Port: 8000 2023-05-24T21:13:13.503 DEBUG:boto:Params: {} 2023-05-24T21:13:13.504 DEBUG:boto:Token: None 2023-05-24T21:13:13.504 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:13:13 GMT /qskrps-15/ 2023-05-24T21:13:13.504 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:VVmV7D4vmWTe11wvSNhrd/FMTLU= 2023-05-24T21:13:13.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:13 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:VVmV7D4vmWTe11wvSNhrd/FMTLU=', 'Content-Length': '0'} 2023-05-24T21:13:13.507 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002500e03c25002e98-00646e7de9-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:13:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:13.508 DEBUG:boto:path=/qskrps-15/obj 2023-05-24T21:13:13.508 DEBUG:boto:auth_path=/qskrps-15/obj 2023-05-24T21:13:13.509 DEBUG:boto:Method: PUT 2023-05-24T21:13:13.509 DEBUG:boto:Path: /qskrps-15/obj 2023-05-24T21:13:13.509 DEBUG:boto:Data: 2023-05-24T21:13:13.510 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-05-24T21:13:13.510 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:13:13.511 DEBUG:boto:Port: 8000 2023-05-24T21:13:13.511 DEBUG:boto:Params: {} 2023-05-24T21:13:13.511 DEBUG:boto:Token: None 2023-05-24T21:13:13.512 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Wed, 24 May 2023 21:13:13 GMT /qskrps-15/obj 2023-05-24T21:13:13.512 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:2goaeVrb0dEWkrNqV4S7FfaZY1s= 2023-05-24T21:13:13.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:13:13 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:2goaeVrb0dEWkrNqV4S7FfaZY1s='} 2023-05-24T21:13:13.531 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'xW.Bqf-y7QqZiapDjLACMtTI-TEWX9q'), ('x-amz-request-id', 'tx0000051d42f353b7f607f-00646e7de9-110d-a1'), ('Date', 'Wed, 24 May 2023 21:13:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:13.532 DEBUG:boto:path=/qskrps-15/ 2023-05-24T21:13:13.533 DEBUG:boto:auth_path=/qskrps-15/ 2023-05-24T21:13:13.533 DEBUG:boto:Method: HEAD 2023-05-24T21:13:13.533 DEBUG:boto:Path: /qskrps-15/ 2023-05-24T21:13:13.534 DEBUG:boto:Data: 2023-05-24T21:13:13.534 DEBUG:boto:Headers: {} 2023-05-24T21:13:13.535 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:13:13.535 DEBUG:boto:Port: 8000 2023-05-24T21:13:13.535 DEBUG:boto:Params: {} 2023-05-24T21:13:13.536 DEBUG:boto:Token: None 2023-05-24T21:13:13.537 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:13:13 GMT /qskrps-15/ 2023-05-24T21:13:13.537 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:VVmV7D4vmWTe11wvSNhrd/FMTLU= 2023-05-24T21:13:13.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:13 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:VVmV7D4vmWTe11wvSNhrd/FMTLU=', 'Content-Length': '0'} 2023-05-24T21:13:13.540 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004079074a91b883c3-00646e7de9-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:13:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:13.540 DEBUG:boto:path=/qskrps-15/obj 2023-05-24T21:13:13.541 DEBUG:boto:auth_path=/qskrps-15/obj 2023-05-24T21:13:13.541 DEBUG:boto:Method: DELETE 2023-05-24T21:13:13.542 DEBUG:boto:Path: /qskrps-15/obj 2023-05-24T21:13:13.542 DEBUG:boto:Data: 2023-05-24T21:13:13.542 DEBUG:boto:Headers: {} 2023-05-24T21:13:13.543 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:13:13.544 DEBUG:boto:Port: 8000 2023-05-24T21:13:13.544 DEBUG:boto:Params: {} 2023-05-24T21:13:13.545 DEBUG:boto:Token: None 2023-05-24T21:13:13.546 DEBUG:boto:StringToSign: DELETE Wed, 24 May 2023 21:13:13 GMT /qskrps-15/obj 2023-05-24T21:13:13.547 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:UC9ytZZEfiulPWcjTnKZpL41bv4= 2023-05-24T21:13:13.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:13 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:UC9ytZZEfiulPWcjTnKZpL41bv4=', 'Content-Length': '0'} 2023-05-24T21:13:13.553 DEBUG:boto:Response headers: [('x-amz-version-id', 'WxMuLjogdZXKjqgBmDHHurL1F-dBcaS'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx000008a0b451d2ba22a85-00646e7de9-110d-a1'), ('Date', 'Wed, 24 May 2023 21:13:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:13.555 DEBUG:boto:path=/qskrps-16/ 2023-05-24T21:13:13.555 DEBUG:boto:auth_path=/qskrps-16/ 2023-05-24T21:13:13.556 DEBUG:boto:Method: HEAD 2023-05-24T21:13:13.557 DEBUG:boto:Path: /qskrps-16/ 2023-05-24T21:13:13.558 DEBUG:boto:Data: 2023-05-24T21:13:13.558 DEBUG:boto:Headers: {} 2023-05-24T21:13:13.559 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:13:13.559 DEBUG:boto:Port: 8001 2023-05-24T21:13:13.559 DEBUG:boto:Params: {} 2023-05-24T21:13:13.560 DEBUG:boto:Token: None 2023-05-24T21:13:13.561 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:13:13 GMT /qskrps-16/ 2023-05-24T21:13:13.561 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:KyVlZQ6jxDD7AgZ1AvpNXc2Mvj0= 2023-05-24T21:13:13.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:13 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:KyVlZQ6jxDD7AgZ1AvpNXc2Mvj0=', 'Content-Length': '0'} 2023-05-24T21:13:13.564 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d7695f5b3044d065-00646e7de9-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:13:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:13.565 DEBUG:boto:path=/qskrps-16/obj 2023-05-24T21:13:13.566 DEBUG:boto:auth_path=/qskrps-16/obj 2023-05-24T21:13:13.567 DEBUG:boto:Method: PUT 2023-05-24T21:13:13.568 DEBUG:boto:Path: /qskrps-16/obj 2023-05-24T21:13:13.568 DEBUG:boto:Data: 2023-05-24T21:13:13.569 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-05-24T21:13:13.570 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:13:13.570 DEBUG:boto:Port: 8001 2023-05-24T21:13:13.571 DEBUG:boto:Params: {} 2023-05-24T21:13:13.572 DEBUG:boto:Token: None 2023-05-24T21:13:13.573 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Wed, 24 May 2023 21:13:13 GMT /qskrps-16/obj 2023-05-24T21:13:13.573 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:zhcnek7Y/0IzpI1G7l41E9wOpqg= 2023-05-24T21:13:13.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:13:13 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:zhcnek7Y/0IzpI1G7l41E9wOpqg='} 2023-05-24T21:13:13.584 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'mWp-FBuLKZQQhyq1w2NLI1gYdoFzB36'), ('x-amz-request-id', 'tx00000b924685e25f151ea-00646e7de9-1111-a2'), ('Date', 'Wed, 24 May 2023 21:13:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:13.584 DEBUG:boto:path=/qskrps-16/ 2023-05-24T21:13:13.585 DEBUG:boto:auth_path=/qskrps-16/ 2023-05-24T21:13:13.585 DEBUG:boto:Method: HEAD 2023-05-24T21:13:13.586 DEBUG:boto:Path: /qskrps-16/ 2023-05-24T21:13:13.586 DEBUG:boto:Data: 2023-05-24T21:13:13.586 DEBUG:boto:Headers: {} 2023-05-24T21:13:13.587 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:13:13.587 DEBUG:boto:Port: 8001 2023-05-24T21:13:13.588 DEBUG:boto:Params: {} 2023-05-24T21:13:13.588 DEBUG:boto:Token: None 2023-05-24T21:13:13.589 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:13:13 GMT /qskrps-16/ 2023-05-24T21:13:13.589 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:KyVlZQ6jxDD7AgZ1AvpNXc2Mvj0= 2023-05-24T21:13:13.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:13 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:KyVlZQ6jxDD7AgZ1AvpNXc2Mvj0=', 'Content-Length': '0'} 2023-05-24T21:13:13.591 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003f7329d145b10f70-00646e7de9-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:13:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:13.592 DEBUG:boto:path=/qskrps-16/obj 2023-05-24T21:13:13.593 DEBUG:boto:auth_path=/qskrps-16/obj 2023-05-24T21:13:13.594 DEBUG:boto:Method: DELETE 2023-05-24T21:13:13.594 DEBUG:boto:Path: /qskrps-16/obj 2023-05-24T21:13:13.595 DEBUG:boto:Data: 2023-05-24T21:13:13.595 DEBUG:boto:Headers: {} 2023-05-24T21:13:13.596 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:13:13.596 DEBUG:boto:Port: 8001 2023-05-24T21:13:13.596 DEBUG:boto:Params: {} 2023-05-24T21:13:13.597 DEBUG:boto:Token: None 2023-05-24T21:13:13.597 DEBUG:boto:StringToSign: DELETE Wed, 24 May 2023 21:13:13 GMT /qskrps-16/obj 2023-05-24T21:13:13.598 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:jKPvHWwPupVXDEkoHn4UpaSN+Hk= 2023-05-24T21:13:13.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:13 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:jKPvHWwPupVXDEkoHn4UpaSN+Hk=', 'Content-Length': '0'} 2023-05-24T21:13:13.606 DEBUG:boto:Response headers: [('x-amz-version-id', 'HvNjjCsKAXfmubdGebv7RT-07d7eEQb'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx0000056fa3f8132d6ffbf-00646e7de9-1111-a2'), ('Date', 'Wed, 24 May 2023 21:13:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:13.607 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=qskrps-15 2023-05-24T21:13:13.608 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-15', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:13:13.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-15', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:13:13.610 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-15 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:13:13.667 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:13:13.667 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:13:13.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:13:13.695+0000 7f88ecea7780 1 waiting to reach incremental sync.. 2023-05-24T21:13:18.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:13:18.696+0000 7f88ecea7780 1 waiting to reach incremental sync.. 2023-05-24T21:13:23.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:13:23.696+0000 7f88ecea7780 1 waiting to reach incremental sync.. 2023-05-24T21:13:28.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:13:28.696+0000 7f88ecea7780 1 waiting to reach incremental sync.. 2023-05-24T21:13:33.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:13:33.698+0000 7f88ecea7780 1 bucket sync caught up with source: 2023-05-24T21:13:33.705 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000003.16.12, , ] 2023-05-24T21:13:33.706 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000003.16.12, , ] 2023-05-24T21:13:33.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:13:33.698+0000 7f88ecea7780 0 bucket checkpoint complete 2023-05-24T21:13:33.721 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=qskrps-15 2023-05-24T21:13:33.722 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-15', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:13:33.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-15', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:13:33.723 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-15 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:13:33.789 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:13:33.790 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:13:33.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:13:33.823+0000 7f9aa7dd8780 1 waiting to reach incremental sync.. 2023-05-24T21:13:38.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:13:38.824+0000 7f9aa7dd8780 1 waiting to reach incremental sync.. 2023-05-24T21:13:43.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:13:43.824+0000 7f9aa7dd8780 1 waiting to reach incremental sync.. 2023-05-24T21:13:48.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:13:48.825+0000 7f9aa7dd8780 1 waiting to reach incremental sync.. 2023-05-24T21:13:53.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:13:53.827+0000 7f9aa7dd8780 1 bucket sync caught up with source: 2023-05-24T21:13:53.833 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000002.24.6, , ] 2023-05-24T21:13:53.833 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000002.24.6, , ] 2023-05-24T21:13:53.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:13:53.827+0000 7f9aa7dd8780 0 bucket checkpoint complete 2023-05-24T21:13:53.849 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qskrps-15 zones={a2, a1} 2023-05-24T21:13:53.850 DEBUG:boto:path=/qskrps-15/ 2023-05-24T21:13:53.850 DEBUG:boto:auth_path=/qskrps-15/ 2023-05-24T21:13:53.850 DEBUG:boto:Method: HEAD 2023-05-24T21:13:53.851 DEBUG:boto:Path: /qskrps-15/ 2023-05-24T21:13:53.851 DEBUG:boto:Data: 2023-05-24T21:13:53.852 DEBUG:boto:Headers: {} 2023-05-24T21:13:53.852 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:13:53.853 DEBUG:boto:Port: 8001 2023-05-24T21:13:53.853 DEBUG:boto:Params: {} 2023-05-24T21:13:53.853 DEBUG:boto:Token: None 2023-05-24T21:13:53.854 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:13:53 GMT /qskrps-15/ 2023-05-24T21:13:53.854 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:KVCgSc+Ag+sl6Lhg9Q6clswndb4= 2023-05-24T21:13:53.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:KVCgSc+Ag+sl6Lhg9Q6clswndb4=', 'Content-Length': '0'} 2023-05-24T21:13:53.857 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000484f2beeea5f50da-00646e7e11-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:13:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:53.858 DEBUG:boto:path=/qskrps-15/ 2023-05-24T21:13:53.858 DEBUG:boto:auth_path=/qskrps-15/ 2023-05-24T21:13:53.858 DEBUG:boto:Method: HEAD 2023-05-24T21:13:53.859 DEBUG:boto:Path: /qskrps-15/ 2023-05-24T21:13:53.859 DEBUG:boto:Data: 2023-05-24T21:13:53.859 DEBUG:boto:Headers: {} 2023-05-24T21:13:53.860 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:13:53.860 DEBUG:boto:Port: 8000 2023-05-24T21:13:53.860 DEBUG:boto:Params: {} 2023-05-24T21:13:53.861 DEBUG:boto:Token: None 2023-05-24T21:13:53.861 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:13:53 GMT /qskrps-15/ 2023-05-24T21:13:53.862 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:KVCgSc+Ag+sl6Lhg9Q6clswndb4= 2023-05-24T21:13:53.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:KVCgSc+Ag+sl6Lhg9Q6clswndb4=', 'Content-Length': '0'} 2023-05-24T21:13:53.864 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000af44548521033e69-00646e7e11-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:13:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:53.865 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-24T21:13:53.865 DEBUG:boto:path=/qskrps-15/ 2023-05-24T21:13:53.866 DEBUG:boto:auth_path=/qskrps-15/ 2023-05-24T21:13:53.866 DEBUG:boto:path=/qskrps-15/?versions&max-keys=999 2023-05-24T21:13:53.866 DEBUG:boto:auth_path=/qskrps-15/?versions&max-keys=999 2023-05-24T21:13:53.867 DEBUG:boto:Method: GET 2023-05-24T21:13:53.867 DEBUG:boto:Path: /qskrps-15/?versions&max-keys=999 2023-05-24T21:13:53.867 DEBUG:boto:Data: 2023-05-24T21:13:53.868 DEBUG:boto:Headers: {} 2023-05-24T21:13:53.868 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:13:53.869 DEBUG:boto:Port: 8001 2023-05-24T21:13:53.869 DEBUG:boto:Params: {} 2023-05-24T21:13:53.869 DEBUG:boto:Token: None 2023-05-24T21:13:53.870 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:13:53 GMT /qskrps-15/?versions 2023-05-24T21:13:53.870 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:H+0sSMznZOxz2CJrCFRI3q5jJDE= 2023-05-24T21:13:53.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:H+0sSMznZOxz2CJrCFRI3q5jJDE=', 'Content-Length': '0'} 2023-05-24T21:13:53.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004854151e6f1e071a-00646e7e11-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:13:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:53.874 DEBUG:boto:b'qskrps-15999falseobjWxMuLjogdZXKjqgBmDHHurL1F-dBcaStrue2023-05-24T21:13:13.544Zrgw-multisite-test-userMultisite Test UserNormalobjxW.Bqf-y7QqZiapDjLACMtTI-TEWX9qfalse2023-05-24T21:13:13.510Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:13:53.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:13:53.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:13:53.875 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-24T21:13:53.875 DEBUG:boto:path=/qskrps-15/ 2023-05-24T21:13:53.876 DEBUG:boto:auth_path=/qskrps-15/ 2023-05-24T21:13:53.876 DEBUG:boto:path=/qskrps-15/?versions&max-keys=999 2023-05-24T21:13:53.877 DEBUG:boto:auth_path=/qskrps-15/?versions&max-keys=999 2023-05-24T21:13:53.877 DEBUG:boto:Method: GET 2023-05-24T21:13:53.877 DEBUG:boto:Path: /qskrps-15/?versions&max-keys=999 2023-05-24T21:13:53.878 DEBUG:boto:Data: 2023-05-24T21:13:53.878 DEBUG:boto:Headers: {} 2023-05-24T21:13:53.878 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:13:53.879 DEBUG:boto:Port: 8000 2023-05-24T21:13:53.879 DEBUG:boto:Params: {} 2023-05-24T21:13:53.879 DEBUG:boto:Token: None 2023-05-24T21:13:53.880 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:13:53 GMT /qskrps-15/?versions 2023-05-24T21:13:53.880 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:H+0sSMznZOxz2CJrCFRI3q5jJDE= 2023-05-24T21:13:53.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:H+0sSMznZOxz2CJrCFRI3q5jJDE=', 'Content-Length': '0'} 2023-05-24T21:13:53.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078c1b2568641949c-00646e7e11-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:13:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:53.884 DEBUG:boto:b'qskrps-15999falseobjWxMuLjogdZXKjqgBmDHHurL1F-dBcaStrue2023-05-24T21:13:13.544Zrgw-multisite-test-userMultisite Test UserNormalobjxW.Bqf-y7QqZiapDjLACMtTI-TEWX9qfalse2023-05-24T21:13:13.510Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:13:53.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:13:53.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:13:53.885 DEBUG:boto:path=/qskrps-15/ 2023-05-24T21:13:53.885 DEBUG:boto:auth_path=/qskrps-15/ 2023-05-24T21:13:53.886 DEBUG:boto:path=/qskrps-15/?versions&max-keys=999 2023-05-24T21:13:53.886 DEBUG:boto:auth_path=/qskrps-15/?versions&max-keys=999 2023-05-24T21:13:53.886 DEBUG:boto:Method: GET 2023-05-24T21:13:53.887 DEBUG:boto:Path: /qskrps-15/?versions&max-keys=999 2023-05-24T21:13:53.887 DEBUG:boto:Data: 2023-05-24T21:13:53.887 DEBUG:boto:Headers: {} 2023-05-24T21:13:53.888 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:13:53.888 DEBUG:boto:Port: 8001 2023-05-24T21:13:53.888 DEBUG:boto:Params: {} 2023-05-24T21:13:53.889 DEBUG:boto:Token: None 2023-05-24T21:13:53.889 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:13:53 GMT /qskrps-15/?versions 2023-05-24T21:13:53.890 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:H+0sSMznZOxz2CJrCFRI3q5jJDE= 2023-05-24T21:13:53.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:H+0sSMznZOxz2CJrCFRI3q5jJDE=', 'Content-Length': '0'} 2023-05-24T21:13:53.892 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e369f798f67a309-00646e7e11-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:13:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:53.893 DEBUG:boto:b'qskrps-15999falseobjWxMuLjogdZXKjqgBmDHHurL1F-dBcaStrue2023-05-24T21:13:13.544Zrgw-multisite-test-userMultisite Test UserNormalobjxW.Bqf-y7QqZiapDjLACMtTI-TEWX9qfalse2023-05-24T21:13:13.510Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:13:53.894 DEBUG:boto:path=/qskrps-15/ 2023-05-24T21:13:53.894 DEBUG:boto:auth_path=/qskrps-15/ 2023-05-24T21:13:53.894 DEBUG:boto:path=/qskrps-15/?versions&max-keys=999 2023-05-24T21:13:53.895 DEBUG:boto:auth_path=/qskrps-15/?versions&max-keys=999 2023-05-24T21:13:53.895 DEBUG:boto:Method: GET 2023-05-24T21:13:53.895 DEBUG:boto:Path: /qskrps-15/?versions&max-keys=999 2023-05-24T21:13:53.896 DEBUG:boto:Data: 2023-05-24T21:13:53.896 DEBUG:boto:Headers: {} 2023-05-24T21:13:53.896 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:13:53.897 DEBUG:boto:Port: 8000 2023-05-24T21:13:53.897 DEBUG:boto:Params: {} 2023-05-24T21:13:53.897 DEBUG:boto:Token: None 2023-05-24T21:13:53.898 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:13:53 GMT /qskrps-15/?versions 2023-05-24T21:13:53.898 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:H+0sSMznZOxz2CJrCFRI3q5jJDE= 2023-05-24T21:13:53.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:H+0sSMznZOxz2CJrCFRI3q5jJDE=', 'Content-Length': '0'} 2023-05-24T21:13:53.901 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ea3aeb59db90a48-00646e7e11-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:13:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:53.902 DEBUG:boto:b'qskrps-15999falseobjWxMuLjogdZXKjqgBmDHHurL1F-dBcaStrue2023-05-24T21:13:13.544Zrgw-multisite-test-userMultisite Test UserNormalobjxW.Bqf-y7QqZiapDjLACMtTI-TEWX9qfalse2023-05-24T21:13:13.510Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:13:53.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:13:53.903 DEBUG:boto:path=/qskrps-15/obj 2023-05-24T21:13:53.903 DEBUG:boto:auth_path=/qskrps-15/obj 2023-05-24T21:13:53.903 DEBUG:boto:Method: HEAD 2023-05-24T21:13:53.904 DEBUG:boto:Path: /qskrps-15/obj 2023-05-24T21:13:53.904 DEBUG:boto:Data: 2023-05-24T21:13:53.904 DEBUG:boto:Headers: {} 2023-05-24T21:13:53.905 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:13:53.905 DEBUG:boto:Port: 8001 2023-05-24T21:13:53.905 DEBUG:boto:Params: {} 2023-05-24T21:13:53.906 DEBUG:boto:Token: None 2023-05-24T21:13:53.906 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:13:53 GMT /qskrps-15/obj 2023-05-24T21:13:53.907 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:f8CoSj+raaeFG/CJv9+flGJKWQY= 2023-05-24T21:13:53.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:f8CoSj+raaeFG/CJv9+flGJKWQY=', 'Content-Length': '0'} 2023-05-24T21:13:53.909 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000050c77d24eeb57526-00646e7e11-1111-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:13:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:53.910 DEBUG:boto:path=/qskrps-15/obj 2023-05-24T21:13:53.910 DEBUG:boto:auth_path=/qskrps-15/obj 2023-05-24T21:13:53.910 DEBUG:boto:Method: HEAD 2023-05-24T21:13:53.911 DEBUG:boto:Path: /qskrps-15/obj 2023-05-24T21:13:53.911 DEBUG:boto:Data: 2023-05-24T21:13:53.912 DEBUG:boto:Headers: {} 2023-05-24T21:13:53.913 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:13:53.913 DEBUG:boto:Port: 8000 2023-05-24T21:13:53.914 DEBUG:boto:Params: {} 2023-05-24T21:13:53.915 DEBUG:boto:Token: None 2023-05-24T21:13:53.917 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:13:53 GMT /qskrps-15/obj 2023-05-24T21:13:53.918 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:f8CoSj+raaeFG/CJv9+flGJKWQY= 2023-05-24T21:13:53.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:f8CoSj+raaeFG/CJv9+flGJKWQY=', 'Content-Length': '0'} 2023-05-24T21:13:53.921 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000f4eb7cb1cce4eed0-00646e7e11-110d-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:13:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:53.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:13:53.922 DEBUG:boto:path=/qskrps-15/obj 2023-05-24T21:13:53.923 DEBUG:boto:auth_path=/qskrps-15/obj 2023-05-24T21:13:53.923 DEBUG:boto:path=/qskrps-15/obj?versionId=xW.Bqf-y7QqZiapDjLACMtTI-TEWX9q 2023-05-24T21:13:53.923 DEBUG:boto:auth_path=/qskrps-15/obj?versionId=xW.Bqf-y7QqZiapDjLACMtTI-TEWX9q 2023-05-24T21:13:53.924 DEBUG:boto:Method: GET 2023-05-24T21:13:53.924 DEBUG:boto:Path: /qskrps-15/obj?versionId=xW.Bqf-y7QqZiapDjLACMtTI-TEWX9q 2023-05-24T21:13:53.925 DEBUG:boto:Data: 2023-05-24T21:13:53.925 DEBUG:boto:Headers: {} 2023-05-24T21:13:53.925 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:13:53.926 DEBUG:boto:Port: 8001 2023-05-24T21:13:53.927 DEBUG:boto:Params: {} 2023-05-24T21:13:53.927 DEBUG:boto:Token: None 2023-05-24T21:13:53.927 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:13:53 GMT /qskrps-15/obj?versionId=xW.Bqf-y7QqZiapDjLACMtTI-TEWX9q 2023-05-24T21:13:53.928 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5vMYPq5eNNJsF4Y07z+/TQSjQwQ= 2023-05-24T21:13:53.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5vMYPq5eNNJsF4Y07z+/TQSjQwQ=', 'Content-Length': '0'} 2023-05-24T21:13:53.930 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:13:13 GMT'), ('x-amz-version-id', 'xW.Bqf-y7QqZiapDjLACMtTI-TEWX9q'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000073b3cacd72c186d3-00646e7e11-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:13:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:53.931 DEBUG:boto:path=/qskrps-15/obj 2023-05-24T21:13:53.931 DEBUG:boto:auth_path=/qskrps-15/obj 2023-05-24T21:13:53.932 DEBUG:boto:path=/qskrps-15/obj?versionId=xW.Bqf-y7QqZiapDjLACMtTI-TEWX9q 2023-05-24T21:13:53.932 DEBUG:boto:auth_path=/qskrps-15/obj?versionId=xW.Bqf-y7QqZiapDjLACMtTI-TEWX9q 2023-05-24T21:13:53.932 DEBUG:boto:Method: GET 2023-05-24T21:13:53.933 DEBUG:boto:Path: /qskrps-15/obj?versionId=xW.Bqf-y7QqZiapDjLACMtTI-TEWX9q 2023-05-24T21:13:53.933 DEBUG:boto:Data: 2023-05-24T21:13:53.934 DEBUG:boto:Headers: {} 2023-05-24T21:13:53.934 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:13:53.934 DEBUG:boto:Port: 8000 2023-05-24T21:13:53.935 DEBUG:boto:Params: {} 2023-05-24T21:13:53.935 DEBUG:boto:Token: None 2023-05-24T21:13:53.936 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:13:53 GMT /qskrps-15/obj?versionId=xW.Bqf-y7QqZiapDjLACMtTI-TEWX9q 2023-05-24T21:13:53.936 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5vMYPq5eNNJsF4Y07z+/TQSjQwQ= 2023-05-24T21:13:53.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5vMYPq5eNNJsF4Y07z+/TQSjQwQ=', 'Content-Length': '0'} 2023-05-24T21:13:53.938 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:13:13 GMT'), ('x-amz-version-id', 'xW.Bqf-y7QqZiapDjLACMtTI-TEWX9q'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ba8ff59118493d1d-00646e7e11-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:13:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:53.939 DEBUG:boto:path=/qskrps-15/obj 2023-05-24T21:13:53.939 DEBUG:boto:auth_path=/qskrps-15/obj 2023-05-24T21:13:53.940 DEBUG:boto:path=/qskrps-15/obj?versionId=xW.Bqf-y7QqZiapDjLACMtTI-TEWX9q 2023-05-24T21:13:53.941 DEBUG:boto:auth_path=/qskrps-15/obj?versionId=xW.Bqf-y7QqZiapDjLACMtTI-TEWX9q 2023-05-24T21:13:53.942 DEBUG:boto:Method: HEAD 2023-05-24T21:13:53.942 DEBUG:boto:Path: /qskrps-15/obj?versionId=xW.Bqf-y7QqZiapDjLACMtTI-TEWX9q 2023-05-24T21:13:53.943 DEBUG:boto:Data: 2023-05-24T21:13:53.944 DEBUG:boto:Headers: {} 2023-05-24T21:13:53.945 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:13:53.946 DEBUG:boto:Port: 8001 2023-05-24T21:13:53.946 DEBUG:boto:Params: {} 2023-05-24T21:13:53.947 DEBUG:boto:Token: None 2023-05-24T21:13:53.948 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:13:53 GMT /qskrps-15/obj?versionId=xW.Bqf-y7QqZiapDjLACMtTI-TEWX9q 2023-05-24T21:13:53.949 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:rWfv3Z08SBrBnXwxcN3JzgdRsK8= 2023-05-24T21:13:53.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:rWfv3Z08SBrBnXwxcN3JzgdRsK8=', 'Content-Length': '0'} 2023-05-24T21:13:53.952 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:13:13 GMT'), ('x-amz-version-id', 'xW.Bqf-y7QqZiapDjLACMtTI-TEWX9q'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f7fd03f4282d12af-00646e7e11-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:13:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:53.953 DEBUG:boto:path=/qskrps-15/obj 2023-05-24T21:13:53.953 DEBUG:boto:auth_path=/qskrps-15/obj 2023-05-24T21:13:53.954 DEBUG:boto:path=/qskrps-15/obj?versionId=xW.Bqf-y7QqZiapDjLACMtTI-TEWX9q 2023-05-24T21:13:53.955 DEBUG:boto:auth_path=/qskrps-15/obj?versionId=xW.Bqf-y7QqZiapDjLACMtTI-TEWX9q 2023-05-24T21:13:53.956 DEBUG:boto:Method: HEAD 2023-05-24T21:13:53.957 DEBUG:boto:Path: /qskrps-15/obj?versionId=xW.Bqf-y7QqZiapDjLACMtTI-TEWX9q 2023-05-24T21:13:53.957 DEBUG:boto:Data: 2023-05-24T21:13:53.958 DEBUG:boto:Headers: {} 2023-05-24T21:13:53.959 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:13:53.959 DEBUG:boto:Port: 8000 2023-05-24T21:13:53.960 DEBUG:boto:Params: {} 2023-05-24T21:13:53.961 DEBUG:boto:Token: None 2023-05-24T21:13:53.962 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:13:53 GMT /qskrps-15/obj?versionId=xW.Bqf-y7QqZiapDjLACMtTI-TEWX9q 2023-05-24T21:13:53.963 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:rWfv3Z08SBrBnXwxcN3JzgdRsK8= 2023-05-24T21:13:53.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:rWfv3Z08SBrBnXwxcN3JzgdRsK8=', 'Content-Length': '0'} 2023-05-24T21:13:53.965 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:13:13 GMT'), ('x-amz-version-id', 'xW.Bqf-y7QqZiapDjLACMtTI-TEWX9q'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000010e439cfab8e3774-00646e7e11-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:13:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:53.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:13:53.966 DEBUG:boto:path=/qskrps-15/obj 2023-05-24T21:13:53.966 DEBUG:boto:auth_path=/qskrps-15/obj 2023-05-24T21:13:53.967 DEBUG:boto:path=/qskrps-15/obj?versionId=xW.Bqf-y7QqZiapDjLACMtTI-TEWX9q 2023-05-24T21:13:53.967 DEBUG:boto:auth_path=/qskrps-15/obj?versionId=xW.Bqf-y7QqZiapDjLACMtTI-TEWX9q 2023-05-24T21:13:53.968 DEBUG:boto:Method: GET 2023-05-24T21:13:53.968 DEBUG:boto:Path: /qskrps-15/obj?versionId=xW.Bqf-y7QqZiapDjLACMtTI-TEWX9q 2023-05-24T21:13:53.968 DEBUG:boto:Data: 2023-05-24T21:13:53.969 DEBUG:boto:Headers: {} 2023-05-24T21:13:53.969 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:13:53.969 DEBUG:boto:Port: 8001 2023-05-24T21:13:53.970 DEBUG:boto:Params: {} 2023-05-24T21:13:53.970 DEBUG:boto:Token: None 2023-05-24T21:13:53.971 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:13:53 GMT /qskrps-15/obj?versionId=xW.Bqf-y7QqZiapDjLACMtTI-TEWX9q 2023-05-24T21:13:53.971 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5vMYPq5eNNJsF4Y07z+/TQSjQwQ= 2023-05-24T21:13:53.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5vMYPq5eNNJsF4Y07z+/TQSjQwQ=', 'Content-Length': '0'} 2023-05-24T21:13:53.974 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:13:13 GMT'), ('x-amz-version-id', 'xW.Bqf-y7QqZiapDjLACMtTI-TEWX9q'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009bac6c97bb6b9729-00646e7e11-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:13:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:53.974 DEBUG:boto:path=/qskrps-15/obj 2023-05-24T21:13:53.975 DEBUG:boto:auth_path=/qskrps-15/obj 2023-05-24T21:13:53.975 DEBUG:boto:path=/qskrps-15/obj?versionId=xW.Bqf-y7QqZiapDjLACMtTI-TEWX9q 2023-05-24T21:13:53.976 DEBUG:boto:auth_path=/qskrps-15/obj?versionId=xW.Bqf-y7QqZiapDjLACMtTI-TEWX9q 2023-05-24T21:13:53.976 DEBUG:boto:Method: GET 2023-05-24T21:13:53.977 DEBUG:boto:Path: /qskrps-15/obj?versionId=xW.Bqf-y7QqZiapDjLACMtTI-TEWX9q 2023-05-24T21:13:53.977 DEBUG:boto:Data: 2023-05-24T21:13:53.977 DEBUG:boto:Headers: {} 2023-05-24T21:13:53.978 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:13:53.978 DEBUG:boto:Port: 8000 2023-05-24T21:13:53.978 DEBUG:boto:Params: {} 2023-05-24T21:13:53.979 DEBUG:boto:Token: None 2023-05-24T21:13:53.979 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:13:53 GMT /qskrps-15/obj?versionId=xW.Bqf-y7QqZiapDjLACMtTI-TEWX9q 2023-05-24T21:13:53.980 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5vMYPq5eNNJsF4Y07z+/TQSjQwQ= 2023-05-24T21:13:53.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5vMYPq5eNNJsF4Y07z+/TQSjQwQ=', 'Content-Length': '0'} 2023-05-24T21:13:53.982 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:13:13 GMT'), ('x-amz-version-id', 'xW.Bqf-y7QqZiapDjLACMtTI-TEWX9q'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000042651e51efbfc2c1-00646e7e11-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:13:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:53.983 DEBUG:boto:path=/qskrps-15/ 2023-05-24T21:13:53.983 DEBUG:boto:auth_path=/qskrps-15/ 2023-05-24T21:13:53.983 DEBUG:boto:path=/qskrps-15/?prefix=obj 2023-05-24T21:13:53.984 DEBUG:boto:auth_path=/qskrps-15/?prefix=obj 2023-05-24T21:13:53.984 DEBUG:boto:Method: GET 2023-05-24T21:13:53.984 DEBUG:boto:Path: /qskrps-15/?prefix=obj 2023-05-24T21:13:53.985 DEBUG:boto:Data: 2023-05-24T21:13:53.985 DEBUG:boto:Headers: {} 2023-05-24T21:13:53.986 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:13:53.986 DEBUG:boto:Port: 8001 2023-05-24T21:13:53.987 DEBUG:boto:Params: {} 2023-05-24T21:13:53.987 DEBUG:boto:Token: None 2023-05-24T21:13:53.987 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:13:53 GMT /qskrps-15/ 2023-05-24T21:13:53.989 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:WDqjC1qp/vDPljhxB7jhxl1W+a8= 2023-05-24T21:13:53.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WDqjC1qp/vDPljhxB7jhxl1W+a8=', 'Content-Length': '0'} 2023-05-24T21:13:53.992 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d736fcdd594dddc5-00646e7e11-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:13:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:54.032 DEBUG:boto:b'qskrps-15obj1000false' 2023-05-24T21:13:54.034 DEBUG:boto:path=/qskrps-15/ 2023-05-24T21:13:54.035 DEBUG:boto:auth_path=/qskrps-15/ 2023-05-24T21:13:54.036 DEBUG:boto:path=/qskrps-15/?prefix=obj 2023-05-24T21:13:54.037 DEBUG:boto:auth_path=/qskrps-15/?prefix=obj 2023-05-24T21:13:54.038 DEBUG:boto:Method: GET 2023-05-24T21:13:54.039 DEBUG:boto:Path: /qskrps-15/?prefix=obj 2023-05-24T21:13:54.039 DEBUG:boto:Data: 2023-05-24T21:13:54.040 DEBUG:boto:Headers: {} 2023-05-24T21:13:54.040 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:13:54.041 DEBUG:boto:Port: 8000 2023-05-24T21:13:54.041 DEBUG:boto:Params: {} 2023-05-24T21:13:54.041 DEBUG:boto:Token: None 2023-05-24T21:13:54.042 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:13:54 GMT /qskrps-15/ 2023-05-24T21:13:54.043 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:j48w/Jyl7PGtKkBy3NKMr7eSVDs= 2023-05-24T21:13:54.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:j48w/Jyl7PGtKkBy3NKMr7eSVDs=', 'Content-Length': '0'} 2023-05-24T21:13:54.046 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003beea06c6f13ede-00646e7e12-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:13:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:54.047 DEBUG:boto:b'qskrps-15obj1000false' 2023-05-24T21:13:54.049 DEBUG:boto:path=/qskrps-15/obj 2023-05-24T21:13:54.049 DEBUG:boto:auth_path=/qskrps-15/obj 2023-05-24T21:13:54.050 DEBUG:boto:Method: HEAD 2023-05-24T21:13:54.050 DEBUG:boto:Path: /qskrps-15/obj 2023-05-24T21:13:54.050 DEBUG:boto:Data: 2023-05-24T21:13:54.051 DEBUG:boto:Headers: {} 2023-05-24T21:13:54.051 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:13:54.051 DEBUG:boto:Port: 8001 2023-05-24T21:13:54.052 DEBUG:boto:Params: {} 2023-05-24T21:13:54.052 DEBUG:boto:Token: None 2023-05-24T21:13:54.053 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:13:54 GMT /qskrps-15/obj 2023-05-24T21:13:54.053 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:UwSSq3rjcGtcDd6IKd4RTRhd4vA= 2023-05-24T21:13:54.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:UwSSq3rjcGtcDd6IKd4RTRhd4vA=', 'Content-Length': '0'} 2023-05-24T21:13:54.056 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000004b52fab13c465e05-00646e7e12-1111-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:13:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:54.057 DEBUG:boto:path=/qskrps-15/obj 2023-05-24T21:13:54.057 DEBUG:boto:auth_path=/qskrps-15/obj 2023-05-24T21:13:54.058 DEBUG:boto:Method: HEAD 2023-05-24T21:13:54.058 DEBUG:boto:Path: /qskrps-15/obj 2023-05-24T21:13:54.059 DEBUG:boto:Data: 2023-05-24T21:13:54.060 DEBUG:boto:Headers: {} 2023-05-24T21:13:54.060 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:13:54.060 DEBUG:boto:Port: 8000 2023-05-24T21:13:54.061 DEBUG:boto:Params: {} 2023-05-24T21:13:54.061 DEBUG:boto:Token: None 2023-05-24T21:13:54.062 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:13:54 GMT /qskrps-15/obj 2023-05-24T21:13:54.062 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:UwSSq3rjcGtcDd6IKd4RTRhd4vA= 2023-05-24T21:13:54.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:UwSSq3rjcGtcDd6IKd4RTRhd4vA=', 'Content-Length': '0'} 2023-05-24T21:13:54.065 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000006315784720c2c766-00646e7e12-110d-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:13:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:54.066 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qskrps-15 zones={a2, a1} 2023-05-24T21:13:54.067 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=qskrps-16 2023-05-24T21:13:54.068 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-16', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:13:54.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-16', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:13:54.069 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-16 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:13:54.133 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:13:54.133 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:13:54.170 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:13:54.164+0000 7efe2aa8d780 1 bucket sync caught up with source: 2023-05-24T21:13:54.171 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000002.32.6, , ] 2023-05-24T21:13:54.171 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000002.32.6, , ] 2023-05-24T21:13:54.172 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:13:54.164+0000 7efe2aa8d780 0 bucket checkpoint complete 2023-05-24T21:13:54.186 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=qskrps-16 2023-05-24T21:13:54.187 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-16', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:13:54.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-16', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:13:54.189 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-16 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:13:54.246 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:13:54.247 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:13:54.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:13:54.297+0000 7f46738e2780 1 bucket sync caught up with source: 2023-05-24T21:13:54.303 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000003.37.12, , ] 2023-05-24T21:13:54.304 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000003.37.12, , ] 2023-05-24T21:13:54.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:13:54.297+0000 7f46738e2780 0 bucket checkpoint complete 2023-05-24T21:13:54.315 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qskrps-16 zones={a2, a1} 2023-05-24T21:13:54.316 DEBUG:boto:path=/qskrps-16/ 2023-05-24T21:13:54.316 DEBUG:boto:auth_path=/qskrps-16/ 2023-05-24T21:13:54.317 DEBUG:boto:Method: HEAD 2023-05-24T21:13:54.317 DEBUG:boto:Path: /qskrps-16/ 2023-05-24T21:13:54.318 DEBUG:boto:Data: 2023-05-24T21:13:54.318 DEBUG:boto:Headers: {} 2023-05-24T21:13:54.318 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:13:54.319 DEBUG:boto:Port: 8001 2023-05-24T21:13:54.319 DEBUG:boto:Params: {} 2023-05-24T21:13:54.319 DEBUG:boto:Token: None 2023-05-24T21:13:54.320 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:13:54 GMT /qskrps-16/ 2023-05-24T21:13:54.320 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:c4ijWdu+wg9zKi1yqyAwxpPGLdI= 2023-05-24T21:13:54.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:c4ijWdu+wg9zKi1yqyAwxpPGLdI=', 'Content-Length': '0'} 2023-05-24T21:13:54.324 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b835ba2cbdcfa226-00646e7e12-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:13:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:54.325 DEBUG:boto:path=/qskrps-16/ 2023-05-24T21:13:54.326 DEBUG:boto:auth_path=/qskrps-16/ 2023-05-24T21:13:54.326 DEBUG:boto:Method: HEAD 2023-05-24T21:13:54.327 DEBUG:boto:Path: /qskrps-16/ 2023-05-24T21:13:54.327 DEBUG:boto:Data: 2023-05-24T21:13:54.328 DEBUG:boto:Headers: {} 2023-05-24T21:13:54.329 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:13:54.329 DEBUG:boto:Port: 8000 2023-05-24T21:13:54.330 DEBUG:boto:Params: {} 2023-05-24T21:13:54.331 DEBUG:boto:Token: None 2023-05-24T21:13:54.332 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:13:54 GMT /qskrps-16/ 2023-05-24T21:13:54.333 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:c4ijWdu+wg9zKi1yqyAwxpPGLdI= 2023-05-24T21:13:54.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:c4ijWdu+wg9zKi1yqyAwxpPGLdI=', 'Content-Length': '0'} 2023-05-24T21:13:54.336 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b9696583876c6de2-00646e7e12-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:13:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:54.337 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-24T21:13:54.337 DEBUG:boto:path=/qskrps-16/ 2023-05-24T21:13:54.338 DEBUG:boto:auth_path=/qskrps-16/ 2023-05-24T21:13:54.339 DEBUG:boto:path=/qskrps-16/?versions&max-keys=999 2023-05-24T21:13:54.340 DEBUG:boto:auth_path=/qskrps-16/?versions&max-keys=999 2023-05-24T21:13:54.340 DEBUG:boto:Method: GET 2023-05-24T21:13:54.341 DEBUG:boto:Path: /qskrps-16/?versions&max-keys=999 2023-05-24T21:13:54.342 DEBUG:boto:Data: 2023-05-24T21:13:54.342 DEBUG:boto:Headers: {} 2023-05-24T21:13:54.343 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:13:54.344 DEBUG:boto:Port: 8001 2023-05-24T21:13:54.344 DEBUG:boto:Params: {} 2023-05-24T21:13:54.345 DEBUG:boto:Token: None 2023-05-24T21:13:54.346 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:13:54 GMT /qskrps-16/?versions 2023-05-24T21:13:54.347 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:O9tNBxBCj9xzBDddh06HJYC02ZQ= 2023-05-24T21:13:54.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:O9tNBxBCj9xzBDddh06HJYC02ZQ=', 'Content-Length': '0'} 2023-05-24T21:13:54.350 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc8eef1dc5b09956-00646e7e12-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:13:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:54.352 DEBUG:boto:b'qskrps-16999falseobjHvNjjCsKAXfmubdGebv7RT-07d7eEQbtrue2023-05-24T21:13:13.595Zrgw-multisite-test-userMultisite Test UserNormalobjmWp-FBuLKZQQhyq1w2NLI1gYdoFzB36false2023-05-24T21:13:13.570Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:13:54.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:13:54.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:13:54.353 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-24T21:13:54.354 DEBUG:boto:path=/qskrps-16/ 2023-05-24T21:13:54.354 DEBUG:boto:auth_path=/qskrps-16/ 2023-05-24T21:13:54.354 DEBUG:boto:path=/qskrps-16/?versions&max-keys=999 2023-05-24T21:13:54.355 DEBUG:boto:auth_path=/qskrps-16/?versions&max-keys=999 2023-05-24T21:13:54.355 DEBUG:boto:Method: GET 2023-05-24T21:13:54.356 DEBUG:boto:Path: /qskrps-16/?versions&max-keys=999 2023-05-24T21:13:54.356 DEBUG:boto:Data: 2023-05-24T21:13:54.356 DEBUG:boto:Headers: {} 2023-05-24T21:13:54.357 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:13:54.357 DEBUG:boto:Port: 8000 2023-05-24T21:13:54.357 DEBUG:boto:Params: {} 2023-05-24T21:13:54.358 DEBUG:boto:Token: None 2023-05-24T21:13:54.358 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:13:54 GMT /qskrps-16/?versions 2023-05-24T21:13:54.359 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:O9tNBxBCj9xzBDddh06HJYC02ZQ= 2023-05-24T21:13:54.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:O9tNBxBCj9xzBDddh06HJYC02ZQ=', 'Content-Length': '0'} 2023-05-24T21:13:54.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eaaeb6973df3cf81-00646e7e12-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:13:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:54.362 DEBUG:boto:b'qskrps-16999falseobjHvNjjCsKAXfmubdGebv7RT-07d7eEQbtrue2023-05-24T21:13:13.595Zrgw-multisite-test-userMultisite Test UserNormalobjmWp-FBuLKZQQhyq1w2NLI1gYdoFzB36false2023-05-24T21:13:13.570Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:13:54.363 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:13:54.363 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:13:54.363 DEBUG:boto:path=/qskrps-16/ 2023-05-24T21:13:54.364 DEBUG:boto:auth_path=/qskrps-16/ 2023-05-24T21:13:54.364 DEBUG:boto:path=/qskrps-16/?versions&max-keys=999 2023-05-24T21:13:54.365 DEBUG:boto:auth_path=/qskrps-16/?versions&max-keys=999 2023-05-24T21:13:54.365 DEBUG:boto:Method: GET 2023-05-24T21:13:54.365 DEBUG:boto:Path: /qskrps-16/?versions&max-keys=999 2023-05-24T21:13:54.366 DEBUG:boto:Data: 2023-05-24T21:13:54.366 DEBUG:boto:Headers: {} 2023-05-24T21:13:54.366 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:13:54.367 DEBUG:boto:Port: 8001 2023-05-24T21:13:54.367 DEBUG:boto:Params: {} 2023-05-24T21:13:54.368 DEBUG:boto:Token: None 2023-05-24T21:13:54.368 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:13:54 GMT /qskrps-16/?versions 2023-05-24T21:13:54.369 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:O9tNBxBCj9xzBDddh06HJYC02ZQ= 2023-05-24T21:13:54.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:O9tNBxBCj9xzBDddh06HJYC02ZQ=', 'Content-Length': '0'} 2023-05-24T21:13:54.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c312b606769149f6-00646e7e12-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:13:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:54.372 DEBUG:boto:b'qskrps-16999falseobjHvNjjCsKAXfmubdGebv7RT-07d7eEQbtrue2023-05-24T21:13:13.595Zrgw-multisite-test-userMultisite Test UserNormalobjmWp-FBuLKZQQhyq1w2NLI1gYdoFzB36false2023-05-24T21:13:13.570Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:13:54.373 DEBUG:boto:path=/qskrps-16/ 2023-05-24T21:13:54.374 DEBUG:boto:auth_path=/qskrps-16/ 2023-05-24T21:13:54.374 DEBUG:boto:path=/qskrps-16/?versions&max-keys=999 2023-05-24T21:13:54.374 DEBUG:boto:auth_path=/qskrps-16/?versions&max-keys=999 2023-05-24T21:13:54.375 DEBUG:boto:Method: GET 2023-05-24T21:13:54.375 DEBUG:boto:Path: /qskrps-16/?versions&max-keys=999 2023-05-24T21:13:54.375 DEBUG:boto:Data: 2023-05-24T21:13:54.376 DEBUG:boto:Headers: {} 2023-05-24T21:13:54.376 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:13:54.377 DEBUG:boto:Port: 8000 2023-05-24T21:13:54.378 DEBUG:boto:Params: {} 2023-05-24T21:13:54.378 DEBUG:boto:Token: None 2023-05-24T21:13:54.378 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:13:54 GMT /qskrps-16/?versions 2023-05-24T21:13:54.379 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:O9tNBxBCj9xzBDddh06HJYC02ZQ= 2023-05-24T21:13:54.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:O9tNBxBCj9xzBDddh06HJYC02ZQ=', 'Content-Length': '0'} 2023-05-24T21:13:54.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf2438ed699d621d-00646e7e12-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:13:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:54.382 DEBUG:boto:b'qskrps-16999falseobjHvNjjCsKAXfmubdGebv7RT-07d7eEQbtrue2023-05-24T21:13:13.595Zrgw-multisite-test-userMultisite Test UserNormalobjmWp-FBuLKZQQhyq1w2NLI1gYdoFzB36false2023-05-24T21:13:13.570Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:13:54.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:13:54.383 DEBUG:boto:path=/qskrps-16/obj 2023-05-24T21:13:54.384 DEBUG:boto:auth_path=/qskrps-16/obj 2023-05-24T21:13:54.384 DEBUG:boto:Method: HEAD 2023-05-24T21:13:54.384 DEBUG:boto:Path: /qskrps-16/obj 2023-05-24T21:13:54.385 DEBUG:boto:Data: 2023-05-24T21:13:54.386 DEBUG:boto:Headers: {} 2023-05-24T21:13:54.386 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:13:54.386 DEBUG:boto:Port: 8001 2023-05-24T21:13:54.387 DEBUG:boto:Params: {} 2023-05-24T21:13:54.387 DEBUG:boto:Token: None 2023-05-24T21:13:54.387 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:13:54 GMT /qskrps-16/obj 2023-05-24T21:13:54.388 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ajEuq8X3uhngfTZNAc/izuKyids= 2023-05-24T21:13:54.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ajEuq8X3uhngfTZNAc/izuKyids=', 'Content-Length': '0'} 2023-05-24T21:13:54.390 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000005dc29f2bb1b6ed7d-00646e7e12-1111-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:13:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:54.391 DEBUG:boto:path=/qskrps-16/obj 2023-05-24T21:13:54.391 DEBUG:boto:auth_path=/qskrps-16/obj 2023-05-24T21:13:54.392 DEBUG:boto:Method: HEAD 2023-05-24T21:13:54.392 DEBUG:boto:Path: /qskrps-16/obj 2023-05-24T21:13:54.392 DEBUG:boto:Data: 2023-05-24T21:13:54.393 DEBUG:boto:Headers: {} 2023-05-24T21:13:54.393 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:13:54.393 DEBUG:boto:Port: 8000 2023-05-24T21:13:54.394 DEBUG:boto:Params: {} 2023-05-24T21:13:54.394 DEBUG:boto:Token: None 2023-05-24T21:13:54.395 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:13:54 GMT /qskrps-16/obj 2023-05-24T21:13:54.395 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ajEuq8X3uhngfTZNAc/izuKyids= 2023-05-24T21:13:54.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ajEuq8X3uhngfTZNAc/izuKyids=', 'Content-Length': '0'} 2023-05-24T21:13:54.397 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000000db99aed71eea0e1-00646e7e12-110d-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:13:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:54.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:13:54.398 DEBUG:boto:path=/qskrps-16/obj 2023-05-24T21:13:54.399 DEBUG:boto:auth_path=/qskrps-16/obj 2023-05-24T21:13:54.399 DEBUG:boto:path=/qskrps-16/obj?versionId=mWp-FBuLKZQQhyq1w2NLI1gYdoFzB36 2023-05-24T21:13:54.399 DEBUG:boto:auth_path=/qskrps-16/obj?versionId=mWp-FBuLKZQQhyq1w2NLI1gYdoFzB36 2023-05-24T21:13:54.400 DEBUG:boto:Method: GET 2023-05-24T21:13:54.400 DEBUG:boto:Path: /qskrps-16/obj?versionId=mWp-FBuLKZQQhyq1w2NLI1gYdoFzB36 2023-05-24T21:13:54.401 DEBUG:boto:Data: 2023-05-24T21:13:54.401 DEBUG:boto:Headers: {} 2023-05-24T21:13:54.401 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:13:54.402 DEBUG:boto:Port: 8001 2023-05-24T21:13:54.402 DEBUG:boto:Params: {} 2023-05-24T21:13:54.403 DEBUG:boto:Token: None 2023-05-24T21:13:54.403 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:13:54 GMT /qskrps-16/obj?versionId=mWp-FBuLKZQQhyq1w2NLI1gYdoFzB36 2023-05-24T21:13:54.404 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:LxA6ah79Idt4zDx15/7OMQEEpdc= 2023-05-24T21:13:54.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:LxA6ah79Idt4zDx15/7OMQEEpdc=', 'Content-Length': '0'} 2023-05-24T21:13:54.406 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:13:13 GMT'), ('x-amz-version-id', 'mWp-FBuLKZQQhyq1w2NLI1gYdoFzB36'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004f449ee80727fef7-00646e7e12-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:13:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:54.406 DEBUG:boto:path=/qskrps-16/obj 2023-05-24T21:13:54.407 DEBUG:boto:auth_path=/qskrps-16/obj 2023-05-24T21:13:54.407 DEBUG:boto:path=/qskrps-16/obj?versionId=mWp-FBuLKZQQhyq1w2NLI1gYdoFzB36 2023-05-24T21:13:54.408 DEBUG:boto:auth_path=/qskrps-16/obj?versionId=mWp-FBuLKZQQhyq1w2NLI1gYdoFzB36 2023-05-24T21:13:54.408 DEBUG:boto:Method: GET 2023-05-24T21:13:54.409 DEBUG:boto:Path: /qskrps-16/obj?versionId=mWp-FBuLKZQQhyq1w2NLI1gYdoFzB36 2023-05-24T21:13:54.409 DEBUG:boto:Data: 2023-05-24T21:13:54.409 DEBUG:boto:Headers: {} 2023-05-24T21:13:54.410 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:13:54.410 DEBUG:boto:Port: 8000 2023-05-24T21:13:54.410 DEBUG:boto:Params: {} 2023-05-24T21:13:54.411 DEBUG:boto:Token: None 2023-05-24T21:13:54.411 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:13:54 GMT /qskrps-16/obj?versionId=mWp-FBuLKZQQhyq1w2NLI1gYdoFzB36 2023-05-24T21:13:54.412 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:LxA6ah79Idt4zDx15/7OMQEEpdc= 2023-05-24T21:13:54.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:LxA6ah79Idt4zDx15/7OMQEEpdc=', 'Content-Length': '0'} 2023-05-24T21:13:54.414 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:13:13 GMT'), ('x-amz-version-id', 'mWp-FBuLKZQQhyq1w2NLI1gYdoFzB36'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000097561aef4eaebe81-00646e7e12-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:13:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:54.414 DEBUG:boto:path=/qskrps-16/obj 2023-05-24T21:13:54.415 DEBUG:boto:auth_path=/qskrps-16/obj 2023-05-24T21:13:54.415 DEBUG:boto:path=/qskrps-16/obj?versionId=mWp-FBuLKZQQhyq1w2NLI1gYdoFzB36 2023-05-24T21:13:54.416 DEBUG:boto:auth_path=/qskrps-16/obj?versionId=mWp-FBuLKZQQhyq1w2NLI1gYdoFzB36 2023-05-24T21:13:54.416 DEBUG:boto:Method: HEAD 2023-05-24T21:13:54.416 DEBUG:boto:Path: /qskrps-16/obj?versionId=mWp-FBuLKZQQhyq1w2NLI1gYdoFzB36 2023-05-24T21:13:54.417 DEBUG:boto:Data: 2023-05-24T21:13:54.417 DEBUG:boto:Headers: {} 2023-05-24T21:13:54.417 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:13:54.418 DEBUG:boto:Port: 8001 2023-05-24T21:13:54.418 DEBUG:boto:Params: {} 2023-05-24T21:13:54.418 DEBUG:boto:Token: None 2023-05-24T21:13:54.419 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:13:54 GMT /qskrps-16/obj?versionId=mWp-FBuLKZQQhyq1w2NLI1gYdoFzB36 2023-05-24T21:13:54.419 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:swWcHKkDyNA1fFm26BAdBDDAWoY= 2023-05-24T21:13:54.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:swWcHKkDyNA1fFm26BAdBDDAWoY=', 'Content-Length': '0'} 2023-05-24T21:13:54.421 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:13:13 GMT'), ('x-amz-version-id', 'mWp-FBuLKZQQhyq1w2NLI1gYdoFzB36'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000033f2bb3639f7b9d6-00646e7e12-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:13:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:54.422 DEBUG:boto:path=/qskrps-16/obj 2023-05-24T21:13:54.422 DEBUG:boto:auth_path=/qskrps-16/obj 2023-05-24T21:13:54.423 DEBUG:boto:path=/qskrps-16/obj?versionId=mWp-FBuLKZQQhyq1w2NLI1gYdoFzB36 2023-05-24T21:13:54.423 DEBUG:boto:auth_path=/qskrps-16/obj?versionId=mWp-FBuLKZQQhyq1w2NLI1gYdoFzB36 2023-05-24T21:13:54.424 DEBUG:boto:Method: HEAD 2023-05-24T21:13:54.424 DEBUG:boto:Path: /qskrps-16/obj?versionId=mWp-FBuLKZQQhyq1w2NLI1gYdoFzB36 2023-05-24T21:13:54.424 DEBUG:boto:Data: 2023-05-24T21:13:54.425 DEBUG:boto:Headers: {} 2023-05-24T21:13:54.425 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:13:54.425 DEBUG:boto:Port: 8000 2023-05-24T21:13:54.426 DEBUG:boto:Params: {} 2023-05-24T21:13:54.426 DEBUG:boto:Token: None 2023-05-24T21:13:54.427 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:13:54 GMT /qskrps-16/obj?versionId=mWp-FBuLKZQQhyq1w2NLI1gYdoFzB36 2023-05-24T21:13:54.427 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:swWcHKkDyNA1fFm26BAdBDDAWoY= 2023-05-24T21:13:54.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:swWcHKkDyNA1fFm26BAdBDDAWoY=', 'Content-Length': '0'} 2023-05-24T21:13:54.429 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:13:13 GMT'), ('x-amz-version-id', 'mWp-FBuLKZQQhyq1w2NLI1gYdoFzB36'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000060095f1b0e61f98e-00646e7e12-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:13:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:54.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:13:54.430 DEBUG:boto:path=/qskrps-16/obj 2023-05-24T21:13:54.430 DEBUG:boto:auth_path=/qskrps-16/obj 2023-05-24T21:13:54.431 DEBUG:boto:path=/qskrps-16/obj?versionId=mWp-FBuLKZQQhyq1w2NLI1gYdoFzB36 2023-05-24T21:13:54.431 DEBUG:boto:auth_path=/qskrps-16/obj?versionId=mWp-FBuLKZQQhyq1w2NLI1gYdoFzB36 2023-05-24T21:13:54.431 DEBUG:boto:Method: GET 2023-05-24T21:13:54.432 DEBUG:boto:Path: /qskrps-16/obj?versionId=mWp-FBuLKZQQhyq1w2NLI1gYdoFzB36 2023-05-24T21:13:54.432 DEBUG:boto:Data: 2023-05-24T21:13:54.433 DEBUG:boto:Headers: {} 2023-05-24T21:13:54.433 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:13:54.433 DEBUG:boto:Port: 8001 2023-05-24T21:13:54.434 DEBUG:boto:Params: {} 2023-05-24T21:13:54.434 DEBUG:boto:Token: None 2023-05-24T21:13:54.434 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:13:54 GMT /qskrps-16/obj?versionId=mWp-FBuLKZQQhyq1w2NLI1gYdoFzB36 2023-05-24T21:13:54.435 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:LxA6ah79Idt4zDx15/7OMQEEpdc= 2023-05-24T21:13:54.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:LxA6ah79Idt4zDx15/7OMQEEpdc=', 'Content-Length': '0'} 2023-05-24T21:13:54.437 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:13:13 GMT'), ('x-amz-version-id', 'mWp-FBuLKZQQhyq1w2NLI1gYdoFzB36'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000accc2b666d3f8fb-00646e7e12-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:13:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:54.438 DEBUG:boto:path=/qskrps-16/obj 2023-05-24T21:13:54.438 DEBUG:boto:auth_path=/qskrps-16/obj 2023-05-24T21:13:54.439 DEBUG:boto:path=/qskrps-16/obj?versionId=mWp-FBuLKZQQhyq1w2NLI1gYdoFzB36 2023-05-24T21:13:54.439 DEBUG:boto:auth_path=/qskrps-16/obj?versionId=mWp-FBuLKZQQhyq1w2NLI1gYdoFzB36 2023-05-24T21:13:54.439 DEBUG:boto:Method: GET 2023-05-24T21:13:54.440 DEBUG:boto:Path: /qskrps-16/obj?versionId=mWp-FBuLKZQQhyq1w2NLI1gYdoFzB36 2023-05-24T21:13:54.440 DEBUG:boto:Data: 2023-05-24T21:13:54.441 DEBUG:boto:Headers: {} 2023-05-24T21:13:54.441 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:13:54.441 DEBUG:boto:Port: 8000 2023-05-24T21:13:54.442 DEBUG:boto:Params: {} 2023-05-24T21:13:54.442 DEBUG:boto:Token: None 2023-05-24T21:13:54.443 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:13:54 GMT /qskrps-16/obj?versionId=mWp-FBuLKZQQhyq1w2NLI1gYdoFzB36 2023-05-24T21:13:54.443 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:LxA6ah79Idt4zDx15/7OMQEEpdc= 2023-05-24T21:13:54.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:LxA6ah79Idt4zDx15/7OMQEEpdc=', 'Content-Length': '0'} 2023-05-24T21:13:54.445 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:13:13 GMT'), ('x-amz-version-id', 'mWp-FBuLKZQQhyq1w2NLI1gYdoFzB36'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009a9a2ebb167dded3-00646e7e12-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:13:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:54.446 DEBUG:boto:path=/qskrps-16/ 2023-05-24T21:13:54.446 DEBUG:boto:auth_path=/qskrps-16/ 2023-05-24T21:13:54.447 DEBUG:boto:path=/qskrps-16/?prefix=obj 2023-05-24T21:13:54.447 DEBUG:boto:auth_path=/qskrps-16/?prefix=obj 2023-05-24T21:13:54.448 DEBUG:boto:Method: GET 2023-05-24T21:13:54.448 DEBUG:boto:Path: /qskrps-16/?prefix=obj 2023-05-24T21:13:54.448 DEBUG:boto:Data: 2023-05-24T21:13:54.449 DEBUG:boto:Headers: {} 2023-05-24T21:13:54.449 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:13:54.450 DEBUG:boto:Port: 8001 2023-05-24T21:13:54.450 DEBUG:boto:Params: {} 2023-05-24T21:13:54.451 DEBUG:boto:Token: None 2023-05-24T21:13:54.451 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:13:54 GMT /qskrps-16/ 2023-05-24T21:13:54.452 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:noKxI27UvcMlrkC1UMBvezqEbfs= 2023-05-24T21:13:54.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:noKxI27UvcMlrkC1UMBvezqEbfs=', 'Content-Length': '0'} 2023-05-24T21:13:54.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de92ceca7644dad3-00646e7e12-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:13:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:54.455 DEBUG:boto:b'qskrps-16obj1000false' 2023-05-24T21:13:54.456 DEBUG:boto:path=/qskrps-16/ 2023-05-24T21:13:54.457 DEBUG:boto:auth_path=/qskrps-16/ 2023-05-24T21:13:54.458 DEBUG:boto:path=/qskrps-16/?prefix=obj 2023-05-24T21:13:54.459 DEBUG:boto:auth_path=/qskrps-16/?prefix=obj 2023-05-24T21:13:54.460 DEBUG:boto:Method: GET 2023-05-24T21:13:54.461 DEBUG:boto:Path: /qskrps-16/?prefix=obj 2023-05-24T21:13:54.461 DEBUG:boto:Data: 2023-05-24T21:13:54.462 DEBUG:boto:Headers: {} 2023-05-24T21:13:54.462 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:13:54.463 DEBUG:boto:Port: 8000 2023-05-24T21:13:54.463 DEBUG:boto:Params: {} 2023-05-24T21:13:54.464 DEBUG:boto:Token: None 2023-05-24T21:13:54.464 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:13:54 GMT /qskrps-16/ 2023-05-24T21:13:54.465 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:noKxI27UvcMlrkC1UMBvezqEbfs= 2023-05-24T21:13:54.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:noKxI27UvcMlrkC1UMBvezqEbfs=', 'Content-Length': '0'} 2023-05-24T21:13:54.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cac11c5b6d586569-00646e7e12-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:13:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:54.468 DEBUG:boto:b'qskrps-16obj1000false' 2023-05-24T21:13:54.469 DEBUG:boto:path=/qskrps-16/obj 2023-05-24T21:13:54.469 DEBUG:boto:auth_path=/qskrps-16/obj 2023-05-24T21:13:54.469 DEBUG:boto:Method: HEAD 2023-05-24T21:13:54.470 DEBUG:boto:Path: /qskrps-16/obj 2023-05-24T21:13:54.470 DEBUG:boto:Data: 2023-05-24T21:13:54.471 DEBUG:boto:Headers: {} 2023-05-24T21:13:54.471 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:13:54.472 DEBUG:boto:Port: 8001 2023-05-24T21:13:54.472 DEBUG:boto:Params: {} 2023-05-24T21:13:54.473 DEBUG:boto:Token: None 2023-05-24T21:13:54.473 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:13:54 GMT /qskrps-16/obj 2023-05-24T21:13:54.474 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ajEuq8X3uhngfTZNAc/izuKyids= 2023-05-24T21:13:54.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ajEuq8X3uhngfTZNAc/izuKyids=', 'Content-Length': '0'} 2023-05-24T21:13:54.476 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000007512603fbe9252d4-00646e7e12-1111-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:13:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:54.477 DEBUG:boto:path=/qskrps-16/obj 2023-05-24T21:13:54.477 DEBUG:boto:auth_path=/qskrps-16/obj 2023-05-24T21:13:54.477 DEBUG:boto:Method: HEAD 2023-05-24T21:13:54.478 DEBUG:boto:Path: /qskrps-16/obj 2023-05-24T21:13:54.478 DEBUG:boto:Data: 2023-05-24T21:13:54.478 DEBUG:boto:Headers: {} 2023-05-24T21:13:54.479 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:13:54.479 DEBUG:boto:Port: 8000 2023-05-24T21:13:54.479 DEBUG:boto:Params: {} 2023-05-24T21:13:54.480 DEBUG:boto:Token: None 2023-05-24T21:13:54.481 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:13:54 GMT /qskrps-16/obj 2023-05-24T21:13:54.481 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ajEuq8X3uhngfTZNAc/izuKyids= 2023-05-24T21:13:54.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ajEuq8X3uhngfTZNAc/izuKyids=', 'Content-Length': '0'} 2023-05-24T21:13:54.484 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000326799d715ac3425-00646e7e12-110d-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:13:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:54.485 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qskrps-16 zones={a2, a1} 2023-05-24T21:13:54.486 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_delete_marker_full_sync ... ok 2023-05-24T21:13:54.487 INFO:rgw_multi.tests:create bucket zone=a1 name=qskrps-17 2023-05-24T21:13:54.488 DEBUG:boto:path=/qskrps-17/ 2023-05-24T21:13:54.489 DEBUG:boto:auth_path=/qskrps-17/ 2023-05-24T21:13:54.489 DEBUG:boto:Method: PUT 2023-05-24T21:13:54.490 DEBUG:boto:Path: /qskrps-17/ 2023-05-24T21:13:54.490 DEBUG:boto:Data: 2023-05-24T21:13:54.490 DEBUG:boto:Headers: {} 2023-05-24T21:13:54.491 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:13:54.491 DEBUG:boto:Port: 8000 2023-05-24T21:13:54.492 DEBUG:boto:Params: {} 2023-05-24T21:13:54.492 DEBUG:boto:Token: None 2023-05-24T21:13:54.492 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:13:54 GMT /qskrps-17/ 2023-05-24T21:13:54.493 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:NGmkmEKemw5YRmbX+WWmBYh7fM0= 2023-05-24T21:13:54.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:NGmkmEKemw5YRmbX+WWmBYh7fM0=', 'Content-Length': '0'} 2023-05-24T21:13:54.507 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b5d6df370458729b-00646e7e12-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:13:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:54.508 INFO:rgw_multi.tests:create bucket zone=a2 name=qskrps-18 2023-05-24T21:13:54.508 DEBUG:boto:path=/qskrps-18/ 2023-05-24T21:13:54.509 DEBUG:boto:auth_path=/qskrps-18/ 2023-05-24T21:13:54.509 DEBUG:boto:Method: PUT 2023-05-24T21:13:54.510 DEBUG:boto:Path: /qskrps-18/ 2023-05-24T21:13:54.510 DEBUG:boto:Data: 2023-05-24T21:13:54.510 DEBUG:boto:Headers: {} 2023-05-24T21:13:54.511 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:13:54.511 DEBUG:boto:Port: 8001 2023-05-24T21:13:54.512 DEBUG:boto:Params: {} 2023-05-24T21:13:54.512 DEBUG:boto:Token: None 2023-05-24T21:13:54.512 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:13:54 GMT /qskrps-18/ 2023-05-24T21:13:54.513 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:6z1Wm6OHf38xah8rbvIqKcYq6Ww= 2023-05-24T21:13:54.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:6z1Wm6OHf38xah8rbvIqKcYq6Ww=', 'Content-Length': '0'} 2023-05-24T21:13:54.548 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a31d6ba23497b754-00646e7e12-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:13:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:54.549 DEBUG:boto:path=/qskrps-17/ 2023-05-24T21:13:54.549 DEBUG:boto:auth_path=/qskrps-17/ 2023-05-24T21:13:54.550 DEBUG:boto:path=/qskrps-17/?versioning 2023-05-24T21:13:54.550 DEBUG:boto:auth_path=/qskrps-17/?versioning 2023-05-24T21:13:54.550 DEBUG:boto:Method: PUT 2023-05-24T21:13:54.551 DEBUG:boto:Path: /qskrps-17/?versioning 2023-05-24T21:13:54.551 DEBUG:boto:Data: Enabled Disabled 2023-05-24T21:13:54.551 DEBUG:boto:Headers: {} 2023-05-24T21:13:54.552 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:13:54.552 DEBUG:boto:Port: 8000 2023-05-24T21:13:54.552 DEBUG:boto:Params: {} 2023-05-24T21:13:54.553 DEBUG:boto:Token: None 2023-05-24T21:13:54.553 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:13:54 GMT /qskrps-17/?versioning 2023-05-24T21:13:54.554 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:FQJ1+9GZIcFpFs0TDVvZxYL6mlE= 2023-05-24T21:13:54.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:FQJ1+9GZIcFpFs0TDVvZxYL6mlE=', 'Content-Length': '228'} 2023-05-24T21:13:54.562 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000eaa5f1d1beb9dde1-00646e7e12-110d-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:13:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:54.563 DEBUG:boto:path=/qskrps-17/ 2023-05-24T21:13:54.563 DEBUG:boto:auth_path=/qskrps-17/ 2023-05-24T21:13:54.563 DEBUG:boto:path=/qskrps-17/?versioning 2023-05-24T21:13:54.564 DEBUG:boto:auth_path=/qskrps-17/?versioning 2023-05-24T21:13:54.564 DEBUG:boto:Method: PUT 2023-05-24T21:13:54.564 DEBUG:boto:Path: /qskrps-17/?versioning 2023-05-24T21:13:54.565 DEBUG:boto:Data: Suspended Disabled 2023-05-24T21:13:54.565 DEBUG:boto:Headers: {} 2023-05-24T21:13:54.565 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:13:54.566 DEBUG:boto:Port: 8000 2023-05-24T21:13:54.566 DEBUG:boto:Params: {} 2023-05-24T21:13:54.566 DEBUG:boto:Token: None 2023-05-24T21:13:54.567 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:13:54 GMT /qskrps-17/?versioning 2023-05-24T21:13:54.567 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:FQJ1+9GZIcFpFs0TDVvZxYL6mlE= 2023-05-24T21:13:54.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:FQJ1+9GZIcFpFs0TDVvZxYL6mlE=', 'Content-Length': '230'} 2023-05-24T21:13:54.572 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000177b476e5f308b2d-00646e7e12-110d-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:13:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:54.573 DEBUG:boto:path=/qskrps-18/ 2023-05-24T21:13:54.573 DEBUG:boto:auth_path=/qskrps-18/ 2023-05-24T21:13:54.573 DEBUG:boto:path=/qskrps-18/?versioning 2023-05-24T21:13:54.574 DEBUG:boto:auth_path=/qskrps-18/?versioning 2023-05-24T21:13:54.574 DEBUG:boto:Method: PUT 2023-05-24T21:13:54.575 DEBUG:boto:Path: /qskrps-18/?versioning 2023-05-24T21:13:54.575 DEBUG:boto:Data: Enabled Disabled 2023-05-24T21:13:54.575 DEBUG:boto:Headers: {} 2023-05-24T21:13:54.576 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:13:54.576 DEBUG:boto:Port: 8001 2023-05-24T21:13:54.576 DEBUG:boto:Params: {} 2023-05-24T21:13:54.577 DEBUG:boto:Token: None 2023-05-24T21:13:54.577 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:13:54 GMT /qskrps-18/?versioning 2023-05-24T21:13:54.578 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:s2e3VdYtlhMH29MPKP/rPwXTERs= 2023-05-24T21:13:54.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:s2e3VdYtlhMH29MPKP/rPwXTERs=', 'Content-Length': '228'} 2023-05-24T21:13:54.587 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a54bdc35199f7d0f-00646e7e12-1111-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:13:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:54.587 DEBUG:boto:path=/qskrps-18/ 2023-05-24T21:13:54.588 DEBUG:boto:auth_path=/qskrps-18/ 2023-05-24T21:13:54.588 DEBUG:boto:path=/qskrps-18/?versioning 2023-05-24T21:13:54.589 DEBUG:boto:auth_path=/qskrps-18/?versioning 2023-05-24T21:13:54.589 DEBUG:boto:Method: PUT 2023-05-24T21:13:54.590 DEBUG:boto:Path: /qskrps-18/?versioning 2023-05-24T21:13:54.590 DEBUG:boto:Data: Suspended Disabled 2023-05-24T21:13:54.591 DEBUG:boto:Headers: {} 2023-05-24T21:13:54.591 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:13:54.591 DEBUG:boto:Port: 8001 2023-05-24T21:13:54.592 DEBUG:boto:Params: {} 2023-05-24T21:13:54.592 DEBUG:boto:Token: None 2023-05-24T21:13:54.592 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:13:54 GMT /qskrps-18/?versioning 2023-05-24T21:13:54.593 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:s2e3VdYtlhMH29MPKP/rPwXTERs= 2023-05-24T21:13:54.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:54 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:s2e3VdYtlhMH29MPKP/rPwXTERs=', 'Content-Length': '230'} 2023-05-24T21:13:54.601 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c363196930a96c41-00646e7e12-1111-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:13:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:54.601 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:13:54.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:13:54.602 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:13:54.651 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:13:54.651 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:13:54.692 INFO:teuthology.orchestra.run.smithi071.stderr:No --period given, using current period=5c04936b-70ea-4415-b4eb-3aad3dc612e5 2023-05-24T21:13:54.709 INFO:teuthology.orchestra.run.smithi071.stdout:[ 2023-05-24T21:13:54.710 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:13:54.710 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962834.592955_179.1", 2023-05-24T21:13:54.710 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:13:54.592955Z" 2023-05-24T21:13:54.710 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:13:54.711 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:13:54.711 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:13:54.711 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:06:32.333844Z" 2023-05-24T21:13:54.711 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:13:54.711 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:13:54.711 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:13:54.711 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:13:54.712 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:13:54.712 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:13:54.712 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:13:54.712 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:13:54.712 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:13:54.713 INFO:teuthology.orchestra.run.smithi071.stdout:] 2023-05-24T21:13:54.713 DEBUG:rgw_multi.tests:master meta markers={0: '1_1684962834.592955_179.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:13:54.714 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-24T21:13:54.714 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:13:54.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:13:54.715 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:13:54.771 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:13:54.772 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:13:54.826 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:13:54.820+0000 7f5c77936780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:13:54.826 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:13:54.821+0000 7f5c77936780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:13:54.838 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:13:54.838 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:13:54.838 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:13:54.839 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:13:54.839 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:13:54.839 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:13:54.839 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:13:54.839 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:13:54.839 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:13:54.839 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:13:54.840 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:13:54.840 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:13:54.840 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:13:54.840 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962834.517321_168.1", 2023-05-24T21:13:54.840 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:13:54.840 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:13:54.841 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:13:54.841 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:13:54.517321Z", 2023-05-24T21:13:54.841 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:13:54.841 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:13:54.841 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:13:54.841 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:13:54.841 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:13:54.842 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:13:54.842 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:13:54.842 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:13:54.842 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:13:54.842 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:13:54.842 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:13:54.842 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:13:54.843 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:13:54.843 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:13:54.843 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:13:54.843 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:13:54.843 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:13:54.843 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:13:54.843 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:13:54.844 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:13:54.844 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:13:54.844 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:13:54.844 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:13:54.844 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:13:54.844 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:13:54.845 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:13:54.845 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:13:54.845 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:13:54.845 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:13:54.845 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:13:54.845 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:13:54.845 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:13:54.846 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:13:54.846 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:13:54.846 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:13:54.846 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:13:54.846 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:13:54.846 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:13:54.846 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:13:54.847 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:13:54.847 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:13:54.847 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:13:54.847 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:13:54.847 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:13:54.847 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:13:54.848 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:13:54Z" 2023-05-24T21:13:54.848 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:13:54.849 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962834.517321_168.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:13:54.517321Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:13:54Z" } 2023-05-24T21:13:54.849 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962834.517321_168.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:13:54.517321Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:13:54.850 DEBUG:rgw_multi.tests:log_status={0: '1_1684962834.592955_179.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:13:54.850 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962834.517321_168.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:13:54.851 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684962834.592955_179.1 target=1_1684962834.517321_168.1 2023-05-24T21:13:59.852 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:13:59.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:13:59.853 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:13:59.910 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:13:59.911 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:13:59.953 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:13:59.947+0000 7f2245894780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:13:59.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:13:59.948+0000 7f2245894780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:13:59.976 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:13:59.976 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:13:59.976 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:13:59.976 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:13:59.977 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:13:59.977 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:13:59.977 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:13:59.977 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:13:59.977 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:13:59.978 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:13:59.978 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:13:59.978 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:13:59.978 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:13:59.978 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962834.592955_179.1", 2023-05-24T21:13:59.979 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:13:59.979 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:13:59.979 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:13:59.979 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:13:54.592955Z", 2023-05-24T21:13:59.979 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:13:59.979 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:13:59.980 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:13:59.980 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:13:59.980 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:13:59.980 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:13:59.980 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:13:59.980 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:13:59.981 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:13:59.981 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:13:59.981 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:13:59.981 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:13:59.981 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:13:59.981 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:13:59.981 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:13:59.982 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:13:59.982 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:13:59.982 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:13:59.982 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:13:59.982 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:13:59.983 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:13:59.983 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:13:59.983 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:13:59.983 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:13:59.983 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:13:59.983 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:13:59.983 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:13:59.984 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:13:59.984 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:13:59.984 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:13:59.984 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:13:59.984 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:13:59.985 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:13:59.985 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:13:59.985 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:13:59.985 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:13:59.985 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:13:59.985 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:13:59.986 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:13:59.986 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:13:59.986 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:13:59.986 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:13:59.987 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:13:59.987 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:13:59.987 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:13:59.987 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:13:59Z" 2023-05-24T21:13:59.987 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:13:59.989 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962834.592955_179.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:13:54.592955Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:13:59Z" } 2023-05-24T21:13:59.989 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962834.592955_179.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:13:54.592955Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:13:59.990 DEBUG:rgw_multi.tests:log_status={0: '1_1684962834.592955_179.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:13:59.990 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962834.592955_179.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:13:59.990 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-24T21:13:59.991 DEBUG:boto:path=/qskrps-17/ 2023-05-24T21:13:59.991 DEBUG:boto:auth_path=/qskrps-17/ 2023-05-24T21:13:59.991 DEBUG:boto:Method: HEAD 2023-05-24T21:13:59.992 DEBUG:boto:Path: /qskrps-17/ 2023-05-24T21:13:59.992 DEBUG:boto:Data: 2023-05-24T21:13:59.992 DEBUG:boto:Headers: {} 2023-05-24T21:13:59.993 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:13:59.993 DEBUG:boto:Port: 8000 2023-05-24T21:13:59.993 DEBUG:boto:Params: {} 2023-05-24T21:13:59.994 DEBUG:boto:Token: None 2023-05-24T21:13:59.994 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:13:59 GMT /qskrps-17/ 2023-05-24T21:13:59.995 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:0u5oKZ5eGTZJ22ve0jG4EwqcNms= 2023-05-24T21:13:59.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:59 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:0u5oKZ5eGTZJ22ve0jG4EwqcNms=', 'Content-Length': '0'} 2023-05-24T21:13:59.997 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002eaa695b312cb877-00646e7e17-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:13:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:13:59.998 DEBUG:boto:path=/qskrps-17/obj 2023-05-24T21:13:59.998 DEBUG:boto:auth_path=/qskrps-17/obj 2023-05-24T21:13:59.999 DEBUG:boto:Method: PUT 2023-05-24T21:13:59.999 DEBUG:boto:Path: /qskrps-17/obj 2023-05-24T21:14:00.000 DEBUG:boto:Data: 2023-05-24T21:14:00.000 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-05-24T21:14:00.000 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:14:00.001 DEBUG:boto:Port: 8000 2023-05-24T21:14:00.001 DEBUG:boto:Params: {} 2023-05-24T21:14:00.001 DEBUG:boto:Token: None 2023-05-24T21:14:00.002 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Wed, 24 May 2023 21:14:00 GMT /qskrps-17/obj 2023-05-24T21:14:00.002 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98://zbT81MZQqhPFx+E2URSW5XeX4= 2023-05-24T21:14:00.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:14:00 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98://zbT81MZQqhPFx+E2URSW5XeX4='} 2023-05-24T21:14:00.011 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000a651ef540129e21-00646e7e17-110d-a1'), ('Date', 'Wed, 24 May 2023 21:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:00.012 DEBUG:boto:path=/qskrps-17/ 2023-05-24T21:14:00.013 DEBUG:boto:auth_path=/qskrps-17/ 2023-05-24T21:14:00.013 DEBUG:boto:Method: HEAD 2023-05-24T21:14:00.013 DEBUG:boto:Path: /qskrps-17/ 2023-05-24T21:14:00.014 DEBUG:boto:Data: 2023-05-24T21:14:00.014 DEBUG:boto:Headers: {} 2023-05-24T21:14:00.015 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:14:00.015 DEBUG:boto:Port: 8000 2023-05-24T21:14:00.016 DEBUG:boto:Params: {} 2023-05-24T21:14:00.016 DEBUG:boto:Token: None 2023-05-24T21:14:00.017 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:14:00 GMT /qskrps-17/ 2023-05-24T21:14:00.017 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:cVabWngv0r877wZrZCQLScsoC4Y= 2023-05-24T21:14:00.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:00 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:cVabWngv0r877wZrZCQLScsoC4Y=', 'Content-Length': '0'} 2023-05-24T21:14:00.020 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000090fe0b0e6919fdf5-00646e7e18-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:00.020 DEBUG:boto:path=/qskrps-17/obj 2023-05-24T21:14:00.021 DEBUG:boto:auth_path=/qskrps-17/obj 2023-05-24T21:14:00.021 DEBUG:boto:Method: DELETE 2023-05-24T21:14:00.022 DEBUG:boto:Path: /qskrps-17/obj 2023-05-24T21:14:00.022 DEBUG:boto:Data: 2023-05-24T21:14:00.022 DEBUG:boto:Headers: {} 2023-05-24T21:14:00.023 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:14:00.023 DEBUG:boto:Port: 8000 2023-05-24T21:14:00.023 DEBUG:boto:Params: {} 2023-05-24T21:14:00.024 DEBUG:boto:Token: None 2023-05-24T21:14:00.024 DEBUG:boto:StringToSign: DELETE Wed, 24 May 2023 21:14:00 GMT /qskrps-17/obj 2023-05-24T21:14:00.025 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:FpMLjivPvhWg2HOwn8nCGBX4MjM= 2023-05-24T21:14:00.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:00 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:FpMLjivPvhWg2HOwn8nCGBX4MjM=', 'Content-Length': '0'} 2023-05-24T21:14:00.043 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx000004af2bce160d35128-00646e7e18-110d-a1'), ('Date', 'Wed, 24 May 2023 21:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:00.044 DEBUG:boto:path=/qskrps-18/ 2023-05-24T21:14:00.045 DEBUG:boto:auth_path=/qskrps-18/ 2023-05-24T21:14:00.045 DEBUG:boto:Method: HEAD 2023-05-24T21:14:00.046 DEBUG:boto:Path: /qskrps-18/ 2023-05-24T21:14:00.046 DEBUG:boto:Data: 2023-05-24T21:14:00.047 DEBUG:boto:Headers: {} 2023-05-24T21:14:00.047 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:14:00.048 DEBUG:boto:Port: 8001 2023-05-24T21:14:00.048 DEBUG:boto:Params: {} 2023-05-24T21:14:00.049 DEBUG:boto:Token: None 2023-05-24T21:14:00.049 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:14:00 GMT /qskrps-18/ 2023-05-24T21:14:00.049 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:x/ANDAD6XmBA4IGLY/aJdqZ+NhQ= 2023-05-24T21:14:00.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:00 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:x/ANDAD6XmBA4IGLY/aJdqZ+NhQ=', 'Content-Length': '0'} 2023-05-24T21:14:00.052 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e0cfe8a4e02925c8-00646e7e18-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:00.053 DEBUG:boto:path=/qskrps-18/obj 2023-05-24T21:14:00.053 DEBUG:boto:auth_path=/qskrps-18/obj 2023-05-24T21:14:00.054 DEBUG:boto:Method: PUT 2023-05-24T21:14:00.054 DEBUG:boto:Path: /qskrps-18/obj 2023-05-24T21:14:00.054 DEBUG:boto:Data: 2023-05-24T21:14:00.055 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-05-24T21:14:00.055 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:14:00.056 DEBUG:boto:Port: 8001 2023-05-24T21:14:00.056 DEBUG:boto:Params: {} 2023-05-24T21:14:00.057 DEBUG:boto:Token: None 2023-05-24T21:14:00.057 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Wed, 24 May 2023 21:14:00 GMT /qskrps-18/obj 2023-05-24T21:14:00.058 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:PFyKw51cNesyxURA4qlrdnZajbw= 2023-05-24T21:14:00.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:14:00 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PFyKw51cNesyxURA4qlrdnZajbw='} 2023-05-24T21:14:00.066 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000803b5f937314ed8-00646e7e18-1111-a2'), ('Date', 'Wed, 24 May 2023 21:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:00.067 DEBUG:boto:path=/qskrps-18/ 2023-05-24T21:14:00.068 DEBUG:boto:auth_path=/qskrps-18/ 2023-05-24T21:14:00.069 DEBUG:boto:Method: HEAD 2023-05-24T21:14:00.070 DEBUG:boto:Path: /qskrps-18/ 2023-05-24T21:14:00.071 DEBUG:boto:Data: 2023-05-24T21:14:00.072 DEBUG:boto:Headers: {} 2023-05-24T21:14:00.072 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:14:00.073 DEBUG:boto:Port: 8001 2023-05-24T21:14:00.074 DEBUG:boto:Params: {} 2023-05-24T21:14:00.075 DEBUG:boto:Token: None 2023-05-24T21:14:00.076 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:14:00 GMT /qskrps-18/ 2023-05-24T21:14:00.076 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:x/ANDAD6XmBA4IGLY/aJdqZ+NhQ= 2023-05-24T21:14:00.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:00 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:x/ANDAD6XmBA4IGLY/aJdqZ+NhQ=', 'Content-Length': '0'} 2023-05-24T21:14:00.080 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b4e4499d1bab651e-00646e7e18-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:00.081 DEBUG:boto:path=/qskrps-18/obj 2023-05-24T21:14:00.081 DEBUG:boto:auth_path=/qskrps-18/obj 2023-05-24T21:14:00.082 DEBUG:boto:Method: DELETE 2023-05-24T21:14:00.083 DEBUG:boto:Path: /qskrps-18/obj 2023-05-24T21:14:00.084 DEBUG:boto:Data: 2023-05-24T21:14:00.084 DEBUG:boto:Headers: {} 2023-05-24T21:14:00.085 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:14:00.086 DEBUG:boto:Port: 8001 2023-05-24T21:14:00.087 DEBUG:boto:Params: {} 2023-05-24T21:14:00.087 DEBUG:boto:Token: None 2023-05-24T21:14:00.088 DEBUG:boto:StringToSign: DELETE Wed, 24 May 2023 21:14:00 GMT /qskrps-18/obj 2023-05-24T21:14:00.089 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4V8mY7zZSdmZG/j33r4meB+Qj/4= 2023-05-24T21:14:00.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:00 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4V8mY7zZSdmZG/j33r4meB+Qj/4=', 'Content-Length': '0'} 2023-05-24T21:14:00.103 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx0000007cfa9d185ed94b8-00646e7e18-1111-a2'), ('Date', 'Wed, 24 May 2023 21:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:00.104 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=qskrps-17 2023-05-24T21:14:00.105 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-17', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:14:00.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-17', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:14:00.106 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-17 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:14:00.163 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:14:00.164 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:14:00.198 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:14:00.192+0000 7f5a122e2780 1 waiting to reach incremental sync.. 2023-05-24T21:14:05.199 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:14:05.193+0000 7f5a122e2780 1 waiting to reach incremental sync.. 2023-05-24T21:14:10.200 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:14:10.194+0000 7f5a122e2780 1 waiting to reach incremental sync.. 2023-05-24T21:14:15.202 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:14:15.196+0000 7f5a122e2780 1 bucket sync caught up with source: 2023-05-24T21:14:15.202 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000003.7.7, , ] 2023-05-24T21:14:15.202 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000003.7.7, , ] 2023-05-24T21:14:15.202 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:14:15.196+0000 7f5a122e2780 0 bucket checkpoint complete 2023-05-24T21:14:15.222 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=qskrps-17 2023-05-24T21:14:15.223 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-17', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:14:15.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-17', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:14:15.224 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-17 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:14:15.287 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:14:15.287 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:14:15.324 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:14:15.318+0000 7fdb3e45c780 1 waiting to reach incremental sync.. 2023-05-24T21:14:20.324 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:14:20.317+0000 7fdb3e45c780 1 waiting to reach incremental sync.. 2023-05-24T21:14:25.325 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:14:25.318+0000 7fdb3e45c780 1 waiting to reach incremental sync.. 2023-05-24T21:14:30.325 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:14:30.319+0000 7fdb3e45c780 1 waiting to reach incremental sync.. 2023-05-24T21:14:35.327 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:14:35.320+0000 7fdb3e45c780 1 bucket sync caught up with source: 2023-05-24T21:14:35.328 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000001.6.13, , ] 2023-05-24T21:14:35.328 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000001.6.13, , ] 2023-05-24T21:14:35.328 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:14:35.320+0000 7fdb3e45c780 0 bucket checkpoint complete 2023-05-24T21:14:35.347 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qskrps-17 zones={a2, a1} 2023-05-24T21:14:35.347 DEBUG:boto:path=/qskrps-17/ 2023-05-24T21:14:35.348 DEBUG:boto:auth_path=/qskrps-17/ 2023-05-24T21:14:35.348 DEBUG:boto:Method: HEAD 2023-05-24T21:14:35.348 DEBUG:boto:Path: /qskrps-17/ 2023-05-24T21:14:35.349 DEBUG:boto:Data: 2023-05-24T21:14:35.349 DEBUG:boto:Headers: {} 2023-05-24T21:14:35.349 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:14:35.350 DEBUG:boto:Port: 8001 2023-05-24T21:14:35.350 DEBUG:boto:Params: {} 2023-05-24T21:14:35.351 DEBUG:boto:Token: None 2023-05-24T21:14:35.352 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:14:35 GMT /qskrps-17/ 2023-05-24T21:14:35.352 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:u+f2qndApQSGoEzwq/0CBB1vwTM= 2023-05-24T21:14:35.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:u+f2qndApQSGoEzwq/0CBB1vwTM=', 'Content-Length': '0'} 2023-05-24T21:14:35.355 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003ae9aa4819af122f-00646e7e3b-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:14:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:35.356 DEBUG:boto:path=/qskrps-17/ 2023-05-24T21:14:35.356 DEBUG:boto:auth_path=/qskrps-17/ 2023-05-24T21:14:35.357 DEBUG:boto:Method: HEAD 2023-05-24T21:14:35.357 DEBUG:boto:Path: /qskrps-17/ 2023-05-24T21:14:35.357 DEBUG:boto:Data: 2023-05-24T21:14:35.358 DEBUG:boto:Headers: {} 2023-05-24T21:14:35.358 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:14:35.358 DEBUG:boto:Port: 8000 2023-05-24T21:14:35.359 DEBUG:boto:Params: {} 2023-05-24T21:14:35.359 DEBUG:boto:Token: None 2023-05-24T21:14:35.360 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:14:35 GMT /qskrps-17/ 2023-05-24T21:14:35.360 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:u+f2qndApQSGoEzwq/0CBB1vwTM= 2023-05-24T21:14:35.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:u+f2qndApQSGoEzwq/0CBB1vwTM=', 'Content-Length': '0'} 2023-05-24T21:14:35.363 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000085e4f25ddc363c68-00646e7e3b-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:14:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:35.363 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-24T21:14:35.364 DEBUG:boto:path=/qskrps-17/ 2023-05-24T21:14:35.364 DEBUG:boto:auth_path=/qskrps-17/ 2023-05-24T21:14:35.365 DEBUG:boto:path=/qskrps-17/?versions&max-keys=999 2023-05-24T21:14:35.365 DEBUG:boto:auth_path=/qskrps-17/?versions&max-keys=999 2023-05-24T21:14:35.365 DEBUG:boto:Method: GET 2023-05-24T21:14:35.366 DEBUG:boto:Path: /qskrps-17/?versions&max-keys=999 2023-05-24T21:14:35.366 DEBUG:boto:Data: 2023-05-24T21:14:35.366 DEBUG:boto:Headers: {} 2023-05-24T21:14:35.367 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:14:35.367 DEBUG:boto:Port: 8001 2023-05-24T21:14:35.367 DEBUG:boto:Params: {} 2023-05-24T21:14:35.368 DEBUG:boto:Token: None 2023-05-24T21:14:35.368 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:14:35 GMT /qskrps-17/?versions 2023-05-24T21:14:35.369 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:am1pe9z9/bUbIsC4WMS9ZRPZPfw= 2023-05-24T21:14:35.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:am1pe9z9/bUbIsC4WMS9ZRPZPfw=', 'Content-Length': '0'} 2023-05-24T21:14:35.372 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006902211ff192a734-00646e7e3b-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:14:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:35.372 DEBUG:boto:b'qskrps-17999falseobjnulltrue2023-05-24T21:14:00.022Zrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:14:35.373 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:14:35.373 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-24T21:14:35.374 DEBUG:boto:path=/qskrps-17/ 2023-05-24T21:14:35.374 DEBUG:boto:auth_path=/qskrps-17/ 2023-05-24T21:14:35.374 DEBUG:boto:path=/qskrps-17/?versions&max-keys=999 2023-05-24T21:14:35.375 DEBUG:boto:auth_path=/qskrps-17/?versions&max-keys=999 2023-05-24T21:14:35.375 DEBUG:boto:Method: GET 2023-05-24T21:14:35.376 DEBUG:boto:Path: /qskrps-17/?versions&max-keys=999 2023-05-24T21:14:35.376 DEBUG:boto:Data: 2023-05-24T21:14:35.376 DEBUG:boto:Headers: {} 2023-05-24T21:14:35.377 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:14:35.377 DEBUG:boto:Port: 8000 2023-05-24T21:14:35.378 DEBUG:boto:Params: {} 2023-05-24T21:14:35.378 DEBUG:boto:Token: None 2023-05-24T21:14:35.378 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:14:35 GMT /qskrps-17/?versions 2023-05-24T21:14:35.379 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:am1pe9z9/bUbIsC4WMS9ZRPZPfw= 2023-05-24T21:14:35.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:am1pe9z9/bUbIsC4WMS9ZRPZPfw=', 'Content-Length': '0'} 2023-05-24T21:14:35.381 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051d3b898feab01c1-00646e7e3b-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:14:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:35.382 DEBUG:boto:b'qskrps-17999falseobjnulltrue2023-05-24T21:14:00.022Zrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:14:35.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:14:35.383 DEBUG:boto:path=/qskrps-17/ 2023-05-24T21:14:35.383 DEBUG:boto:auth_path=/qskrps-17/ 2023-05-24T21:14:35.384 DEBUG:boto:path=/qskrps-17/?versions&max-keys=999 2023-05-24T21:14:35.384 DEBUG:boto:auth_path=/qskrps-17/?versions&max-keys=999 2023-05-24T21:14:35.384 DEBUG:boto:Method: GET 2023-05-24T21:14:35.385 DEBUG:boto:Path: /qskrps-17/?versions&max-keys=999 2023-05-24T21:14:35.385 DEBUG:boto:Data: 2023-05-24T21:14:35.385 DEBUG:boto:Headers: {} 2023-05-24T21:14:35.386 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:14:35.386 DEBUG:boto:Port: 8001 2023-05-24T21:14:35.386 DEBUG:boto:Params: {} 2023-05-24T21:14:35.387 DEBUG:boto:Token: None 2023-05-24T21:14:35.387 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:14:35 GMT /qskrps-17/?versions 2023-05-24T21:14:35.388 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:am1pe9z9/bUbIsC4WMS9ZRPZPfw= 2023-05-24T21:14:35.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:am1pe9z9/bUbIsC4WMS9ZRPZPfw=', 'Content-Length': '0'} 2023-05-24T21:14:35.391 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da850d36c03c6788-00646e7e3b-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:14:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:35.391 DEBUG:boto:b'qskrps-17999falseobjnulltrue2023-05-24T21:14:00.022Zrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:14:35.392 DEBUG:boto:path=/qskrps-17/ 2023-05-24T21:14:35.392 DEBUG:boto:auth_path=/qskrps-17/ 2023-05-24T21:14:35.393 DEBUG:boto:path=/qskrps-17/?versions&max-keys=999 2023-05-24T21:14:35.393 DEBUG:boto:auth_path=/qskrps-17/?versions&max-keys=999 2023-05-24T21:14:35.393 DEBUG:boto:Method: GET 2023-05-24T21:14:35.394 DEBUG:boto:Path: /qskrps-17/?versions&max-keys=999 2023-05-24T21:14:35.394 DEBUG:boto:Data: 2023-05-24T21:14:35.394 DEBUG:boto:Headers: {} 2023-05-24T21:14:35.395 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:14:35.395 DEBUG:boto:Port: 8000 2023-05-24T21:14:35.395 DEBUG:boto:Params: {} 2023-05-24T21:14:35.396 DEBUG:boto:Token: None 2023-05-24T21:14:35.396 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:14:35 GMT /qskrps-17/?versions 2023-05-24T21:14:35.397 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:am1pe9z9/bUbIsC4WMS9ZRPZPfw= 2023-05-24T21:14:35.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:am1pe9z9/bUbIsC4WMS9ZRPZPfw=', 'Content-Length': '0'} 2023-05-24T21:14:35.399 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b540713e5e967eac-00646e7e3b-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:14:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:35.440 DEBUG:boto:b'qskrps-17999falseobjnulltrue2023-05-24T21:14:00.022Zrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:14:35.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:14:35.441 DEBUG:boto:path=/qskrps-17/obj 2023-05-24T21:14:35.442 DEBUG:boto:auth_path=/qskrps-17/obj 2023-05-24T21:14:35.442 DEBUG:boto:Method: HEAD 2023-05-24T21:14:35.443 DEBUG:boto:Path: /qskrps-17/obj 2023-05-24T21:14:35.443 DEBUG:boto:Data: 2023-05-24T21:14:35.443 DEBUG:boto:Headers: {} 2023-05-24T21:14:35.444 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:14:35.444 DEBUG:boto:Port: 8001 2023-05-24T21:14:35.444 DEBUG:boto:Params: {} 2023-05-24T21:14:35.445 DEBUG:boto:Token: None 2023-05-24T21:14:35.445 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:14:35 GMT /qskrps-17/obj 2023-05-24T21:14:35.447 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:JDvEDfYEI+lbDN8j79oBtE5VhfQ= 2023-05-24T21:14:35.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:JDvEDfYEI+lbDN8j79oBtE5VhfQ=', 'Content-Length': '0'} 2023-05-24T21:14:35.449 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000007c34c6fa69157b8e-00646e7e3b-1111-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:14:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:35.450 DEBUG:boto:path=/qskrps-17/obj 2023-05-24T21:14:35.451 DEBUG:boto:auth_path=/qskrps-17/obj 2023-05-24T21:14:35.452 DEBUG:boto:Method: HEAD 2023-05-24T21:14:35.453 DEBUG:boto:Path: /qskrps-17/obj 2023-05-24T21:14:35.454 DEBUG:boto:Data: 2023-05-24T21:14:35.455 DEBUG:boto:Headers: {} 2023-05-24T21:14:35.456 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:14:35.457 DEBUG:boto:Port: 8000 2023-05-24T21:14:35.457 DEBUG:boto:Params: {} 2023-05-24T21:14:35.458 DEBUG:boto:Token: None 2023-05-24T21:14:35.459 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:14:35 GMT /qskrps-17/obj 2023-05-24T21:14:35.460 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:JDvEDfYEI+lbDN8j79oBtE5VhfQ= 2023-05-24T21:14:35.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:JDvEDfYEI+lbDN8j79oBtE5VhfQ=', 'Content-Length': '0'} 2023-05-24T21:14:35.462 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000538e3f5b718b7f40-00646e7e3b-110d-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:14:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:35.463 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qskrps-17 zones={a2, a1} 2023-05-24T21:14:35.463 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=qskrps-18 2023-05-24T21:14:35.463 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-18', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:14:35.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-18', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:14:35.464 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-18 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:14:35.527 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:14:35.528 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:14:35.578 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:14:35.571+0000 7fd4ca379780 1 bucket sync caught up with source: 2023-05-24T21:14:35.578 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000001.9.13, , ] 2023-05-24T21:14:35.578 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000001.9.13, , ] 2023-05-24T21:14:35.578 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:14:35.571+0000 7fd4ca379780 0 bucket checkpoint complete 2023-05-24T21:14:35.606 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=qskrps-18 2023-05-24T21:14:35.607 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-18', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:14:35.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-18', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:14:35.608 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-18 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:14:35.664 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:14:35.665 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:14:35.721 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:14:35.714+0000 7f13f8644780 1 bucket sync caught up with source: 2023-05-24T21:14:35.721 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000003.24.7, , ] 2023-05-24T21:14:35.722 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000003.24.7, , ] 2023-05-24T21:14:35.722 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:14:35.714+0000 7f13f8644780 0 bucket checkpoint complete 2023-05-24T21:14:35.736 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qskrps-18 zones={a2, a1} 2023-05-24T21:14:35.736 DEBUG:boto:path=/qskrps-18/ 2023-05-24T21:14:35.737 DEBUG:boto:auth_path=/qskrps-18/ 2023-05-24T21:14:35.737 DEBUG:boto:Method: HEAD 2023-05-24T21:14:35.737 DEBUG:boto:Path: /qskrps-18/ 2023-05-24T21:14:35.738 DEBUG:boto:Data: 2023-05-24T21:14:35.738 DEBUG:boto:Headers: {} 2023-05-24T21:14:35.739 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:14:35.739 DEBUG:boto:Port: 8001 2023-05-24T21:14:35.739 DEBUG:boto:Params: {} 2023-05-24T21:14:35.740 DEBUG:boto:Token: None 2023-05-24T21:14:35.740 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:14:35 GMT /qskrps-18/ 2023-05-24T21:14:35.741 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:rTYCVsD5Cidc2TYySzKzyv4lVqU= 2023-05-24T21:14:35.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:rTYCVsD5Cidc2TYySzKzyv4lVqU=', 'Content-Length': '0'} 2023-05-24T21:14:35.743 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000010a69ac4115b5b62-00646e7e3b-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:14:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:35.744 DEBUG:boto:path=/qskrps-18/ 2023-05-24T21:14:35.744 DEBUG:boto:auth_path=/qskrps-18/ 2023-05-24T21:14:35.744 DEBUG:boto:Method: HEAD 2023-05-24T21:14:35.745 DEBUG:boto:Path: /qskrps-18/ 2023-05-24T21:14:35.745 DEBUG:boto:Data: 2023-05-24T21:14:35.745 DEBUG:boto:Headers: {} 2023-05-24T21:14:35.746 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:14:35.746 DEBUG:boto:Port: 8000 2023-05-24T21:14:35.747 DEBUG:boto:Params: {} 2023-05-24T21:14:35.747 DEBUG:boto:Token: None 2023-05-24T21:14:35.747 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:14:35 GMT /qskrps-18/ 2023-05-24T21:14:35.748 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:rTYCVsD5Cidc2TYySzKzyv4lVqU= 2023-05-24T21:14:35.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:rTYCVsD5Cidc2TYySzKzyv4lVqU=', 'Content-Length': '0'} 2023-05-24T21:14:35.750 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cf22f8fe45899a06-00646e7e3b-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:14:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:35.751 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-24T21:14:35.751 DEBUG:boto:path=/qskrps-18/ 2023-05-24T21:14:35.752 DEBUG:boto:auth_path=/qskrps-18/ 2023-05-24T21:14:35.752 DEBUG:boto:path=/qskrps-18/?versions&max-keys=999 2023-05-24T21:14:35.752 DEBUG:boto:auth_path=/qskrps-18/?versions&max-keys=999 2023-05-24T21:14:35.753 DEBUG:boto:Method: GET 2023-05-24T21:14:35.753 DEBUG:boto:Path: /qskrps-18/?versions&max-keys=999 2023-05-24T21:14:35.753 DEBUG:boto:Data: 2023-05-24T21:14:35.754 DEBUG:boto:Headers: {} 2023-05-24T21:14:35.754 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:14:35.754 DEBUG:boto:Port: 8001 2023-05-24T21:14:35.755 DEBUG:boto:Params: {} 2023-05-24T21:14:35.755 DEBUG:boto:Token: None 2023-05-24T21:14:35.756 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:14:35 GMT /qskrps-18/?versions 2023-05-24T21:14:35.757 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ysP5ucsRLN5Vwx8GM5FCQtUQCzo= 2023-05-24T21:14:35.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ysP5ucsRLN5Vwx8GM5FCQtUQCzo=', 'Content-Length': '0'} 2023-05-24T21:14:35.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005266d0326e436137-00646e7e3b-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:14:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:35.760 DEBUG:boto:b'qskrps-18999falseobjnulltrue2023-05-24T21:14:00.087Zrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:14:35.761 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:14:35.761 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-24T21:14:35.761 DEBUG:boto:path=/qskrps-18/ 2023-05-24T21:14:35.762 DEBUG:boto:auth_path=/qskrps-18/ 2023-05-24T21:14:35.762 DEBUG:boto:path=/qskrps-18/?versions&max-keys=999 2023-05-24T21:14:35.762 DEBUG:boto:auth_path=/qskrps-18/?versions&max-keys=999 2023-05-24T21:14:35.763 DEBUG:boto:Method: GET 2023-05-24T21:14:35.763 DEBUG:boto:Path: /qskrps-18/?versions&max-keys=999 2023-05-24T21:14:35.763 DEBUG:boto:Data: 2023-05-24T21:14:35.764 DEBUG:boto:Headers: {} 2023-05-24T21:14:35.764 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:14:35.764 DEBUG:boto:Port: 8000 2023-05-24T21:14:35.765 DEBUG:boto:Params: {} 2023-05-24T21:14:35.765 DEBUG:boto:Token: None 2023-05-24T21:14:35.766 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:14:35 GMT /qskrps-18/?versions 2023-05-24T21:14:35.766 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ysP5ucsRLN5Vwx8GM5FCQtUQCzo= 2023-05-24T21:14:35.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ysP5ucsRLN5Vwx8GM5FCQtUQCzo=', 'Content-Length': '0'} 2023-05-24T21:14:35.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2af7ac0528c6f8f-00646e7e3b-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:14:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:35.769 DEBUG:boto:b'qskrps-18999falseobjnulltrue2023-05-24T21:14:00.087Zrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:14:35.770 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-24T21:14:35.770 DEBUG:boto:path=/qskrps-18/ 2023-05-24T21:14:35.770 DEBUG:boto:auth_path=/qskrps-18/ 2023-05-24T21:14:35.771 DEBUG:boto:path=/qskrps-18/?versions&max-keys=999 2023-05-24T21:14:35.771 DEBUG:boto:auth_path=/qskrps-18/?versions&max-keys=999 2023-05-24T21:14:35.772 DEBUG:boto:Method: GET 2023-05-24T21:14:35.772 DEBUG:boto:Path: /qskrps-18/?versions&max-keys=999 2023-05-24T21:14:35.772 DEBUG:boto:Data: 2023-05-24T21:14:35.773 DEBUG:boto:Headers: {} 2023-05-24T21:14:35.773 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:14:35.773 DEBUG:boto:Port: 8001 2023-05-24T21:14:35.774 DEBUG:boto:Params: {} 2023-05-24T21:14:35.774 DEBUG:boto:Token: None 2023-05-24T21:14:35.774 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:14:35 GMT /qskrps-18/?versions 2023-05-24T21:14:35.775 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ysP5ucsRLN5Vwx8GM5FCQtUQCzo= 2023-05-24T21:14:35.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ysP5ucsRLN5Vwx8GM5FCQtUQCzo=', 'Content-Length': '0'} 2023-05-24T21:14:35.777 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca75db01d5267317-00646e7e3b-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:14:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:35.778 DEBUG:boto:b'qskrps-18999falseobjnulltrue2023-05-24T21:14:00.087Zrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:14:35.778 DEBUG:boto:path=/qskrps-18/ 2023-05-24T21:14:35.779 DEBUG:boto:auth_path=/qskrps-18/ 2023-05-24T21:14:35.779 DEBUG:boto:path=/qskrps-18/?versions&max-keys=999 2023-05-24T21:14:35.780 DEBUG:boto:auth_path=/qskrps-18/?versions&max-keys=999 2023-05-24T21:14:35.780 DEBUG:boto:Method: GET 2023-05-24T21:14:35.780 DEBUG:boto:Path: /qskrps-18/?versions&max-keys=999 2023-05-24T21:14:35.781 DEBUG:boto:Data: 2023-05-24T21:14:35.781 DEBUG:boto:Headers: {} 2023-05-24T21:14:35.781 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:14:35.782 DEBUG:boto:Port: 8000 2023-05-24T21:14:35.782 DEBUG:boto:Params: {} 2023-05-24T21:14:35.782 DEBUG:boto:Token: None 2023-05-24T21:14:35.783 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:14:35 GMT /qskrps-18/?versions 2023-05-24T21:14:35.783 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ysP5ucsRLN5Vwx8GM5FCQtUQCzo= 2023-05-24T21:14:35.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ysP5ucsRLN5Vwx8GM5FCQtUQCzo=', 'Content-Length': '0'} 2023-05-24T21:14:35.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ca889f9c7549860-00646e7e3b-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:14:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:35.786 DEBUG:boto:b'qskrps-18999falseobjnulltrue2023-05-24T21:14:00.087Zrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:14:35.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-24T21:14:35.787 DEBUG:boto:path=/qskrps-18/obj 2023-05-24T21:14:35.788 DEBUG:boto:auth_path=/qskrps-18/obj 2023-05-24T21:14:35.788 DEBUG:boto:Method: HEAD 2023-05-24T21:14:35.788 DEBUG:boto:Path: /qskrps-18/obj 2023-05-24T21:14:35.789 DEBUG:boto:Data: 2023-05-24T21:14:35.789 DEBUG:boto:Headers: {} 2023-05-24T21:14:35.789 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:14:35.790 DEBUG:boto:Port: 8001 2023-05-24T21:14:35.790 DEBUG:boto:Params: {} 2023-05-24T21:14:35.790 DEBUG:boto:Token: None 2023-05-24T21:14:35.791 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:14:35 GMT /qskrps-18/obj 2023-05-24T21:14:35.791 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:mRK6HgAa/5EArKkjgZkeXaTy7Vs= 2023-05-24T21:14:35.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:mRK6HgAa/5EArKkjgZkeXaTy7Vs=', 'Content-Length': '0'} 2023-05-24T21:14:35.793 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000ab8546d7c7c97dea-00646e7e3b-1111-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:14:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:35.796 DEBUG:boto:path=/qskrps-18/obj 2023-05-24T21:14:35.796 DEBUG:boto:auth_path=/qskrps-18/obj 2023-05-24T21:14:35.797 DEBUG:boto:Method: HEAD 2023-05-24T21:14:35.797 DEBUG:boto:Path: /qskrps-18/obj 2023-05-24T21:14:35.797 DEBUG:boto:Data: 2023-05-24T21:14:35.798 DEBUG:boto:Headers: {} 2023-05-24T21:14:35.798 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:14:35.798 DEBUG:boto:Port: 8000 2023-05-24T21:14:35.799 DEBUG:boto:Params: {} 2023-05-24T21:14:35.799 DEBUG:boto:Token: None 2023-05-24T21:14:35.799 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:14:35 GMT /qskrps-18/obj 2023-05-24T21:14:35.800 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:mRK6HgAa/5EArKkjgZkeXaTy7Vs= 2023-05-24T21:14:35.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:mRK6HgAa/5EArKkjgZkeXaTy7Vs=', 'Content-Length': '0'} 2023-05-24T21:14:35.802 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000c2db603f15ad6792-00646e7e3b-110d-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:14:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:35.802 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qskrps-18 zones={a2, a1} 2023-05-24T21:14:35.803 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2023-05-24T21:14:35.803 INFO:rgw_multi.tests:create bucket zone=a1 name=qskrps-19 2023-05-24T21:14:35.804 DEBUG:boto:path=/qskrps-19/ 2023-05-24T21:14:35.804 DEBUG:boto:auth_path=/qskrps-19/ 2023-05-24T21:14:35.805 DEBUG:boto:Method: PUT 2023-05-24T21:14:35.805 DEBUG:boto:Path: /qskrps-19/ 2023-05-24T21:14:35.805 DEBUG:boto:Data: 2023-05-24T21:14:35.806 DEBUG:boto:Headers: {} 2023-05-24T21:14:35.806 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:14:35.806 DEBUG:boto:Port: 8000 2023-05-24T21:14:35.807 DEBUG:boto:Params: {} 2023-05-24T21:14:35.807 DEBUG:boto:Token: None 2023-05-24T21:14:35.807 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:14:35 GMT /qskrps-19/ 2023-05-24T21:14:35.808 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5ZVCSuJhaHI9ir1yKxnyHlbhKak= 2023-05-24T21:14:35.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5ZVCSuJhaHI9ir1yKxnyHlbhKak=', 'Content-Length': '0'} 2023-05-24T21:14:35.824 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001e682dd219905730-00646e7e3b-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:14:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:35.824 INFO:rgw_multi.tests:create bucket zone=a2 name=qskrps-20 2023-05-24T21:14:35.825 DEBUG:boto:path=/qskrps-20/ 2023-05-24T21:14:35.825 DEBUG:boto:auth_path=/qskrps-20/ 2023-05-24T21:14:35.826 DEBUG:boto:Method: PUT 2023-05-24T21:14:35.826 DEBUG:boto:Path: /qskrps-20/ 2023-05-24T21:14:35.827 DEBUG:boto:Data: 2023-05-24T21:14:35.827 DEBUG:boto:Headers: {} 2023-05-24T21:14:35.827 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:14:35.828 DEBUG:boto:Port: 8001 2023-05-24T21:14:35.828 DEBUG:boto:Params: {} 2023-05-24T21:14:35.828 DEBUG:boto:Token: None 2023-05-24T21:14:35.829 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:14:35 GMT /qskrps-20/ 2023-05-24T21:14:35.829 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:91qfbaJcaUmtKOm9pYgwegPU9u8= 2023-05-24T21:14:35.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:91qfbaJcaUmtKOm9pYgwegPU9u8=', 'Content-Length': '0'} 2023-05-24T21:14:35.870 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004e0d37bdd359ad94-00646e7e3b-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:14:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:35.871 DEBUG:boto:Using access key provided by client. 2023-05-24T21:14:35.872 DEBUG:boto:Using secret key provided by client. 2023-05-24T21:14:35.873 DEBUG:boto:Using access key provided by client. 2023-05-24T21:14:35.874 DEBUG:boto:Using secret key provided by client. 2023-05-24T21:14:35.874 DEBUG:boto:Using access key provided by client. 2023-05-24T21:14:35.875 DEBUG:boto:Using secret key provided by client. 2023-05-24T21:14:35.875 DEBUG:boto:Using access key provided by client. 2023-05-24T21:14:35.876 DEBUG:boto:Using secret key provided by client. 2023-05-24T21:14:35.876 INFO:rgw_multi.tests:create bucket zone=b1 name=qskrps-21 2023-05-24T21:14:35.877 DEBUG:boto:path=/qskrps-21/ 2023-05-24T21:14:35.877 DEBUG:boto:auth_path=/qskrps-21/ 2023-05-24T21:14:35.877 DEBUG:boto:Method: PUT 2023-05-24T21:14:35.878 DEBUG:boto:Path: /qskrps-21/ 2023-05-24T21:14:35.878 DEBUG:boto:Data: 2023-05-24T21:14:35.879 DEBUG:boto:Headers: {} 2023-05-24T21:14:35.879 DEBUG:boto:Host: smithi157.front.sepia.ceph.com:8000 2023-05-24T21:14:35.879 DEBUG:boto:Port: 8000 2023-05-24T21:14:35.880 DEBUG:boto:Params: {} 2023-05-24T21:14:35.880 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T21:14:35.880 DEBUG:boto:Token: None 2023-05-24T21:14:35.881 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:14:35 GMT /qskrps-21/ 2023-05-24T21:14:35.881 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:md1fAHdDjFXN2kPaziYk8YdrvBk= 2023-05-24T21:14:35.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:md1fAHdDjFXN2kPaziYk8YdrvBk=', 'Content-Length': '0'} 2023-05-24T21:14:35.921 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007ac09a1a8a89e150-00646e7e3b-10fe-b1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:14:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:35.921 INFO:rgw_multi.tests:create bucket zone=b2 name=qskrps-22 2023-05-24T21:14:35.922 DEBUG:boto:path=/qskrps-22/ 2023-05-24T21:14:35.922 DEBUG:boto:auth_path=/qskrps-22/ 2023-05-24T21:14:35.923 DEBUG:boto:Method: PUT 2023-05-24T21:14:35.923 DEBUG:boto:Path: /qskrps-22/ 2023-05-24T21:14:35.923 DEBUG:boto:Data: 2023-05-24T21:14:35.924 DEBUG:boto:Headers: {} 2023-05-24T21:14:35.924 DEBUG:boto:Host: smithi157.front.sepia.ceph.com:8001 2023-05-24T21:14:35.924 DEBUG:boto:Port: 8001 2023-05-24T21:14:35.925 DEBUG:boto:Params: {} 2023-05-24T21:14:35.925 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-24T21:14:35.925 DEBUG:boto:Token: None 2023-05-24T21:14:35.926 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:14:35 GMT /qskrps-22/ 2023-05-24T21:14:35.927 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:PtRSvH89e1tSYagvWb6ppYO+lmw= 2023-05-24T21:14:35.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PtRSvH89e1tSYagvWb6ppYO+lmw=', 'Content-Length': '0'} 2023-05-24T21:14:35.967 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000002fabbd92a9a9c6e-00646e7e3b-1104-b2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:14:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:35.968 DEBUG:boto:path=/qskrps-19/ 2023-05-24T21:14:35.968 DEBUG:boto:auth_path=/qskrps-19/ 2023-05-24T21:14:35.968 DEBUG:boto:path=/qskrps-19/?versioning 2023-05-24T21:14:35.969 DEBUG:boto:auth_path=/qskrps-19/?versioning 2023-05-24T21:14:35.969 DEBUG:boto:Method: PUT 2023-05-24T21:14:35.969 DEBUG:boto:Path: /qskrps-19/?versioning 2023-05-24T21:14:35.970 DEBUG:boto:Data: Enabled Disabled 2023-05-24T21:14:35.970 DEBUG:boto:Headers: {} 2023-05-24T21:14:35.970 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:14:35.971 DEBUG:boto:Port: 8000 2023-05-24T21:14:35.971 DEBUG:boto:Params: {} 2023-05-24T21:14:35.971 DEBUG:boto:Token: None 2023-05-24T21:14:35.972 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:14:35 GMT /qskrps-19/?versioning 2023-05-24T21:14:35.972 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:YGVIurYyPpwwmf2EellWUAII3CY= 2023-05-24T21:14:35.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:YGVIurYyPpwwmf2EellWUAII3CY=', 'Content-Length': '228'} 2023-05-24T21:14:35.980 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000244b431c0119629e-00646e7e3b-110d-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:14:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:35.980 DEBUG:boto:path=/qskrps-19/ 2023-05-24T21:14:35.981 DEBUG:boto:auth_path=/qskrps-19/ 2023-05-24T21:14:35.981 DEBUG:boto:path=/qskrps-19/?versioning 2023-05-24T21:14:35.982 DEBUG:boto:auth_path=/qskrps-19/?versioning 2023-05-24T21:14:35.982 DEBUG:boto:Method: GET 2023-05-24T21:14:35.982 DEBUG:boto:Path: /qskrps-19/?versioning 2023-05-24T21:14:35.983 DEBUG:boto:Data: 2023-05-24T21:14:35.983 DEBUG:boto:Headers: {} 2023-05-24T21:14:35.983 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:14:35.984 DEBUG:boto:Port: 8000 2023-05-24T21:14:35.984 DEBUG:boto:Params: {} 2023-05-24T21:14:35.984 DEBUG:boto:Token: None 2023-05-24T21:14:35.985 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:14:35 GMT /qskrps-19/?versioning 2023-05-24T21:14:35.985 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:HvZvgFTgoAYlZ668t9sZyL58KqI= 2023-05-24T21:14:35.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:35 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:HvZvgFTgoAYlZ668t9sZyL58KqI=', 'Content-Length': '0'} 2023-05-24T21:14:35.987 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000011621076b4c5b93b-00646e7e3b-110d-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Wed, 24 May 2023 21:14:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:36.028 DEBUG:boto:EnabledDisabled 2023-05-24T21:14:36.029 DEBUG:boto:path=/qskrps-20/ 2023-05-24T21:14:36.030 DEBUG:boto:auth_path=/qskrps-20/ 2023-05-24T21:14:36.030 DEBUG:boto:path=/qskrps-20/?versioning 2023-05-24T21:14:36.030 DEBUG:boto:auth_path=/qskrps-20/?versioning 2023-05-24T21:14:36.031 DEBUG:boto:Method: PUT 2023-05-24T21:14:36.031 DEBUG:boto:Path: /qskrps-20/?versioning 2023-05-24T21:14:36.031 DEBUG:boto:Data: Enabled Disabled 2023-05-24T21:14:36.032 DEBUG:boto:Headers: {} 2023-05-24T21:14:36.032 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:14:36.032 DEBUG:boto:Port: 8001 2023-05-24T21:14:36.033 DEBUG:boto:Params: {} 2023-05-24T21:14:36.033 DEBUG:boto:Token: None 2023-05-24T21:14:36.033 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:14:36 GMT /qskrps-20/?versioning 2023-05-24T21:14:36.034 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:m1LD9AwM79W2PTPZ/kSgHTVfX54= 2023-05-24T21:14:36.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:m1LD9AwM79W2PTPZ/kSgHTVfX54=', 'Content-Length': '228'} 2023-05-24T21:14:36.043 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000eda1af8ded962eee-00646e7e3c-1111-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:14:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:36.044 DEBUG:boto:path=/qskrps-20/ 2023-05-24T21:14:36.044 DEBUG:boto:auth_path=/qskrps-20/ 2023-05-24T21:14:36.044 DEBUG:boto:path=/qskrps-20/?versioning 2023-05-24T21:14:36.045 DEBUG:boto:auth_path=/qskrps-20/?versioning 2023-05-24T21:14:36.045 DEBUG:boto:Method: GET 2023-05-24T21:14:36.045 DEBUG:boto:Path: /qskrps-20/?versioning 2023-05-24T21:14:36.046 DEBUG:boto:Data: 2023-05-24T21:14:36.046 DEBUG:boto:Headers: {} 2023-05-24T21:14:36.046 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:14:36.047 DEBUG:boto:Port: 8001 2023-05-24T21:14:36.047 DEBUG:boto:Params: {} 2023-05-24T21:14:36.047 DEBUG:boto:Token: None 2023-05-24T21:14:36.048 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:14:36 GMT /qskrps-20/?versioning 2023-05-24T21:14:36.048 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:pXMplUKl18TEYiOjvxphXrFoHqk= 2023-05-24T21:14:36.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:pXMplUKl18TEYiOjvxphXrFoHqk=', 'Content-Length': '0'} 2023-05-24T21:14:36.049 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000002c03a222950784c-00646e7e3c-1111-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Wed, 24 May 2023 21:14:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:36.092 DEBUG:boto:EnabledDisabled 2023-05-24T21:14:36.093 DEBUG:boto:path=/qskrps-21/ 2023-05-24T21:14:36.093 DEBUG:boto:auth_path=/qskrps-21/ 2023-05-24T21:14:36.093 DEBUG:boto:path=/qskrps-21/?versioning 2023-05-24T21:14:36.095 DEBUG:boto:auth_path=/qskrps-21/?versioning 2023-05-24T21:14:36.095 DEBUG:boto:Method: PUT 2023-05-24T21:14:36.095 DEBUG:boto:Path: /qskrps-21/?versioning 2023-05-24T21:14:36.096 DEBUG:boto:Data: Enabled Disabled 2023-05-24T21:14:36.096 DEBUG:boto:Headers: {} 2023-05-24T21:14:36.096 DEBUG:boto:Host: smithi157.front.sepia.ceph.com:8000 2023-05-24T21:14:36.097 DEBUG:boto:Port: 8000 2023-05-24T21:14:36.097 DEBUG:boto:Params: {} 2023-05-24T21:14:36.097 DEBUG:boto:Token: None 2023-05-24T21:14:36.098 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:14:36 GMT /qskrps-21/?versioning 2023-05-24T21:14:36.098 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:6fniYmXU7ygO3hxnVdRmzOXpYW0= 2023-05-24T21:14:36.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:6fniYmXU7ygO3hxnVdRmzOXpYW0=', 'Content-Length': '228'} 2023-05-24T21:14:36.106 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008363ca3e8b58c3e1-00646e7e3c-10fe-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:14:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:36.107 DEBUG:boto:path=/qskrps-21/ 2023-05-24T21:14:36.107 DEBUG:boto:auth_path=/qskrps-21/ 2023-05-24T21:14:36.108 DEBUG:boto:path=/qskrps-21/?versioning 2023-05-24T21:14:36.108 DEBUG:boto:auth_path=/qskrps-21/?versioning 2023-05-24T21:14:36.108 DEBUG:boto:Method: GET 2023-05-24T21:14:36.109 DEBUG:boto:Path: /qskrps-21/?versioning 2023-05-24T21:14:36.109 DEBUG:boto:Data: 2023-05-24T21:14:36.109 DEBUG:boto:Headers: {} 2023-05-24T21:14:36.110 DEBUG:boto:Host: smithi157.front.sepia.ceph.com:8000 2023-05-24T21:14:36.110 DEBUG:boto:Port: 8000 2023-05-24T21:14:36.110 DEBUG:boto:Params: {} 2023-05-24T21:14:36.111 DEBUG:boto:Token: None 2023-05-24T21:14:36.111 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:14:36 GMT /qskrps-21/?versioning 2023-05-24T21:14:36.111 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:F0Co9Y3Ps+WJTB4z1jk9yxG5qA8= 2023-05-24T21:14:36.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:F0Co9Y3Ps+WJTB4z1jk9yxG5qA8=', 'Content-Length': '0'} 2023-05-24T21:14:36.113 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000501014b85f41bf8d-00646e7e3c-10fe-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Wed, 24 May 2023 21:14:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:36.114 DEBUG:boto:EnabledDisabled 2023-05-24T21:14:36.114 DEBUG:boto:path=/qskrps-22/ 2023-05-24T21:14:36.115 DEBUG:boto:auth_path=/qskrps-22/ 2023-05-24T21:14:36.115 DEBUG:boto:path=/qskrps-22/?versioning 2023-05-24T21:14:36.115 DEBUG:boto:auth_path=/qskrps-22/?versioning 2023-05-24T21:14:36.115 DEBUG:boto:Method: PUT 2023-05-24T21:14:36.116 DEBUG:boto:Path: /qskrps-22/?versioning 2023-05-24T21:14:36.116 DEBUG:boto:Data: Enabled Disabled 2023-05-24T21:14:36.116 DEBUG:boto:Headers: {} 2023-05-24T21:14:36.117 DEBUG:boto:Host: smithi157.front.sepia.ceph.com:8001 2023-05-24T21:14:36.117 DEBUG:boto:Port: 8001 2023-05-24T21:14:36.118 DEBUG:boto:Params: {} 2023-05-24T21:14:36.118 DEBUG:boto:Token: None 2023-05-24T21:14:36.118 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:14:36 GMT /qskrps-22/?versioning 2023-05-24T21:14:36.119 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:JPxXVuzvrhTYKShGQahwXpzs/iw= 2023-05-24T21:14:36.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:JPxXVuzvrhTYKShGQahwXpzs/iw=', 'Content-Length': '228'} 2023-05-24T21:14:36.129 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000055104d5dee4ed505-00646e7e3c-1104-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:14:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:36.130 DEBUG:boto:path=/qskrps-22/ 2023-05-24T21:14:36.130 DEBUG:boto:auth_path=/qskrps-22/ 2023-05-24T21:14:36.131 DEBUG:boto:path=/qskrps-22/?versioning 2023-05-24T21:14:36.132 DEBUG:boto:auth_path=/qskrps-22/?versioning 2023-05-24T21:14:36.132 DEBUG:boto:Method: GET 2023-05-24T21:14:36.132 DEBUG:boto:Path: /qskrps-22/?versioning 2023-05-24T21:14:36.133 DEBUG:boto:Data: 2023-05-24T21:14:36.133 DEBUG:boto:Headers: {} 2023-05-24T21:14:36.133 DEBUG:boto:Host: smithi157.front.sepia.ceph.com:8001 2023-05-24T21:14:36.134 DEBUG:boto:Port: 8001 2023-05-24T21:14:36.134 DEBUG:boto:Params: {} 2023-05-24T21:14:36.134 DEBUG:boto:Token: None 2023-05-24T21:14:36.135 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:14:36 GMT /qskrps-22/?versioning 2023-05-24T21:14:36.135 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:3QiIap6mpkQk0sEXk5j49KHXHhw= 2023-05-24T21:14:36.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:3QiIap6mpkQk0sEXk5j49KHXHhw=', 'Content-Length': '0'} 2023-05-24T21:14:36.137 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000048251392d3bcf071-00646e7e3c-1104-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Wed, 24 May 2023 21:14:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:36.137 DEBUG:boto:EnabledDisabled 2023-05-24T21:14:36.138 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_versioning ... ok 2023-05-24T21:14:36.139 INFO:rgw_multi.tests:create bucket zone=a1 name=qskrps-23 2023-05-24T21:14:36.139 DEBUG:boto:path=/qskrps-23/ 2023-05-24T21:14:36.139 DEBUG:boto:auth_path=/qskrps-23/ 2023-05-24T21:14:36.140 DEBUG:boto:Method: PUT 2023-05-24T21:14:36.140 DEBUG:boto:Path: /qskrps-23/ 2023-05-24T21:14:36.140 DEBUG:boto:Data: 2023-05-24T21:14:36.141 DEBUG:boto:Headers: {} 2023-05-24T21:14:36.141 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:14:36.141 DEBUG:boto:Port: 8000 2023-05-24T21:14:36.142 DEBUG:boto:Params: {} 2023-05-24T21:14:36.142 DEBUG:boto:Token: None 2023-05-24T21:14:36.142 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:14:36 GMT /qskrps-23/ 2023-05-24T21:14:36.143 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:zM17d/nwhOa/TBQzP2rOtE6H99E= 2023-05-24T21:14:36.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:zM17d/nwhOa/TBQzP2rOtE6H99E=', 'Content-Length': '0'} 2023-05-24T21:14:36.168 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c4524422d8a8b355-00646e7e3c-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:14:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:36.168 INFO:rgw_multi.tests:create bucket zone=a2 name=qskrps-24 2023-05-24T21:14:36.169 DEBUG:boto:path=/qskrps-24/ 2023-05-24T21:14:36.169 DEBUG:boto:auth_path=/qskrps-24/ 2023-05-24T21:14:36.169 DEBUG:boto:Method: PUT 2023-05-24T21:14:36.170 DEBUG:boto:Path: /qskrps-24/ 2023-05-24T21:14:36.170 DEBUG:boto:Data: 2023-05-24T21:14:36.170 DEBUG:boto:Headers: {} 2023-05-24T21:14:36.171 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:14:36.171 DEBUG:boto:Port: 8001 2023-05-24T21:14:36.171 DEBUG:boto:Params: {} 2023-05-24T21:14:36.172 DEBUG:boto:Token: None 2023-05-24T21:14:36.173 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:14:36 GMT /qskrps-24/ 2023-05-24T21:14:36.173 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:eYDuK+6HRZc6ZJuRjmtgnNNxTDU= 2023-05-24T21:14:36.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:eYDuK+6HRZc6ZJuRjmtgnNNxTDU=', 'Content-Length': '0'} 2023-05-24T21:14:36.225 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007391f1201694d209-00646e7e3c-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:14:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:36.226 INFO:rgw_multi.tests:create bucket zone=b1 name=qskrps-25 2023-05-24T21:14:36.226 DEBUG:boto:path=/qskrps-25/ 2023-05-24T21:14:36.227 DEBUG:boto:auth_path=/qskrps-25/ 2023-05-24T21:14:36.227 DEBUG:boto:Method: PUT 2023-05-24T21:14:36.227 DEBUG:boto:Path: /qskrps-25/ 2023-05-24T21:14:36.228 DEBUG:boto:Data: 2023-05-24T21:14:36.228 DEBUG:boto:Headers: {} 2023-05-24T21:14:36.228 DEBUG:boto:Host: smithi157.front.sepia.ceph.com:8000 2023-05-24T21:14:36.229 DEBUG:boto:Port: 8000 2023-05-24T21:14:36.229 DEBUG:boto:Params: {} 2023-05-24T21:14:36.229 DEBUG:boto:Token: None 2023-05-24T21:14:36.230 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:14:36 GMT /qskrps-25/ 2023-05-24T21:14:36.230 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:a/okxH/4w3gW3SUCR3hLlnL1tGE= 2023-05-24T21:14:36.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:a/okxH/4w3gW3SUCR3hLlnL1tGE=', 'Content-Length': '0'} 2023-05-24T21:14:36.256 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000633d203e772cfb9a-00646e7e3c-10fe-b1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:14:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:36.257 INFO:rgw_multi.tests:create bucket zone=b2 name=qskrps-26 2023-05-24T21:14:36.257 DEBUG:boto:path=/qskrps-26/ 2023-05-24T21:14:36.258 DEBUG:boto:auth_path=/qskrps-26/ 2023-05-24T21:14:36.258 DEBUG:boto:Method: PUT 2023-05-24T21:14:36.259 DEBUG:boto:Path: /qskrps-26/ 2023-05-24T21:14:36.259 DEBUG:boto:Data: 2023-05-24T21:14:36.259 DEBUG:boto:Headers: {} 2023-05-24T21:14:36.260 DEBUG:boto:Host: smithi157.front.sepia.ceph.com:8001 2023-05-24T21:14:36.260 DEBUG:boto:Port: 8001 2023-05-24T21:14:36.260 DEBUG:boto:Params: {} 2023-05-24T21:14:36.261 DEBUG:boto:Token: None 2023-05-24T21:14:36.261 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:14:36 GMT /qskrps-26/ 2023-05-24T21:14:36.262 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:vXVA0eT/gmlB9JIe20Yn8N5T7hU= 2023-05-24T21:14:36.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:vXVA0eT/gmlB9JIe20Yn8N5T7hU=', 'Content-Length': '0'} 2023-05-24T21:14:36.287 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008e21b0a4d323dc07-00646e7e3c-1104-b2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:14:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:36.287 DEBUG:boto:path=/qskrps-23/ 2023-05-24T21:14:36.288 DEBUG:boto:auth_path=/qskrps-23/ 2023-05-24T21:14:36.288 DEBUG:boto:path=/qskrps-23/?acl 2023-05-24T21:14:36.288 DEBUG:boto:auth_path=/qskrps-23/?acl 2023-05-24T21:14:36.289 DEBUG:boto:Method: GET 2023-05-24T21:14:36.289 DEBUG:boto:Path: /qskrps-23/?acl 2023-05-24T21:14:36.289 DEBUG:boto:Data: 2023-05-24T21:14:36.290 DEBUG:boto:Headers: {} 2023-05-24T21:14:36.290 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:14:36.290 DEBUG:boto:Port: 8000 2023-05-24T21:14:36.291 DEBUG:boto:Params: {} 2023-05-24T21:14:36.291 DEBUG:boto:Token: None 2023-05-24T21:14:36.291 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:14:36 GMT /qskrps-23/?acl 2023-05-24T21:14:36.292 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:BnNkGzkdKZzXE2pl4n87n+Gu05M= 2023-05-24T21:14:36.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:BnNkGzkdKZzXE2pl4n87n+Gu05M=', 'Content-Length': '0'} 2023-05-24T21:14:36.293 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007f1d456338f9c3e4-00646e7e3c-110d-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Wed, 24 May 2023 21:14:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:36.294 DEBUG:boto:path=/qskrps-23/ 2023-05-24T21:14:36.295 DEBUG:boto:auth_path=/qskrps-23/ 2023-05-24T21:14:36.295 DEBUG:boto:path=/qskrps-23/?acl 2023-05-24T21:14:36.295 DEBUG:boto:auth_path=/qskrps-23/?acl 2023-05-24T21:14:36.296 DEBUG:boto:Method: PUT 2023-05-24T21:14:36.296 DEBUG:boto:Path: /qskrps-23/?acl 2023-05-24T21:14:36.296 DEBUG:boto:Data: 2023-05-24T21:14:36.297 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-05-24T21:14:36.297 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:14:36.297 DEBUG:boto:Port: 8000 2023-05-24T21:14:36.298 DEBUG:boto:Params: {} 2023-05-24T21:14:36.298 DEBUG:boto:Token: None 2023-05-24T21:14:36.298 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:14:36 GMT x-amz-acl:public-read /qskrps-23/?acl 2023-05-24T21:14:36.299 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:d7FF8aS8/ZOMNQVGmoYmtZ300dQ= 2023-05-24T21:14:36.299 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:d7FF8aS8/ZOMNQVGmoYmtZ300dQ=', 'Content-Length': '0'} 2023-05-24T21:14:36.304 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000026cb27542bda055e-00646e7e3c-110d-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:14:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:36.305 DEBUG:boto:path=/qskrps-23/ 2023-05-24T21:14:36.306 DEBUG:boto:auth_path=/qskrps-23/ 2023-05-24T21:14:36.306 DEBUG:boto:path=/qskrps-23/?acl 2023-05-24T21:14:36.306 DEBUG:boto:auth_path=/qskrps-23/?acl 2023-05-24T21:14:36.306 DEBUG:boto:Method: GET 2023-05-24T21:14:36.307 DEBUG:boto:Path: /qskrps-23/?acl 2023-05-24T21:14:36.307 DEBUG:boto:Data: 2023-05-24T21:14:36.307 DEBUG:boto:Headers: {} 2023-05-24T21:14:36.308 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:14:36.308 DEBUG:boto:Port: 8000 2023-05-24T21:14:36.308 DEBUG:boto:Params: {} 2023-05-24T21:14:36.309 DEBUG:boto:Token: None 2023-05-24T21:14:36.309 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:14:36 GMT /qskrps-23/?acl 2023-05-24T21:14:36.310 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:BnNkGzkdKZzXE2pl4n87n+Gu05M= 2023-05-24T21:14:36.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:BnNkGzkdKZzXE2pl4n87n+Gu05M=', 'Content-Length': '0'} 2023-05-24T21:14:36.311 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d9fa2ac90638a92d-00646e7e3c-110d-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Wed, 24 May 2023 21:14:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:36.312 DEBUG:boto:path=/qskrps-24/ 2023-05-24T21:14:36.313 DEBUG:boto:auth_path=/qskrps-24/ 2023-05-24T21:14:36.313 DEBUG:boto:path=/qskrps-24/?acl 2023-05-24T21:14:36.313 DEBUG:boto:auth_path=/qskrps-24/?acl 2023-05-24T21:14:36.314 DEBUG:boto:Method: GET 2023-05-24T21:14:36.314 DEBUG:boto:Path: /qskrps-24/?acl 2023-05-24T21:14:36.314 DEBUG:boto:Data: 2023-05-24T21:14:36.315 DEBUG:boto:Headers: {} 2023-05-24T21:14:36.315 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:14:36.316 DEBUG:boto:Port: 8001 2023-05-24T21:14:36.316 DEBUG:boto:Params: {} 2023-05-24T21:14:36.317 DEBUG:boto:Token: None 2023-05-24T21:14:36.317 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:14:36 GMT /qskrps-24/?acl 2023-05-24T21:14:36.318 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:LAp50+lW26eAMteCqOrLAt/OBiE= 2023-05-24T21:14:36.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:LAp50+lW26eAMteCqOrLAt/OBiE=', 'Content-Length': '0'} 2023-05-24T21:14:36.320 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000028a6181a3e58f281-00646e7e3c-1111-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Wed, 24 May 2023 21:14:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:36.320 DEBUG:boto:path=/qskrps-24/ 2023-05-24T21:14:36.321 DEBUG:boto:auth_path=/qskrps-24/ 2023-05-24T21:14:36.321 DEBUG:boto:path=/qskrps-24/?acl 2023-05-24T21:14:36.322 DEBUG:boto:auth_path=/qskrps-24/?acl 2023-05-24T21:14:36.322 DEBUG:boto:Method: PUT 2023-05-24T21:14:36.322 DEBUG:boto:Path: /qskrps-24/?acl 2023-05-24T21:14:36.322 DEBUG:boto:Data: 2023-05-24T21:14:36.323 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-05-24T21:14:36.323 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:14:36.323 DEBUG:boto:Port: 8001 2023-05-24T21:14:36.324 DEBUG:boto:Params: {} 2023-05-24T21:14:36.324 DEBUG:boto:Token: None 2023-05-24T21:14:36.325 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:14:36 GMT x-amz-acl:public-read /qskrps-24/?acl 2023-05-24T21:14:36.325 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4tuexh/5wC5DkaFhZfvMCaKfUeE= 2023-05-24T21:14:36.325 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4tuexh/5wC5DkaFhZfvMCaKfUeE=', 'Content-Length': '0'} 2023-05-24T21:14:36.332 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003d45da93f50baced-00646e7e3c-1111-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:14:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:36.333 DEBUG:boto:path=/qskrps-24/ 2023-05-24T21:14:36.333 DEBUG:boto:auth_path=/qskrps-24/ 2023-05-24T21:14:36.333 DEBUG:boto:path=/qskrps-24/?acl 2023-05-24T21:14:36.334 DEBUG:boto:auth_path=/qskrps-24/?acl 2023-05-24T21:14:36.334 DEBUG:boto:Method: GET 2023-05-24T21:14:36.334 DEBUG:boto:Path: /qskrps-24/?acl 2023-05-24T21:14:36.335 DEBUG:boto:Data: 2023-05-24T21:14:36.335 DEBUG:boto:Headers: {} 2023-05-24T21:14:36.335 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:14:36.336 DEBUG:boto:Port: 8001 2023-05-24T21:14:36.336 DEBUG:boto:Params: {} 2023-05-24T21:14:36.336 DEBUG:boto:Token: None 2023-05-24T21:14:36.337 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:14:36 GMT /qskrps-24/?acl 2023-05-24T21:14:36.337 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:LAp50+lW26eAMteCqOrLAt/OBiE= 2023-05-24T21:14:36.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:LAp50+lW26eAMteCqOrLAt/OBiE=', 'Content-Length': '0'} 2023-05-24T21:14:36.338 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f3d3787b31cbd270-00646e7e3c-1111-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Wed, 24 May 2023 21:14:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:36.339 DEBUG:boto:path=/qskrps-25/ 2023-05-24T21:14:36.339 DEBUG:boto:auth_path=/qskrps-25/ 2023-05-24T21:14:36.340 DEBUG:boto:path=/qskrps-25/?acl 2023-05-24T21:14:36.340 DEBUG:boto:auth_path=/qskrps-25/?acl 2023-05-24T21:14:36.341 DEBUG:boto:Method: GET 2023-05-24T21:14:36.341 DEBUG:boto:Path: /qskrps-25/?acl 2023-05-24T21:14:36.341 DEBUG:boto:Data: 2023-05-24T21:14:36.342 DEBUG:boto:Headers: {} 2023-05-24T21:14:36.342 DEBUG:boto:Host: smithi157.front.sepia.ceph.com:8000 2023-05-24T21:14:36.343 DEBUG:boto:Port: 8000 2023-05-24T21:14:36.343 DEBUG:boto:Params: {} 2023-05-24T21:14:36.343 DEBUG:boto:Token: None 2023-05-24T21:14:36.344 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:14:36 GMT /qskrps-25/?acl 2023-05-24T21:14:36.344 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:0c/WI/tOrazjP7nVLptyIhqKqlI= 2023-05-24T21:14:36.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:0c/WI/tOrazjP7nVLptyIhqKqlI=', 'Content-Length': '0'} 2023-05-24T21:14:36.346 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000037900b1ac552c318-00646e7e3c-10fe-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Wed, 24 May 2023 21:14:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:36.347 DEBUG:boto:path=/qskrps-25/ 2023-05-24T21:14:36.347 DEBUG:boto:auth_path=/qskrps-25/ 2023-05-24T21:14:36.347 DEBUG:boto:path=/qskrps-25/?acl 2023-05-24T21:14:36.348 DEBUG:boto:auth_path=/qskrps-25/?acl 2023-05-24T21:14:36.348 DEBUG:boto:Method: PUT 2023-05-24T21:14:36.349 DEBUG:boto:Path: /qskrps-25/?acl 2023-05-24T21:14:36.349 DEBUG:boto:Data: 2023-05-24T21:14:36.349 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-05-24T21:14:36.350 DEBUG:boto:Host: smithi157.front.sepia.ceph.com:8000 2023-05-24T21:14:36.350 DEBUG:boto:Port: 8000 2023-05-24T21:14:36.350 DEBUG:boto:Params: {} 2023-05-24T21:14:36.351 DEBUG:boto:Token: None 2023-05-24T21:14:36.351 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:14:36 GMT x-amz-acl:public-read /qskrps-25/?acl 2023-05-24T21:14:36.351 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:77n7OPIOoEo5DLCcUYowH5VisMI= 2023-05-24T21:14:36.352 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:77n7OPIOoEo5DLCcUYowH5VisMI=', 'Content-Length': '0'} 2023-05-24T21:14:36.360 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001c54e335a0ceae52-00646e7e3c-10fe-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:14:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:36.361 DEBUG:boto:path=/qskrps-25/ 2023-05-24T21:14:36.362 DEBUG:boto:auth_path=/qskrps-25/ 2023-05-24T21:14:36.362 DEBUG:boto:path=/qskrps-25/?acl 2023-05-24T21:14:36.363 DEBUG:boto:auth_path=/qskrps-25/?acl 2023-05-24T21:14:36.363 DEBUG:boto:Method: GET 2023-05-24T21:14:36.363 DEBUG:boto:Path: /qskrps-25/?acl 2023-05-24T21:14:36.364 DEBUG:boto:Data: 2023-05-24T21:14:36.364 DEBUG:boto:Headers: {} 2023-05-24T21:14:36.364 DEBUG:boto:Host: smithi157.front.sepia.ceph.com:8000 2023-05-24T21:14:36.365 DEBUG:boto:Port: 8000 2023-05-24T21:14:36.365 DEBUG:boto:Params: {} 2023-05-24T21:14:36.366 DEBUG:boto:Token: None 2023-05-24T21:14:36.366 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:14:36 GMT /qskrps-25/?acl 2023-05-24T21:14:36.367 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:0c/WI/tOrazjP7nVLptyIhqKqlI= 2023-05-24T21:14:36.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:0c/WI/tOrazjP7nVLptyIhqKqlI=', 'Content-Length': '0'} 2023-05-24T21:14:36.368 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000036b4fcf6edc43fcf-00646e7e3c-10fe-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Wed, 24 May 2023 21:14:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:36.369 DEBUG:boto:path=/qskrps-26/ 2023-05-24T21:14:36.369 DEBUG:boto:auth_path=/qskrps-26/ 2023-05-24T21:14:36.370 DEBUG:boto:path=/qskrps-26/?acl 2023-05-24T21:14:36.370 DEBUG:boto:auth_path=/qskrps-26/?acl 2023-05-24T21:14:36.370 DEBUG:boto:Method: GET 2023-05-24T21:14:36.371 DEBUG:boto:Path: /qskrps-26/?acl 2023-05-24T21:14:36.371 DEBUG:boto:Data: 2023-05-24T21:14:36.371 DEBUG:boto:Headers: {} 2023-05-24T21:14:36.372 DEBUG:boto:Host: smithi157.front.sepia.ceph.com:8001 2023-05-24T21:14:36.372 DEBUG:boto:Port: 8001 2023-05-24T21:14:36.373 DEBUG:boto:Params: {} 2023-05-24T21:14:36.373 DEBUG:boto:Token: None 2023-05-24T21:14:36.373 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:14:36 GMT /qskrps-26/?acl 2023-05-24T21:14:36.374 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:dyAjktr/O++vFSJYbfBcIwT4rZk= 2023-05-24T21:14:36.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:dyAjktr/O++vFSJYbfBcIwT4rZk=', 'Content-Length': '0'} 2023-05-24T21:14:36.375 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004ce3ae503c04a896-00646e7e3c-1104-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Wed, 24 May 2023 21:14:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:36.376 DEBUG:boto:path=/qskrps-26/ 2023-05-24T21:14:36.377 DEBUG:boto:auth_path=/qskrps-26/ 2023-05-24T21:14:36.377 DEBUG:boto:path=/qskrps-26/?acl 2023-05-24T21:14:36.377 DEBUG:boto:auth_path=/qskrps-26/?acl 2023-05-24T21:14:36.378 DEBUG:boto:Method: PUT 2023-05-24T21:14:36.378 DEBUG:boto:Path: /qskrps-26/?acl 2023-05-24T21:14:36.378 DEBUG:boto:Data: 2023-05-24T21:14:36.379 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-05-24T21:14:36.379 DEBUG:boto:Host: smithi157.front.sepia.ceph.com:8001 2023-05-24T21:14:36.380 DEBUG:boto:Port: 8001 2023-05-24T21:14:36.380 DEBUG:boto:Params: {} 2023-05-24T21:14:36.380 DEBUG:boto:Token: None 2023-05-24T21:14:36.381 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:14:36 GMT x-amz-acl:public-read /qskrps-26/?acl 2023-05-24T21:14:36.381 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:L95GEhhuqRHnjegu3dtAesQubvw= 2023-05-24T21:14:36.382 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:L95GEhhuqRHnjegu3dtAesQubvw=', 'Content-Length': '0'} 2023-05-24T21:14:36.400 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fb47954e441c2813-00646e7e3c-1104-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:14:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:36.401 DEBUG:boto:path=/qskrps-26/ 2023-05-24T21:14:36.402 DEBUG:boto:auth_path=/qskrps-26/ 2023-05-24T21:14:36.402 DEBUG:boto:path=/qskrps-26/?acl 2023-05-24T21:14:36.403 DEBUG:boto:auth_path=/qskrps-26/?acl 2023-05-24T21:14:36.404 DEBUG:boto:Method: GET 2023-05-24T21:14:36.405 DEBUG:boto:Path: /qskrps-26/?acl 2023-05-24T21:14:36.405 DEBUG:boto:Data: 2023-05-24T21:14:36.406 DEBUG:boto:Headers: {} 2023-05-24T21:14:36.407 DEBUG:boto:Host: smithi157.front.sepia.ceph.com:8001 2023-05-24T21:14:36.407 DEBUG:boto:Port: 8001 2023-05-24T21:14:36.408 DEBUG:boto:Params: {} 2023-05-24T21:14:36.408 DEBUG:boto:Token: None 2023-05-24T21:14:36.409 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:14:36 GMT /qskrps-26/?acl 2023-05-24T21:14:36.409 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:dyAjktr/O++vFSJYbfBcIwT4rZk= 2023-05-24T21:14:36.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:dyAjktr/O++vFSJYbfBcIwT4rZk=', 'Content-Length': '0'} 2023-05-24T21:14:36.411 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000dc84a3b79e2377c-00646e7e3c-1104-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Wed, 24 May 2023 21:14:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:36.412 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_acl ... ok 2023-05-24T21:14:36.412 INFO:rgw_multi.tests:create bucket zone=a1 name=qskrps-27 2023-05-24T21:14:36.413 DEBUG:boto:path=/qskrps-27/ 2023-05-24T21:14:36.413 DEBUG:boto:auth_path=/qskrps-27/ 2023-05-24T21:14:36.414 DEBUG:boto:Method: PUT 2023-05-24T21:14:36.414 DEBUG:boto:Path: /qskrps-27/ 2023-05-24T21:14:36.414 DEBUG:boto:Data: 2023-05-24T21:14:36.415 DEBUG:boto:Headers: {} 2023-05-24T21:14:36.415 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:14:36.415 DEBUG:boto:Port: 8000 2023-05-24T21:14:36.416 DEBUG:boto:Params: {} 2023-05-24T21:14:36.416 DEBUG:boto:Token: None 2023-05-24T21:14:36.416 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:14:36 GMT /qskrps-27/ 2023-05-24T21:14:36.417 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:afOtp+ZGb7guEcYFNeXjrBEFhY4= 2023-05-24T21:14:36.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:afOtp+ZGb7guEcYFNeXjrBEFhY4=', 'Content-Length': '0'} 2023-05-24T21:14:36.445 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fbcf651beab46285-00646e7e3c-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:14:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:36.446 INFO:rgw_multi.tests:create bucket zone=a2 name=qskrps-28 2023-05-24T21:14:36.446 DEBUG:boto:path=/qskrps-28/ 2023-05-24T21:14:36.447 DEBUG:boto:auth_path=/qskrps-28/ 2023-05-24T21:14:36.447 DEBUG:boto:Method: PUT 2023-05-24T21:14:36.448 DEBUG:boto:Path: /qskrps-28/ 2023-05-24T21:14:36.448 DEBUG:boto:Data: 2023-05-24T21:14:36.448 DEBUG:boto:Headers: {} 2023-05-24T21:14:36.449 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:14:36.450 DEBUG:boto:Port: 8001 2023-05-24T21:14:36.451 DEBUG:boto:Params: {} 2023-05-24T21:14:36.452 DEBUG:boto:Token: None 2023-05-24T21:14:36.453 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:14:36 GMT /qskrps-28/ 2023-05-24T21:14:36.453 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:lgtb9TmmJg9AZjdivUsWlkd/Wb8= 2023-05-24T21:14:36.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:lgtb9TmmJg9AZjdivUsWlkd/Wb8=', 'Content-Length': '0'} 2023-05-24T21:14:36.479 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000167ba01842a9294b-00646e7e3c-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:14:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:36.480 INFO:rgw_multi.tests:create bucket zone=b1 name=qskrps-29 2023-05-24T21:14:36.480 DEBUG:boto:path=/qskrps-29/ 2023-05-24T21:14:36.480 DEBUG:boto:auth_path=/qskrps-29/ 2023-05-24T21:14:36.481 DEBUG:boto:Method: PUT 2023-05-24T21:14:36.481 DEBUG:boto:Path: /qskrps-29/ 2023-05-24T21:14:36.482 DEBUG:boto:Data: 2023-05-24T21:14:36.482 DEBUG:boto:Headers: {} 2023-05-24T21:14:36.482 DEBUG:boto:Host: smithi157.front.sepia.ceph.com:8000 2023-05-24T21:14:36.483 DEBUG:boto:Port: 8000 2023-05-24T21:14:36.483 DEBUG:boto:Params: {} 2023-05-24T21:14:36.484 DEBUG:boto:Token: None 2023-05-24T21:14:36.484 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:14:36 GMT /qskrps-29/ 2023-05-24T21:14:36.484 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:qzgA9tIdJyIaFJeB+Sm9btrXbeI= 2023-05-24T21:14:36.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qzgA9tIdJyIaFJeB+Sm9btrXbeI=', 'Content-Length': '0'} 2023-05-24T21:14:36.510 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001ecc9e641c3397cb-00646e7e3c-10fe-b1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:14:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:36.511 INFO:rgw_multi.tests:create bucket zone=b2 name=qskrps-30 2023-05-24T21:14:36.512 DEBUG:boto:path=/qskrps-30/ 2023-05-24T21:14:36.512 DEBUG:boto:auth_path=/qskrps-30/ 2023-05-24T21:14:36.512 DEBUG:boto:Method: PUT 2023-05-24T21:14:36.513 DEBUG:boto:Path: /qskrps-30/ 2023-05-24T21:14:36.513 DEBUG:boto:Data: 2023-05-24T21:14:36.513 DEBUG:boto:Headers: {} 2023-05-24T21:14:36.514 DEBUG:boto:Host: smithi157.front.sepia.ceph.com:8001 2023-05-24T21:14:36.514 DEBUG:boto:Port: 8001 2023-05-24T21:14:36.514 DEBUG:boto:Params: {} 2023-05-24T21:14:36.515 DEBUG:boto:Token: None 2023-05-24T21:14:36.515 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:14:36 GMT /qskrps-30/ 2023-05-24T21:14:36.516 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Pd7JoOjxBs7yuUqqM3Q6+Eu7mW4= 2023-05-24T21:14:36.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Pd7JoOjxBs7yuUqqM3Q6+Eu7mW4=', 'Content-Length': '0'} 2023-05-24T21:14:36.543 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003de94303c2b92df5-00646e7e3c-1104-b2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:14:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:36.544 DEBUG:boto:path=/qskrps-27/ 2023-05-24T21:14:36.544 DEBUG:boto:auth_path=/qskrps-27/ 2023-05-24T21:14:36.544 DEBUG:boto:path=/qskrps-27/?cors 2023-05-24T21:14:36.545 DEBUG:boto:auth_path=/qskrps-27/?cors 2023-05-24T21:14:36.546 DEBUG:boto:Method: PUT 2023-05-24T21:14:36.546 DEBUG:boto:Path: /qskrps-27/?cors 2023-05-24T21:14:36.546 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2023-05-24T21:14:36.547 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2023-05-24T21:14:36.547 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:14:36.547 DEBUG:boto:Port: 8000 2023-05-24T21:14:36.548 DEBUG:boto:Params: {} 2023-05-24T21:14:36.548 DEBUG:boto:Token: None 2023-05-24T21:14:36.549 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Wed, 24 May 2023 21:14:36 GMT /qskrps-27/?cors 2023-05-24T21:14:36.549 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:PjTf4ar71etigQVXa9+jY3UkBWY= 2023-05-24T21:14:36.549 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PjTf4ar71etigQVXa9+jY3UkBWY=', 'Content-Length': '218'} 2023-05-24T21:14:36.557 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e480e2bb814b3938-00646e7e3c-110d-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:14:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:36.558 DEBUG:boto:path=/qskrps-27/ 2023-05-24T21:14:36.559 DEBUG:boto:auth_path=/qskrps-27/ 2023-05-24T21:14:36.559 DEBUG:boto:path=/qskrps-27/?cors 2023-05-24T21:14:36.560 DEBUG:boto:auth_path=/qskrps-27/?cors 2023-05-24T21:14:36.561 DEBUG:boto:Method: GET 2023-05-24T21:14:36.561 DEBUG:boto:Path: /qskrps-27/?cors 2023-05-24T21:14:36.562 DEBUG:boto:Data: 2023-05-24T21:14:36.562 DEBUG:boto:Headers: {} 2023-05-24T21:14:36.563 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:14:36.563 DEBUG:boto:Port: 8000 2023-05-24T21:14:36.564 DEBUG:boto:Params: {} 2023-05-24T21:14:36.565 DEBUG:boto:Token: None 2023-05-24T21:14:36.565 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:14:36 GMT /qskrps-27/?cors 2023-05-24T21:14:36.566 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:aTU87ts1mhj3VtNKqnorvuPwkno= 2023-05-24T21:14:36.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:aTU87ts1mhj3VtNKqnorvuPwkno=', 'Content-Length': '0'} 2023-05-24T21:14:36.568 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000537e5cf84c1b6592-00646e7e3c-110d-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Wed, 24 May 2023 21:14:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:36.569 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2023-05-24T21:14:36.570 DEBUG:boto:path=/qskrps-28/ 2023-05-24T21:14:36.571 DEBUG:boto:auth_path=/qskrps-28/ 2023-05-24T21:14:36.572 DEBUG:boto:path=/qskrps-28/?cors 2023-05-24T21:14:36.572 DEBUG:boto:auth_path=/qskrps-28/?cors 2023-05-24T21:14:36.573 DEBUG:boto:Method: PUT 2023-05-24T21:14:36.573 DEBUG:boto:Path: /qskrps-28/?cors 2023-05-24T21:14:36.574 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2023-05-24T21:14:36.575 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2023-05-24T21:14:36.575 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:14:36.576 DEBUG:boto:Port: 8001 2023-05-24T21:14:36.577 DEBUG:boto:Params: {} 2023-05-24T21:14:36.577 DEBUG:boto:Token: None 2023-05-24T21:14:36.578 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Wed, 24 May 2023 21:14:36 GMT /qskrps-28/?cors 2023-05-24T21:14:36.579 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:auf9sC//ZSHdm5NL7WABMieAyEw= 2023-05-24T21:14:36.579 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:36 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:auf9sC//ZSHdm5NL7WABMieAyEw=', 'Content-Length': '218'} 2023-05-24T21:14:37.590 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e8b56ad79a20e9a0-00646e7e3c-1111-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:14:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:37.591 DEBUG:boto:path=/qskrps-28/ 2023-05-24T21:14:37.592 DEBUG:boto:auth_path=/qskrps-28/ 2023-05-24T21:14:37.592 DEBUG:boto:path=/qskrps-28/?cors 2023-05-24T21:14:37.592 DEBUG:boto:auth_path=/qskrps-28/?cors 2023-05-24T21:14:37.593 DEBUG:boto:Method: GET 2023-05-24T21:14:37.593 DEBUG:boto:Path: /qskrps-28/?cors 2023-05-24T21:14:37.593 DEBUG:boto:Data: 2023-05-24T21:14:37.594 DEBUG:boto:Headers: {} 2023-05-24T21:14:37.594 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:14:37.594 DEBUG:boto:Port: 8001 2023-05-24T21:14:37.595 DEBUG:boto:Params: {} 2023-05-24T21:14:37.595 DEBUG:boto:Token: None 2023-05-24T21:14:37.595 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:14:37 GMT /qskrps-28/?cors 2023-05-24T21:14:37.596 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Dgo7FG+DTQHRs6HDXPfA92tjuv0= 2023-05-24T21:14:37.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Dgo7FG+DTQHRs6HDXPfA92tjuv0=', 'Content-Length': '0'} 2023-05-24T21:14:37.598 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000048697ccf695d935a-00646e7e3d-1111-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Wed, 24 May 2023 21:14:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:37.598 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2023-05-24T21:14:37.599 DEBUG:boto:path=/qskrps-29/ 2023-05-24T21:14:37.599 DEBUG:boto:auth_path=/qskrps-29/ 2023-05-24T21:14:37.600 DEBUG:boto:path=/qskrps-29/?cors 2023-05-24T21:14:37.600 DEBUG:boto:auth_path=/qskrps-29/?cors 2023-05-24T21:14:37.600 DEBUG:boto:Method: PUT 2023-05-24T21:14:37.601 DEBUG:boto:Path: /qskrps-29/?cors 2023-05-24T21:14:37.601 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2023-05-24T21:14:37.602 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2023-05-24T21:14:37.602 DEBUG:boto:Host: smithi157.front.sepia.ceph.com:8000 2023-05-24T21:14:37.602 DEBUG:boto:Port: 8000 2023-05-24T21:14:37.602 DEBUG:boto:Params: {} 2023-05-24T21:14:37.603 DEBUG:boto:Token: None 2023-05-24T21:14:37.604 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Wed, 24 May 2023 21:14:37 GMT /qskrps-29/?cors 2023-05-24T21:14:37.604 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:UVfJMQKePlqdT+GCGKe84bw3IQc= 2023-05-24T21:14:37.604 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:UVfJMQKePlqdT+GCGKe84bw3IQc=', 'Content-Length': '218'} 2023-05-24T21:14:38.619 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000505554e3631c4d42-00646e7e3d-10fe-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:14:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:38.620 DEBUG:boto:path=/qskrps-29/ 2023-05-24T21:14:38.621 DEBUG:boto:auth_path=/qskrps-29/ 2023-05-24T21:14:38.621 DEBUG:boto:path=/qskrps-29/?cors 2023-05-24T21:14:38.621 DEBUG:boto:auth_path=/qskrps-29/?cors 2023-05-24T21:14:38.622 DEBUG:boto:Method: GET 2023-05-24T21:14:38.622 DEBUG:boto:Path: /qskrps-29/?cors 2023-05-24T21:14:38.622 DEBUG:boto:Data: 2023-05-24T21:14:38.623 DEBUG:boto:Headers: {} 2023-05-24T21:14:38.623 DEBUG:boto:Host: smithi157.front.sepia.ceph.com:8000 2023-05-24T21:14:38.623 DEBUG:boto:Port: 8000 2023-05-24T21:14:38.623 DEBUG:boto:Params: {} 2023-05-24T21:14:38.624 DEBUG:boto:Token: None 2023-05-24T21:14:38.624 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:14:38 GMT /qskrps-29/?cors 2023-05-24T21:14:38.625 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:fiY7KNKPxF16xKRs/QohAXMiyHw= 2023-05-24T21:14:38.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:fiY7KNKPxF16xKRs/QohAXMiyHw=', 'Content-Length': '0'} 2023-05-24T21:14:38.626 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b81cf6460a67cc46-00646e7e3e-10fe-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Wed, 24 May 2023 21:14:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:38.668 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2023-05-24T21:14:38.669 DEBUG:boto:path=/qskrps-30/ 2023-05-24T21:14:38.670 DEBUG:boto:auth_path=/qskrps-30/ 2023-05-24T21:14:38.670 DEBUG:boto:path=/qskrps-30/?cors 2023-05-24T21:14:38.671 DEBUG:boto:auth_path=/qskrps-30/?cors 2023-05-24T21:14:38.671 DEBUG:boto:Method: PUT 2023-05-24T21:14:38.671 DEBUG:boto:Path: /qskrps-30/?cors 2023-05-24T21:14:38.672 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2023-05-24T21:14:38.672 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2023-05-24T21:14:38.672 DEBUG:boto:Host: smithi157.front.sepia.ceph.com:8001 2023-05-24T21:14:38.673 DEBUG:boto:Port: 8001 2023-05-24T21:14:38.673 DEBUG:boto:Params: {} 2023-05-24T21:14:38.673 DEBUG:boto:Token: None 2023-05-24T21:14:38.674 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Wed, 24 May 2023 21:14:38 GMT /qskrps-30/?cors 2023-05-24T21:14:38.674 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:cFkyLMAxW+dkvXLzX0XeEDhAm7s= 2023-05-24T21:14:38.675 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:cFkyLMAxW+dkvXLzX0XeEDhAm7s=', 'Content-Length': '218'} 2023-05-24T21:14:39.686 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e022bda8911bc43f-00646e7e3e-1104-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:14:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:39.686 DEBUG:boto:path=/qskrps-30/ 2023-05-24T21:14:39.687 DEBUG:boto:auth_path=/qskrps-30/ 2023-05-24T21:14:39.687 DEBUG:boto:path=/qskrps-30/?cors 2023-05-24T21:14:39.688 DEBUG:boto:auth_path=/qskrps-30/?cors 2023-05-24T21:14:39.688 DEBUG:boto:Method: GET 2023-05-24T21:14:39.689 DEBUG:boto:Path: /qskrps-30/?cors 2023-05-24T21:14:39.690 DEBUG:boto:Data: 2023-05-24T21:14:39.690 DEBUG:boto:Headers: {} 2023-05-24T21:14:39.690 DEBUG:boto:Host: smithi157.front.sepia.ceph.com:8001 2023-05-24T21:14:39.691 DEBUG:boto:Port: 8001 2023-05-24T21:14:39.691 DEBUG:boto:Params: {} 2023-05-24T21:14:39.691 DEBUG:boto:Token: None 2023-05-24T21:14:39.692 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:14:39 GMT /qskrps-30/?cors 2023-05-24T21:14:39.692 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:MWoiZzzCo4PuTvtX9hycCgq23hM= 2023-05-24T21:14:39.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:MWoiZzzCo4PuTvtX9hycCgq23hM=', 'Content-Length': '0'} 2023-05-24T21:14:39.694 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000056c84a2e13dc5ae8-00646e7e3f-1104-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Wed, 24 May 2023 21:14:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:39.736 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2023-05-24T21:14:39.737 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_cors ... ok 2023-05-24T21:14:39.738 INFO:rgw_multi.tests:create bucket zone=a1 name=qskrps-31 2023-05-24T21:14:39.739 DEBUG:boto:path=/qskrps-31/ 2023-05-24T21:14:39.739 DEBUG:boto:auth_path=/qskrps-31/ 2023-05-24T21:14:39.739 DEBUG:boto:Method: PUT 2023-05-24T21:14:39.740 DEBUG:boto:Path: /qskrps-31/ 2023-05-24T21:14:39.740 DEBUG:boto:Data: 2023-05-24T21:14:39.741 DEBUG:boto:Headers: {} 2023-05-24T21:14:39.741 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:14:39.741 DEBUG:boto:Port: 8000 2023-05-24T21:14:39.742 DEBUG:boto:Params: {} 2023-05-24T21:14:39.742 DEBUG:boto:Token: None 2023-05-24T21:14:39.743 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:14:39 GMT /qskrps-31/ 2023-05-24T21:14:39.743 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:vLYWcdUnBouA8hcKtyrb5oyrnIo= 2023-05-24T21:14:39.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:vLYWcdUnBouA8hcKtyrb5oyrnIo=', 'Content-Length': '0'} 2023-05-24T21:14:39.765 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000467287a07883a4c0-00646e7e3f-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:14:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:39.766 INFO:rgw_multi.tests:create bucket zone=a2 name=qskrps-32 2023-05-24T21:14:39.766 DEBUG:boto:path=/qskrps-32/ 2023-05-24T21:14:39.767 DEBUG:boto:auth_path=/qskrps-32/ 2023-05-24T21:14:39.767 DEBUG:boto:Method: PUT 2023-05-24T21:14:39.768 DEBUG:boto:Path: /qskrps-32/ 2023-05-24T21:14:39.768 DEBUG:boto:Data: 2023-05-24T21:14:39.768 DEBUG:boto:Headers: {} 2023-05-24T21:14:39.769 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:14:39.769 DEBUG:boto:Port: 8001 2023-05-24T21:14:39.769 DEBUG:boto:Params: {} 2023-05-24T21:14:39.770 DEBUG:boto:Token: None 2023-05-24T21:14:39.770 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:14:39 GMT /qskrps-32/ 2023-05-24T21:14:39.771 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:NGkqNdWhWtOX19Iq/EwcEmjXwCw= 2023-05-24T21:14:39.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:39 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:NGkqNdWhWtOX19Iq/EwcEmjXwCw=', 'Content-Length': '0'} 2023-05-24T21:14:39.811 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e9d820f7b15bf00a-00646e7e3f-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:14:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:39.812 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:14:39.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:14:39.813 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:14:39.863 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:14:39.864 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:14:39.914 INFO:teuthology.orchestra.run.smithi071.stderr:No --period given, using current period=5c04936b-70ea-4415-b4eb-3aad3dc612e5 2023-05-24T21:14:39.932 INFO:teuthology.orchestra.run.smithi071.stdout:[ 2023-05-24T21:14:39.932 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:14:39.933 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962879.789267_261.1", 2023-05-24T21:14:39.933 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:14:39.789267Z" 2023-05-24T21:14:39.933 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:39.933 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:14:39.933 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:14:39.934 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:06:32.333844Z" 2023-05-24T21:14:39.934 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:39.934 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:14:39.934 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:14:39.934 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:14:39.934 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:39.934 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:14:39.935 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:14:39.935 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:14:39.935 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:14:39.935 INFO:teuthology.orchestra.run.smithi071.stdout:] 2023-05-24T21:14:39.936 DEBUG:rgw_multi.tests:master meta markers={0: '1_1684962879.789267_261.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:14:39.937 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-24T21:14:39.937 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:14:39.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:14:39.938 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:14:39.989 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:14:39.989 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:14:40.024 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:14:40.017+0000 7fc7c9cc6780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:14:40.024 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:14:40.018+0000 7fc7c9cc6780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:14:40.036 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:14:40.036 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:14:40.037 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:14:40.037 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:14:40.038 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:14:40.038 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:14:40.038 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:14:40.038 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:40.038 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:14:40.038 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:14:40.038 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:14:40.039 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:14:40.039 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:14:40.039 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962879.789267_261.1", 2023-05-24T21:14:40.040 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:14:40.040 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:14:40.040 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:14:40.040 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:14:39.789267Z", 2023-05-24T21:14:40.040 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:14:40.040 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:14:40.041 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:40.041 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:14:40.041 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:14:40.041 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:14:40.041 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:14:40.042 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:14:40.042 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:14:40.042 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:14:40.042 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:14:40.042 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:14:40.042 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:14:40.042 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:14:40.043 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:40.043 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:14:40.043 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:14:40.043 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:14:40.043 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:14:40.043 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:14:40.044 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:14:40.044 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:14:40.044 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:14:40.044 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:14:40.044 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:14:40.044 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:14:40.044 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:40.045 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:14:40.045 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:14:40.045 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:14:40.045 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:14:40.045 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:14:40.045 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:14:40.045 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:14:40.046 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:14:40.046 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:14:40.046 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:14:40.046 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:14:40.046 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:14:40.046 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:14:40.047 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:40.047 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:14:40.047 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:14:40.047 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:14:40.047 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:40.047 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:14:40Z" 2023-05-24T21:14:40.048 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:14:40.048 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962879.789267_261.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:14:39.789267Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:14:40Z" } 2023-05-24T21:14:40.049 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962879.789267_261.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:14:39.789267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:14:40.049 DEBUG:rgw_multi.tests:log_status={0: '1_1684962879.789267_261.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:14:40.050 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962879.789267_261.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:14:40.050 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-24T21:14:40.050 DEBUG:boto:path=/qskrps-31/ 2023-05-24T21:14:40.051 DEBUG:boto:auth_path=/qskrps-31/ 2023-05-24T21:14:40.051 DEBUG:boto:Method: HEAD 2023-05-24T21:14:40.052 DEBUG:boto:Path: /qskrps-31/ 2023-05-24T21:14:40.053 DEBUG:boto:Data: 2023-05-24T21:14:40.053 DEBUG:boto:Headers: {} 2023-05-24T21:14:40.054 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:14:40.055 DEBUG:boto:Port: 8000 2023-05-24T21:14:40.056 DEBUG:boto:Params: {} 2023-05-24T21:14:40.057 DEBUG:boto:Token: None 2023-05-24T21:14:40.057 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:14:40 GMT /qskrps-31/ 2023-05-24T21:14:40.058 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:RUPgXHOZ7mebmu8kUaiGmM1Vv8w= 2023-05-24T21:14:40.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:40 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:RUPgXHOZ7mebmu8kUaiGmM1Vv8w=', 'Content-Length': '0'} 2023-05-24T21:14:40.060 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ba263b7cdf80f543-00646e7e40-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:14:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:40.061 DEBUG:boto:path=/qskrps-31/foo 2023-05-24T21:14:40.061 DEBUG:boto:auth_path=/qskrps-31/foo 2023-05-24T21:14:40.062 DEBUG:boto:Method: PUT 2023-05-24T21:14:40.062 DEBUG:boto:Path: /qskrps-31/foo 2023-05-24T21:14:40.062 DEBUG:boto:Data: 2023-05-24T21:14:40.063 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-24T21:14:40.063 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:14:40.063 DEBUG:boto:Port: 8000 2023-05-24T21:14:40.064 DEBUG:boto:Params: {} 2023-05-24T21:14:40.064 DEBUG:boto:Token: None 2023-05-24T21:14:40.064 DEBUG:boto:StringToSign: PUT N7UdGUp1E+RbVvZSTy1R8g== application/octet-stream Wed, 24 May 2023 21:14:40 GMT /qskrps-31/foo 2023-05-24T21:14:40.065 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:+HSLyJivYwg0rNo6KHHXOZ1qoio= 2023-05-24T21:14:40.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:14:40 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:+HSLyJivYwg0rNo6KHHXOZ1qoio='} 2023-05-24T21:14:40.071 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"37b51d194a7513e45b56f6524f2d51f2"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000058c36ac928235c64-00646e7e40-110d-a1'), ('Date', 'Wed, 24 May 2023 21:14:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:40.072 DEBUG:boto:path=/qskrps-31/ 2023-05-24T21:14:40.072 DEBUG:boto:auth_path=/qskrps-31/ 2023-05-24T21:14:40.072 DEBUG:boto:Method: DELETE 2023-05-24T21:14:40.073 DEBUG:boto:Path: /qskrps-31/ 2023-05-24T21:14:40.073 DEBUG:boto:Data: 2023-05-24T21:14:40.073 DEBUG:boto:Headers: {} 2023-05-24T21:14:40.074 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:14:40.074 DEBUG:boto:Port: 8000 2023-05-24T21:14:40.074 DEBUG:boto:Params: {} 2023-05-24T21:14:40.074 DEBUG:boto:Token: None 2023-05-24T21:14:40.075 DEBUG:boto:StringToSign: DELETE Wed, 24 May 2023 21:14:40 GMT /qskrps-31/ 2023-05-24T21:14:40.075 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:cvIlJ9uInpiV+Tp0KtJlGC96HgQ= 2023-05-24T21:14:40.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:40 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:cvIlJ9uInpiV+Tp0KtJlGC96HgQ=', 'Content-Length': '0'} 2023-05-24T21:14:40.081 DEBUG:boto:Response headers: [('Content-Length', '224'), ('x-amz-request-id', 'tx00000bfed3906dec3e314-00646e7e40-110d-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:14:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:40.124 DEBUG:boto:path=/qskrps-32/ 2023-05-24T21:14:40.125 DEBUG:boto:auth_path=/qskrps-32/ 2023-05-24T21:14:40.125 DEBUG:boto:Method: HEAD 2023-05-24T21:14:40.126 DEBUG:boto:Path: /qskrps-32/ 2023-05-24T21:14:40.126 DEBUG:boto:Data: 2023-05-24T21:14:40.126 DEBUG:boto:Headers: {} 2023-05-24T21:14:40.127 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:14:40.127 DEBUG:boto:Port: 8001 2023-05-24T21:14:40.127 DEBUG:boto:Params: {} 2023-05-24T21:14:40.128 DEBUG:boto:Token: None 2023-05-24T21:14:40.128 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:14:40 GMT /qskrps-32/ 2023-05-24T21:14:40.129 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:YfkMepT3nAU2N1BzMqsKIQ/tGZk= 2023-05-24T21:14:40.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:40 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:YfkMepT3nAU2N1BzMqsKIQ/tGZk=', 'Content-Length': '0'} 2023-05-24T21:14:40.131 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004b788b7f003ba3b8-00646e7e40-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:14:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:40.132 DEBUG:boto:path=/qskrps-32/foo 2023-05-24T21:14:40.132 DEBUG:boto:auth_path=/qskrps-32/foo 2023-05-24T21:14:40.132 DEBUG:boto:Method: PUT 2023-05-24T21:14:40.133 DEBUG:boto:Path: /qskrps-32/foo 2023-05-24T21:14:40.133 DEBUG:boto:Data: 2023-05-24T21:14:40.134 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-24T21:14:40.134 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:14:40.134 DEBUG:boto:Port: 8001 2023-05-24T21:14:40.135 DEBUG:boto:Params: {} 2023-05-24T21:14:40.135 DEBUG:boto:Token: None 2023-05-24T21:14:40.135 DEBUG:boto:StringToSign: PUT N7UdGUp1E+RbVvZSTy1R8g== application/octet-stream Wed, 24 May 2023 21:14:40 GMT /qskrps-32/foo 2023-05-24T21:14:40.136 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:mJwG5yhbCdMJ5Vg667cq+8SgEP0= 2023-05-24T21:14:40.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:14:40 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:mJwG5yhbCdMJ5Vg667cq+8SgEP0='} 2023-05-24T21:14:40.142 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"37b51d194a7513e45b56f6524f2d51f2"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d277757df7aad8cb-00646e7e40-1111-a2'), ('Date', 'Wed, 24 May 2023 21:14:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:40.143 DEBUG:boto:path=/qskrps-32/ 2023-05-24T21:14:40.143 DEBUG:boto:auth_path=/qskrps-32/ 2023-05-24T21:14:40.143 DEBUG:boto:Method: DELETE 2023-05-24T21:14:40.144 DEBUG:boto:Path: /qskrps-32/ 2023-05-24T21:14:40.144 DEBUG:boto:Data: 2023-05-24T21:14:40.144 DEBUG:boto:Headers: {} 2023-05-24T21:14:40.145 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:14:40.145 DEBUG:boto:Port: 8001 2023-05-24T21:14:40.145 DEBUG:boto:Params: {} 2023-05-24T21:14:40.146 DEBUG:boto:Token: None 2023-05-24T21:14:40.146 DEBUG:boto:StringToSign: DELETE Wed, 24 May 2023 21:14:40 GMT /qskrps-32/ 2023-05-24T21:14:40.146 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:P3Z3N4qTGQU5Shj12ZC4BL2jyh4= 2023-05-24T21:14:40.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:40 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:P3Z3N4qTGQU5Shj12ZC4BL2jyh4=', 'Content-Length': '0'} 2023-05-24T21:14:40.153 DEBUG:boto:Response headers: [('Content-Length', '224'), ('x-amz-request-id', 'tx00000b5e4f65cbe4395a6-00646e7e40-1111-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:14:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:40.196 DEBUG:boto:path=/qskrps-31/ 2023-05-24T21:14:40.197 DEBUG:boto:auth_path=/qskrps-31/ 2023-05-24T21:14:40.197 DEBUG:boto:Method: HEAD 2023-05-24T21:14:40.198 DEBUG:boto:Path: /qskrps-31/ 2023-05-24T21:14:40.198 DEBUG:boto:Data: 2023-05-24T21:14:40.198 DEBUG:boto:Headers: {} 2023-05-24T21:14:40.199 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:14:40.199 DEBUG:boto:Port: 8000 2023-05-24T21:14:40.199 DEBUG:boto:Params: {} 2023-05-24T21:14:40.200 DEBUG:boto:Token: None 2023-05-24T21:14:40.200 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:14:40 GMT /qskrps-31/ 2023-05-24T21:14:40.201 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:RUPgXHOZ7mebmu8kUaiGmM1Vv8w= 2023-05-24T21:14:40.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:40 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:RUPgXHOZ7mebmu8kUaiGmM1Vv8w=', 'Content-Length': '0'} 2023-05-24T21:14:40.204 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009e7318abb2e793ca-00646e7e40-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:14:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:40.204 DEBUG:boto:path=/qskrps-32/ 2023-05-24T21:14:40.205 DEBUG:boto:auth_path=/qskrps-32/ 2023-05-24T21:14:40.205 DEBUG:boto:Method: HEAD 2023-05-24T21:14:40.205 DEBUG:boto:Path: /qskrps-32/ 2023-05-24T21:14:40.206 DEBUG:boto:Data: 2023-05-24T21:14:40.206 DEBUG:boto:Headers: {} 2023-05-24T21:14:40.206 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:14:40.207 DEBUG:boto:Port: 8000 2023-05-24T21:14:40.207 DEBUG:boto:Params: {} 2023-05-24T21:14:40.207 DEBUG:boto:Token: None 2023-05-24T21:14:40.208 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:14:40 GMT /qskrps-32/ 2023-05-24T21:14:40.208 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:YfkMepT3nAU2N1BzMqsKIQ/tGZk= 2023-05-24T21:14:40.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:40 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:YfkMepT3nAU2N1BzMqsKIQ/tGZk=', 'Content-Length': '0'} 2023-05-24T21:14:40.210 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000caa22f35110d24a8-00646e7e40-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:14:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:40.211 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_delete_notempty ... ok 2023-05-24T21:14:40.212 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_period_incremental_sync ... SKIP: test_multi_period_incremental_sync skipped. Requires 3 or more zones in master zonegroup. 2023-05-24T21:14:40.237 INFO:rgw_multi.tests:create bucket zone=a1 name=qskrps-33 2023-05-24T21:14:40.237 DEBUG:boto:path=/qskrps-33/ 2023-05-24T21:14:40.237 DEBUG:boto:auth_path=/qskrps-33/ 2023-05-24T21:14:40.238 DEBUG:boto:Method: PUT 2023-05-24T21:14:40.238 DEBUG:boto:Path: /qskrps-33/ 2023-05-24T21:14:40.238 DEBUG:boto:Data: 2023-05-24T21:14:40.239 DEBUG:boto:Headers: {} 2023-05-24T21:14:40.239 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:14:40.239 DEBUG:boto:Port: 8000 2023-05-24T21:14:40.240 DEBUG:boto:Params: {} 2023-05-24T21:14:40.240 DEBUG:boto:Token: None 2023-05-24T21:14:40.241 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:14:40 GMT /qskrps-33/ 2023-05-24T21:14:40.241 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:cWf2YuvvAzCJuI5VU3LWTtW/M/o= 2023-05-24T21:14:40.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:40 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:cWf2YuvvAzCJuI5VU3LWTtW/M/o=', 'Content-Length': '0'} 2023-05-24T21:14:40.255 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006fcd9d2f88e07927-00646e7e40-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:14:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:40.256 INFO:rgw_multi.tests:create bucket zone=a2 name=qskrps-34 2023-05-24T21:14:40.256 DEBUG:boto:path=/qskrps-34/ 2023-05-24T21:14:40.257 DEBUG:boto:auth_path=/qskrps-34/ 2023-05-24T21:14:40.257 DEBUG:boto:Method: PUT 2023-05-24T21:14:40.257 DEBUG:boto:Path: /qskrps-34/ 2023-05-24T21:14:40.258 DEBUG:boto:Data: 2023-05-24T21:14:40.258 DEBUG:boto:Headers: {} 2023-05-24T21:14:40.258 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:14:40.259 DEBUG:boto:Port: 8001 2023-05-24T21:14:40.259 DEBUG:boto:Params: {} 2023-05-24T21:14:40.259 DEBUG:boto:Token: None 2023-05-24T21:14:40.260 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:14:40 GMT /qskrps-34/ 2023-05-24T21:14:40.260 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4enNehl9rBmrowZyCVg7K3B3oZQ= 2023-05-24T21:14:40.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:40 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4enNehl9rBmrowZyCVg7K3B3oZQ=', 'Content-Length': '0'} 2023-05-24T21:14:40.298 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a76520c1a3013553-00646e7e40-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:14:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:40.299 DEBUG:boto:path=/qskrps-33/ 2023-05-24T21:14:40.299 DEBUG:boto:auth_path=/qskrps-33/ 2023-05-24T21:14:40.300 DEBUG:boto:Method: HEAD 2023-05-24T21:14:40.300 DEBUG:boto:Path: /qskrps-33/ 2023-05-24T21:14:40.301 DEBUG:boto:Data: 2023-05-24T21:14:40.301 DEBUG:boto:Headers: {} 2023-05-24T21:14:40.302 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:14:40.302 DEBUG:boto:Port: 8000 2023-05-24T21:14:40.302 DEBUG:boto:Params: {} 2023-05-24T21:14:40.303 DEBUG:boto:Token: None 2023-05-24T21:14:40.303 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:14:40 GMT /qskrps-33/ 2023-05-24T21:14:40.304 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:RPCoaNhOGgVSmf8XjfgM7KsXgNQ= 2023-05-24T21:14:40.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:40 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:RPCoaNhOGgVSmf8XjfgM7KsXgNQ=', 'Content-Length': '0'} 2023-05-24T21:14:40.307 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005d7dc13b603cb431-00646e7e40-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:14:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:40.307 DEBUG:boto:path=/qskrps-33/key 2023-05-24T21:14:40.308 DEBUG:boto:auth_path=/qskrps-33/key 2023-05-24T21:14:40.308 DEBUG:boto:Method: PUT 2023-05-24T21:14:40.309 DEBUG:boto:Path: /qskrps-33/key 2023-05-24T21:14:40.310 DEBUG:boto:Data: 2023-05-24T21:14:40.310 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2023-05-24T21:14:40.311 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:14:40.312 DEBUG:boto:Port: 8000 2023-05-24T21:14:40.313 DEBUG:boto:Params: {} 2023-05-24T21:14:40.313 DEBUG:boto:Token: None 2023-05-24T21:14:40.314 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Wed, 24 May 2023 21:14:40 GMT /qskrps-33/key 2023-05-24T21:14:40.315 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:YKN/kBi6tQbCz3pdD8cr/7k83QA= 2023-05-24T21:14:40.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:14:40 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:YKN/kBi6tQbCz3pdD8cr/7k83QA='} 2023-05-24T21:14:40.329 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000338d6f1bc6be2a45-00646e7e40-110d-a1'), ('Date', 'Wed, 24 May 2023 21:14:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:40.330 DEBUG:boto:path=/qskrps-34/ 2023-05-24T21:14:40.330 DEBUG:boto:auth_path=/qskrps-34/ 2023-05-24T21:14:40.331 DEBUG:boto:Method: HEAD 2023-05-24T21:14:40.331 DEBUG:boto:Path: /qskrps-34/ 2023-05-24T21:14:40.332 DEBUG:boto:Data: 2023-05-24T21:14:40.332 DEBUG:boto:Headers: {} 2023-05-24T21:14:40.332 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:14:40.333 DEBUG:boto:Port: 8001 2023-05-24T21:14:40.333 DEBUG:boto:Params: {} 2023-05-24T21:14:40.333 DEBUG:boto:Token: None 2023-05-24T21:14:40.334 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:14:40 GMT /qskrps-34/ 2023-05-24T21:14:40.334 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:LvhaWO3P3WLldUtStskMQ7gcevw= 2023-05-24T21:14:40.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:40 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:LvhaWO3P3WLldUtStskMQ7gcevw=', 'Content-Length': '0'} 2023-05-24T21:14:40.337 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005038a4da2cbc8de5-00646e7e40-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:14:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:40.337 DEBUG:boto:path=/qskrps-34/key 2023-05-24T21:14:40.338 DEBUG:boto:auth_path=/qskrps-34/key 2023-05-24T21:14:40.338 DEBUG:boto:Method: PUT 2023-05-24T21:14:40.339 DEBUG:boto:Path: /qskrps-34/key 2023-05-24T21:14:40.339 DEBUG:boto:Data: 2023-05-24T21:14:40.339 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2023-05-24T21:14:40.339 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:14:40.340 DEBUG:boto:Port: 8001 2023-05-24T21:14:40.340 DEBUG:boto:Params: {} 2023-05-24T21:14:40.341 DEBUG:boto:Token: None 2023-05-24T21:14:40.341 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Wed, 24 May 2023 21:14:40 GMT /qskrps-34/key 2023-05-24T21:14:40.341 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:oVEp4OtNG9PYXMdkHiKsicBARM8= 2023-05-24T21:14:40.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:14:40 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:oVEp4OtNG9PYXMdkHiKsicBARM8='} 2023-05-24T21:14:40.350 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c03ceaca3dc49210-00646e7e40-1111-a2'), ('Date', 'Wed, 24 May 2023 21:14:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:14:40.351 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:14:40.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:14:40.351 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:14:40.402 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:14:40.402 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:14:40.453 INFO:teuthology.orchestra.run.smithi071.stderr:No --period given, using current period=5c04936b-70ea-4415-b4eb-3aad3dc612e5 2023-05-24T21:14:40.471 INFO:teuthology.orchestra.run.smithi071.stdout:[ 2023-05-24T21:14:40.471 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:14:40.471 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962880.273629_269.1", 2023-05-24T21:14:40.472 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:14:40.273629Z" 2023-05-24T21:14:40.472 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:40.472 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:14:40.472 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:14:40.472 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:06:32.333844Z" 2023-05-24T21:14:40.472 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:40.473 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:14:40.473 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:14:40.473 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:14:40.473 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:40.473 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:14:40.473 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:14:40.473 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:14:40.474 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:14:40.474 INFO:teuthology.orchestra.run.smithi071.stdout:] 2023-05-24T21:14:40.475 DEBUG:rgw_multi.tests:master meta markers={0: '1_1684962880.273629_269.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:14:40.475 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-24T21:14:40.476 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:14:40.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:14:40.476 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:14:40.525 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:14:40.526 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:14:40.555 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:14:40.548+0000 7f125cfe4780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:14:40.556 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:14:40.549+0000 7f125cfe4780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:14:40.568 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:14:40.568 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:14:40.568 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:14:40.569 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:14:40.569 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:14:40.569 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:14:40.569 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:14:40.569 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:40.569 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:14:40.570 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:14:40.570 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:14:40.570 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:14:40.570 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:14:40.570 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962880.243018_262.1", 2023-05-24T21:14:40.571 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:14:40.571 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:14:40.571 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:14:40.572 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:14:40.243018Z", 2023-05-24T21:14:40.572 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:14:40.572 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:14:40.573 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:40.573 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:14:40.573 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:14:40.574 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:14:40.574 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:14:40.574 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:14:40.575 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:14:40.575 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:14:40.575 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:14:40.576 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:14:40.576 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:14:40.576 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:14:40.577 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:40.577 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:14:40.577 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:14:40.578 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:14:40.578 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:14:40.578 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:14:40.578 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:14:40.579 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:14:40.579 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:14:40.579 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:14:40.579 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:14:40.580 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:14:40.580 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:40.580 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:14:40.581 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:14:40.581 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:14:40.581 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:14:40.581 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:14:40.581 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:14:40.582 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:14:40.582 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:14:40.582 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:14:40.582 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:14:40.582 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:14:40.583 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:14:40.583 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:14:40.583 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:40.583 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:14:40.583 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:14:40.583 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:14:40.583 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:40.584 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:14:40Z" 2023-05-24T21:14:40.584 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:14:40.585 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962880.243018_262.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:14:40.243018Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:14:40Z" } 2023-05-24T21:14:40.585 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962880.243018_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:14:40.243018Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:14:40.586 DEBUG:rgw_multi.tests:log_status={0: '1_1684962880.273629_269.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:14:40.586 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962880.243018_262.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:14:40.587 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684962880.273629_269.1 target=1_1684962880.243018_262.1 2023-05-24T21:14:45.587 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:14:45.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:14:45.589 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:14:45.645 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:14:45.646 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:14:45.690 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:14:45.684+0000 7f0a0b595780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:14:45.691 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:14:45.686+0000 7f0a0b595780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:14:45.703 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:14:45.704 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:14:45.704 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:14:45.704 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:14:45.704 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:14:45.704 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:14:45.705 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:14:45.705 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:45.705 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:14:45.705 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:14:45.705 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:14:45.705 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:14:45.705 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:14:45.706 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962880.273629_269.1", 2023-05-24T21:14:45.706 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:14:45.706 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:14:45.706 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:14:45.706 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:14:40.273629Z", 2023-05-24T21:14:45.706 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:14:45.707 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:14:45.707 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:45.707 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:14:45.707 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:14:45.707 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:14:45.707 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:14:45.707 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:14:45.708 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:14:45.708 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:14:45.708 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:14:45.708 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:14:45.708 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:14:45.708 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:14:45.708 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:45.709 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:14:45.709 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:14:45.709 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:14:45.709 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:14:45.709 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:14:45.709 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:14:45.709 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:14:45.710 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:14:45.710 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:14:45.710 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:14:45.710 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:14:45.710 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:45.710 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:14:45.710 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:14:45.711 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:14:45.711 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:14:45.711 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:14:45.711 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:14:45.711 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:14:45.711 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:14:45.711 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:14:45.712 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:14:45.712 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:14:45.712 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:14:45.712 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:14:45.712 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:45.712 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:14:45.712 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:14:45.713 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:14:45.713 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:45.713 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:14:45Z" 2023-05-24T21:14:45.713 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:14:45.714 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962880.273629_269.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:14:40.273629Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:14:45Z" } 2023-05-24T21:14:45.714 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962880.273629_269.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:14:40.273629Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:14:45.715 DEBUG:rgw_multi.tests:log_status={0: '1_1684962880.273629_269.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:14:45.715 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962880.273629_269.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:14:45.715 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-24T21:14:45.716 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-05-24T21:14:45.716 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:14:45.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:14:45.717 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:14:45.769 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:14:45.770 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:14:45.824 INFO:teuthology.orchestra.run.smithi071.stdout:[ 2023-05-24T21:14:45.824 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:14:45.825 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000002627", 2023-05-24T21:14:45.825 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:10:45.893198Z" 2023-05-24T21:14:45.825 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:45.825 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:14:45.825 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000001068", 2023-05-24T21:14:45.825 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:14:40.065760Z" 2023-05-24T21:14:45.826 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:45.826 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:14:45.826 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-05-24T21:14:45.826 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:06:31.177575Z" 2023-05-24T21:14:45.826 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:45.826 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:14:45.826 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000004452", 2023-05-24T21:14:45.827 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:14:40.318777Z" 2023-05-24T21:14:45.827 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:14:45.827 INFO:teuthology.orchestra.run.smithi071.stdout:] 2023-05-24T21:14:45.828 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000002627', 1: '00000000000000000000:00000000000000001068', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004452'} 2023-05-24T21:14:45.828 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-05-24T21:14:45.829 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:14:45.829 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:14:45.830 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:14:45.880 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:14:45.881 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:14:45.953 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:14:45.954 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:14:45.954 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:14:45.954 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:14:45.954 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:14:45.954 INFO:teuthology.orchestra.run.smithi071.stdout: "instance_id": 13190874283549403954 2023-05-24T21:14:45.955 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:45.955 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:14:45.955 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:14:45.955 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:14:45.955 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:14:45.955 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:14:45.956 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000002627", 2023-05-24T21:14:45.956 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:14:45.956 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:14:45.956 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:14:45.956 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:10:45.893198Z" 2023-05-24T21:14:45.956 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:14:45.957 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:45.957 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:14:45.957 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:14:45.957 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:14:45.957 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:14:45.957 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-05-24T21:14:45.957 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:14:45.958 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:14:45.958 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:14:45.958 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:03:11.196517Z" 2023-05-24T21:14:45.958 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:14:45.958 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:45.958 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:14:45.959 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:14:45.959 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:14:45.959 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:14:45.959 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-05-24T21:14:45.959 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:14:45.959 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:14:45.959 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:14:45.960 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:31.177575Z" 2023-05-24T21:14:45.960 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:14:45.960 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:45.960 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:14:45.961 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:14:45.961 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:14:45.961 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:14:45.961 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000004311", 2023-05-24T21:14:45.961 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:14:45.961 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:14:45.961 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:14:45.962 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:14:25.907870Z" 2023-05-24T21:14:45.962 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:14:45.962 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:14:45.962 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:14:45.962 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:45.962 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:14:45.962 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 0, 2023-05-24T21:14:45.963 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 0 2023-05-24T21:14:45.963 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:45.963 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:14:45Z" 2023-05-24T21:14:45.963 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:14:45.964 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13190874283549403954 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002627", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:10:45.893198Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000929", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:03:11.196517Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:31.177575Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004311", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:14:25.907870Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-05-24T21:14:45Z" } 2023-05-24T21:14:45.965 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002627', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:10:45.893198Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000929', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:03:11.196517Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:31.177575Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004311', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:14:25.907870Z'}}] 2023-05-24T21:14:45.965 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002627', 1: '00000000000000000000:00000000000000001068', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004452'} 2023-05-24T21:14:45.965 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002627', 1: '00000000000000000000:00000000000000000929', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004311'} 2023-05-24T21:14:45.966 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000001068 target=00000000000000000000:00000000000000000929, shard=3 master=00000000000000000000:00000000000000004452 target=00000000000000000000:00000000000000004311 2023-05-24T21:14:50.967 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:14:50.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:14:50.968 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:14:51.027 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:14:51.028 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:14:51.085 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:14:51.086 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:14:51.086 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:14:51.086 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:14:51.086 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:14:51.087 INFO:teuthology.orchestra.run.smithi071.stdout: "instance_id": 13190874283549403954 2023-05-24T21:14:51.087 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:51.087 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:14:51.087 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:14:51.088 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:14:51.088 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:14:51.088 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:14:51.088 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000002627", 2023-05-24T21:14:51.088 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:14:51.089 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:14:51.089 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:14:51.089 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:10:45.893198Z" 2023-05-24T21:14:51.089 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:14:51.089 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:51.089 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:14:51.089 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:14:51.090 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:14:51.090 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:14:51.090 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-05-24T21:14:51.090 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:14:51.090 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:14:51.090 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:14:51.091 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:03:11.196517Z" 2023-05-24T21:14:51.091 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:14:51.091 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:51.091 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:14:51.092 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:14:51.092 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:14:51.092 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:14:51.092 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-05-24T21:14:51.092 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:14:51.092 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:14:51.093 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:14:51.093 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:31.177575Z" 2023-05-24T21:14:51.093 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:14:51.093 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:51.093 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:14:51.093 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:14:51.094 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:14:51.094 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:14:51.094 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000004311", 2023-05-24T21:14:51.094 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:14:51.094 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:14:51.094 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:14:51.095 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:14:25.907870Z" 2023-05-24T21:14:51.095 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:14:51.095 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:14:51.095 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:14:51.096 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:51.096 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:14:51.096 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 0, 2023-05-24T21:14:51.097 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 0 2023-05-24T21:14:51.097 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:51.097 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:14:51Z" 2023-05-24T21:14:51.097 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:14:51.099 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13190874283549403954 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002627", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:10:45.893198Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000929", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:03:11.196517Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:31.177575Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004311", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:14:25.907870Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-05-24T21:14:51Z" } 2023-05-24T21:14:51.099 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002627', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:10:45.893198Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000929', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:03:11.196517Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:31.177575Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004311', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:14:25.907870Z'}}] 2023-05-24T21:14:51.100 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002627', 1: '00000000000000000000:00000000000000001068', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004452'} 2023-05-24T21:14:51.100 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002627', 1: '00000000000000000000:00000000000000000929', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004311'} 2023-05-24T21:14:51.100 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000001068 target=00000000000000000000:00000000000000000929, shard=3 master=00000000000000000000:00000000000000004452 target=00000000000000000000:00000000000000004311 2023-05-24T21:14:56.102 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:14:56.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:14:56.103 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:14:56.162 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:14:56.162 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:14:56.225 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:14:56.225 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:14:56.225 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:14:56.225 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:14:56.226 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:14:56.226 INFO:teuthology.orchestra.run.smithi071.stdout: "instance_id": 13190874283549403954 2023-05-24T21:14:56.226 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:56.226 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:14:56.226 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:14:56.226 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:14:56.226 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:14:56.227 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:14:56.227 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000002627", 2023-05-24T21:14:56.227 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:14:56.227 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:14:56.227 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:14:56.227 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:10:45.893198Z" 2023-05-24T21:14:56.227 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:14:56.228 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:56.228 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:14:56.228 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:14:56.228 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:14:56.228 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:14:56.228 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000001068", 2023-05-24T21:14:56.229 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:14:56.229 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:14:56.229 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:14:56.229 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:14:40.065760Z" 2023-05-24T21:14:56.229 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:14:56.229 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:56.229 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:14:56.230 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:14:56.230 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:14:56.230 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:14:56.230 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-05-24T21:14:56.230 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:14:56.230 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:14:56.230 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:14:56.231 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:31.177575Z" 2023-05-24T21:14:56.231 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:14:56.231 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:56.231 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:14:56.231 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:14:56.231 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:14:56.231 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:14:56.232 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000004452", 2023-05-24T21:14:56.232 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:14:56.232 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:14:56.232 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:14:56.232 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:14:40.318777Z" 2023-05-24T21:14:56.232 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:14:56.232 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:14:56.233 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:14:56.233 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:56.233 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:14:56.233 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 0, 2023-05-24T21:14:56.233 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 0 2023-05-24T21:14:56.233 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:56.233 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:14:56Z" 2023-05-24T21:14:56.234 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:14:56.234 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13190874283549403954 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002627", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:10:45.893198Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001068", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:14:40.065760Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:31.177575Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004452", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:14:40.318777Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-05-24T21:14:56Z" } 2023-05-24T21:14:56.235 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002627', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:10:45.893198Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001068', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:14:40.065760Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:31.177575Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004452', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:14:40.318777Z'}}] 2023-05-24T21:14:56.235 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002627', 1: '00000000000000000000:00000000000000001068', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004452'} 2023-05-24T21:14:56.236 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002627', 1: '00000000000000000000:00000000000000001068', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004452'} 2023-05-24T21:14:56.236 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-05-24T21:14:56.236 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-05-24T21:14:56.237 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:14:56.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:14:56.238 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:14:56.286 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:14:56.287 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:14:56.338 INFO:teuthology.orchestra.run.smithi071.stdout:[ 2023-05-24T21:14:56.339 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:14:56.339 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000002345", 2023-05-24T21:14:56.339 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:10:23.920015Z" 2023-05-24T21:14:56.339 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:56.339 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:14:56.340 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-05-24T21:14:56.340 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:14:51.309908Z" 2023-05-24T21:14:56.340 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:56.340 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:14:56.340 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-05-24T21:14:56.340 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:03:11.225401Z" 2023-05-24T21:14:56.341 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:56.341 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:14:56.341 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000004593", 2023-05-24T21:14:56.341 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:14:40.343934Z" 2023-05-24T21:14:56.341 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:14:56.341 INFO:teuthology.orchestra.run.smithi071.stdout:] 2023-05-24T21:14:56.342 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000002345', 1: '00000000000000000000:00000000000000001210', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004593'} 2023-05-24T21:14:56.343 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-05-24T21:14:56.343 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:14:56.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:14:56.344 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:14:56.394 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:14:56.395 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:14:56.444 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:14:56.444 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:14:56.445 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:14:56.445 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:14:56.445 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:14:56.445 INFO:teuthology.orchestra.run.smithi071.stdout: "instance_id": 6585121019900467362 2023-05-24T21:14:56.445 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:56.445 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:14:56.445 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:14:56.446 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:14:56.446 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:14:56.446 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:14:56.446 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000002345", 2023-05-24T21:14:56.446 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:14:56.446 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:14:56.447 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:14:56.447 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:10:23.920015Z" 2023-05-24T21:14:56.447 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:14:56.447 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:56.447 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:14:56.447 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:14:56.448 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:14:56.448 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:14:56.448 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000001068", 2023-05-24T21:14:56.448 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:14:56.449 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:14:56.449 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:14:56.449 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:14:40.136506Z" 2023-05-24T21:14:56.449 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:14:56.449 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:56.449 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:14:56.449 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:14:56.450 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:14:56.450 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:14:56.450 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-05-24T21:14:56.450 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:14:56.450 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:14:56.450 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:14:56.450 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:03:11.225401Z" 2023-05-24T21:14:56.451 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:14:56.451 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:56.451 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:14:56.451 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:14:56.451 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:14:56.452 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:14:56.452 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000004593", 2023-05-24T21:14:56.452 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:14:56.452 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:14:56.452 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:14:56.452 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:14:40.343934Z" 2023-05-24T21:14:56.452 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:14:56.453 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:14:56.453 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:14:56.453 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:56.453 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:14:56.454 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 0, 2023-05-24T21:14:56.454 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 0 2023-05-24T21:14:56.454 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:14:56.454 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:14:56Z" 2023-05-24T21:14:56.454 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:14:56.457 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6585121019900467362 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002345", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:10:23.920015Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001068", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:14:40.136506Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:03:11.225401Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004593", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:14:40.343934Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-05-24T21:14:56Z" } 2023-05-24T21:14:56.458 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002345', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:10:23.920015Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001068', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:14:40.136506Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:03:11.225401Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004593', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:14:40.343934Z'}}] 2023-05-24T21:14:56.458 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002345', 1: '00000000000000000000:00000000000000001210', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004593'} 2023-05-24T21:14:56.459 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002345', 1: '00000000000000000000:00000000000000001068', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004593'} 2023-05-24T21:14:56.459 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=1 master=00000000000000000000:00000000000000001210 target=00000000000000000000:00000000000000001068 2023-05-24T21:15:01.459 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:15:01.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:15:01.461 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:15:01.521 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:15:01.522 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:15:01.588 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:15:01.589 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:15:01.589 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:15:01.589 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:15:01.589 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:15:01.589 INFO:teuthology.orchestra.run.smithi071.stdout: "instance_id": 6585121019900467362 2023-05-24T21:15:01.590 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:15:01.590 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:15:01.590 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:15:01.590 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:15:01.590 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:15:01.590 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:15:01.591 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000002345", 2023-05-24T21:15:01.591 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:15:01.591 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:15:01.591 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:15:01.591 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:10:23.920015Z" 2023-05-24T21:15:01.591 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:15:01.591 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:15:01.592 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:15:01.592 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:15:01.592 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:15:01.592 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:15:01.592 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000001068", 2023-05-24T21:15:01.592 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:15:01.593 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:15:01.593 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:15:01.593 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:14:40.136506Z" 2023-05-24T21:15:01.593 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:15:01.593 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:15:01.593 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:15:01.593 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:15:01.594 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:15:01.594 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:15:01.594 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-05-24T21:15:01.594 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:15:01.594 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:15:01.594 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:15:01.594 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:03:11.225401Z" 2023-05-24T21:15:01.595 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:15:01.595 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:15:01.595 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:15:01.595 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:15:01.595 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:15:01.595 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:15:01.595 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000004593", 2023-05-24T21:15:01.596 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:15:01.596 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:15:01.596 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:15:01.596 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:14:40.343934Z" 2023-05-24T21:15:01.596 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:15:01.596 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:15:01.597 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:15:01.597 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:15:01.597 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:15:01.597 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 0, 2023-05-24T21:15:01.597 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 0 2023-05-24T21:15:01.597 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:15:01.597 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:15:01Z" 2023-05-24T21:15:01.598 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:15:01.598 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6585121019900467362 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002345", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:10:23.920015Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001068", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:14:40.136506Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:03:11.225401Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004593", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:14:40.343934Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-05-24T21:15:01Z" } 2023-05-24T21:15:01.599 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002345', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:10:23.920015Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001068', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:14:40.136506Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:03:11.225401Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004593', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:14:40.343934Z'}}] 2023-05-24T21:15:01.599 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002345', 1: '00000000000000000000:00000000000000001210', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004593'} 2023-05-24T21:15:01.600 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002345', 1: '00000000000000000000:00000000000000001068', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004593'} 2023-05-24T21:15:01.600 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=1 master=00000000000000000000:00000000000000001210 target=00000000000000000000:00000000000000001068 2023-05-24T21:15:06.601 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:15:06.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:15:06.602 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:15:06.661 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:15:06.661 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:15:06.725 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:15:06.725 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:15:06.725 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:15:06.725 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:15:06.726 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:15:06.726 INFO:teuthology.orchestra.run.smithi071.stdout: "instance_id": 6585121019900467362 2023-05-24T21:15:06.726 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:15:06.726 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:15:06.726 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:15:06.726 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:15:06.726 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:15:06.727 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:15:06.727 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000002345", 2023-05-24T21:15:06.727 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:15:06.727 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:15:06.727 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:15:06.727 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:10:23.920015Z" 2023-05-24T21:15:06.728 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:15:06.728 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:15:06.728 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:15:06.728 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:15:06.728 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:15:06.728 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:15:06.728 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000001068", 2023-05-24T21:15:06.729 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:15:06.729 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:15:06.729 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:15:06.729 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:14:40.136506Z" 2023-05-24T21:15:06.729 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:15:06.729 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:15:06.729 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:15:06.730 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:15:06.730 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:15:06.730 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:15:06.730 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-05-24T21:15:06.730 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:15:06.730 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:15:06.730 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:15:06.731 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:03:11.225401Z" 2023-05-24T21:15:06.731 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:15:06.731 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:15:06.731 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:15:06.731 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:15:06.731 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:15:06.731 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:15:06.732 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000004593", 2023-05-24T21:15:06.732 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:15:06.732 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:15:06.732 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:15:06.732 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:14:40.343934Z" 2023-05-24T21:15:06.732 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:15:06.733 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:15:06.733 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:15:06.733 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:15:06.733 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:15:06.733 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 0, 2023-05-24T21:15:06.733 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 0 2023-05-24T21:15:06.733 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:15:06.733 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:15:06Z" 2023-05-24T21:15:06.734 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:15:06.735 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6585121019900467362 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002345", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:10:23.920015Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001068", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:14:40.136506Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:03:11.225401Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004593", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:14:40.343934Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-05-24T21:15:06Z" } 2023-05-24T21:15:06.735 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002345', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:10:23.920015Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001068', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:14:40.136506Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:03:11.225401Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004593', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:14:40.343934Z'}}] 2023-05-24T21:15:06.736 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002345', 1: '00000000000000000000:00000000000000001210', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004593'} 2023-05-24T21:15:06.736 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002345', 1: '00000000000000000000:00000000000000001068', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004593'} 2023-05-24T21:15:06.736 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=1 master=00000000000000000000:00000000000000001210 target=00000000000000000000:00000000000000001068 2023-05-24T21:15:11.737 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:15:11.738 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:15:11.739 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:15:11.800 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:15:11.801 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:15:11.864 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:15:11.864 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:15:11.864 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:15:11.865 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:15:11.865 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:15:11.865 INFO:teuthology.orchestra.run.smithi071.stdout: "instance_id": 6585121019900467362 2023-05-24T21:15:11.865 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:15:11.865 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:15:11.865 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:15:11.866 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:15:11.866 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:15:11.866 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:15:11.866 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000002345", 2023-05-24T21:15:11.867 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:15:11.867 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:15:11.867 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:15:11.867 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:10:23.920015Z" 2023-05-24T21:15:11.867 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:15:11.867 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:15:11.867 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:15:11.868 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:15:11.868 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:15:11.868 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:15:11.868 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-05-24T21:15:11.868 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:15:11.869 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:15:11.869 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:15:11.869 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:14:51.309908Z" 2023-05-24T21:15:11.869 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:15:11.869 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:15:11.869 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:15:11.869 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:15:11.870 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:15:11.870 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:15:11.870 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-05-24T21:15:11.870 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:15:11.870 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:15:11.871 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:15:11.871 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:03:11.225401Z" 2023-05-24T21:15:11.871 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:15:11.871 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:15:11.871 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:15:11.871 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:15:11.871 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:15:11.872 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:15:11.872 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000004593", 2023-05-24T21:15:11.872 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:15:11.872 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:15:11.872 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:15:11.872 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:14:40.343934Z" 2023-05-24T21:15:11.873 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:15:11.873 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:15:11.873 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:15:11.873 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:15:11.874 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:15:11.874 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 0, 2023-05-24T21:15:11.874 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 0 2023-05-24T21:15:11.874 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:15:11.874 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:15:11Z" 2023-05-24T21:15:11.874 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:15:11.875 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6585121019900467362 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002345", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:10:23.920015Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001210", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:14:51.309908Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:03:11.225401Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004593", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:14:40.343934Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-05-24T21:15:11Z" } 2023-05-24T21:15:11.876 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002345', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:10:23.920015Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:14:51.309908Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:03:11.225401Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004593', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:14:40.343934Z'}}] 2023-05-24T21:15:11.876 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002345', 1: '00000000000000000000:00000000000000001210', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004593'} 2023-05-24T21:15:11.877 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002345', 1: '00000000000000000000:00000000000000001210', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004593'} 2023-05-24T21:15:11.877 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-05-24T21:15:11.877 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:15:11.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:15:11.878 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:15:11.924 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:15:11.924 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:15:12.008 INFO:teuthology.orchestra.run.smithi071.stdout:[ 2023-05-24T21:15:12.008 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:15:12.009 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000002627", 2023-05-24T21:15:12.009 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:10:45.893198Z" 2023-05-24T21:15:12.009 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:15:12.009 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:15:12.009 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-05-24T21:15:12.009 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:14:51.314836Z" 2023-05-24T21:15:12.010 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:15:12.010 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:15:12.010 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-05-24T21:15:12.010 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:06:31.177575Z" 2023-05-24T21:15:12.010 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:15:12.010 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:15:12.010 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000004593", 2023-05-24T21:15:12.011 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:14:52.334363Z" 2023-05-24T21:15:12.011 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:15:12.011 INFO:teuthology.orchestra.run.smithi071.stdout:] 2023-05-24T21:15:12.012 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:15:12.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:15:12.013 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-24T21:15:12.056 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:15:12.057 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:15:12.125 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:15:12.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:15:12.187 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 0 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:15:12.237 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:15:12.237 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:15:12.315 INFO:teuthology.orchestra.run.smithi071.stdout:[] 2023-05-24T21:15:12.317 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:15:12.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:15:12.318 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 1 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:15:12.368 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:15:12.369 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:15:12.422 INFO:teuthology.orchestra.run.smithi071.stdout:[] 2023-05-24T21:15:12.423 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:15:12.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:15:12.424 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 2 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:15:12.473 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:15:12.473 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:15:12.517 INFO:teuthology.orchestra.run.smithi071.stdout:[] 2023-05-24T21:15:12.518 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:15:12.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:15:12.519 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 3 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:15:12.568 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:15:12.569 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:15:12.621 INFO:teuthology.orchestra.run.smithi071.stdout:[ 2023-05-24T21:15:12.621 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:15:12.622 INFO:teuthology.orchestra.run.smithi071.stdout: "entity_type": "bucket", 2023-05-24T21:15:12.622 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "qskrps-34:299a3349-fae0-459e-8632-431fc36beb1b.4368.34:8", 2023-05-24T21:15:12.622 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:14:52.334048Z", 2023-05-24T21:15:12.622 INFO:teuthology.orchestra.run.smithi071.stdout: "gen": 0 2023-05-24T21:15:12.622 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:15:12.622 INFO:teuthology.orchestra.run.smithi071.stdout:] 2023-05-24T21:15:12.624 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_datalog_autotrim ... FAIL 2023-05-24T21:15:12.625 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:15:12.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:15:12.626 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --sync-from-all=false --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-24T21:15:12.674 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:15:12.674 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:15:12.695 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:15:12.695 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "5af1588c-f9b9-4687-a040-ce6e0cc808d9", 2023-05-24T21:15:12.696 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a2", 2023-05-24T21:15:12.696 INFO:teuthology.orchestra.run.smithi071.stdout: "domain_root": "a2.rgw.meta:root", 2023-05-24T21:15:12.696 INFO:teuthology.orchestra.run.smithi071.stdout: "control_pool": "a2.rgw.control", 2023-05-24T21:15:12.696 INFO:teuthology.orchestra.run.smithi071.stdout: "gc_pool": "a2.rgw.log:gc", 2023-05-24T21:15:12.696 INFO:teuthology.orchestra.run.smithi071.stdout: "lc_pool": "a2.rgw.log:lc", 2023-05-24T21:15:12.696 INFO:teuthology.orchestra.run.smithi071.stdout: "log_pool": "a2.rgw.log", 2023-05-24T21:15:12.697 INFO:teuthology.orchestra.run.smithi071.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-05-24T21:15:12.697 INFO:teuthology.orchestra.run.smithi071.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-05-24T21:15:12.697 INFO:teuthology.orchestra.run.smithi071.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-05-24T21:15:12.697 INFO:teuthology.orchestra.run.smithi071.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-05-24T21:15:12.697 INFO:teuthology.orchestra.run.smithi071.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-05-24T21:15:12.697 INFO:teuthology.orchestra.run.smithi071.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-05-24T21:15:12.697 INFO:teuthology.orchestra.run.smithi071.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-05-24T21:15:12.698 INFO:teuthology.orchestra.run.smithi071.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-05-24T21:15:12.698 INFO:teuthology.orchestra.run.smithi071.stdout: "otp_pool": "a2.rgw.otp", 2023-05-24T21:15:12.698 INFO:teuthology.orchestra.run.smithi071.stdout: "system_key": { 2023-05-24T21:15:12.698 INFO:teuthology.orchestra.run.smithi071.stdout: "access_key": "Z6KKVUYPP0489K03", 2023-05-24T21:15:12.698 INFO:teuthology.orchestra.run.smithi071.stdout: "secret_key": "EX5IDzHTe9AVqa8b1PSD8B5Y1CKI3jBg" 2023-05-24T21:15:12.698 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:15:12.698 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_pools": [ 2023-05-24T21:15:12.699 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:15:12.699 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "default-placement", 2023-05-24T21:15:12.699 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:15:12.699 INFO:teuthology.orchestra.run.smithi071.stdout: "index_pool": "a2.rgw.buckets.index", 2023-05-24T21:15:12.699 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": { 2023-05-24T21:15:12.699 INFO:teuthology.orchestra.run.smithi071.stdout: "STANDARD": { 2023-05-24T21:15:12.699 INFO:teuthology.orchestra.run.smithi071.stdout: "data_pool": "a2.rgw.buckets.data", 2023-05-24T21:15:12.700 INFO:teuthology.orchestra.run.smithi071.stdout: "compression_type": "random" 2023-05-24T21:15:12.700 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:15:12.700 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:15:12.700 INFO:teuthology.orchestra.run.smithi071.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-05-24T21:15:12.700 INFO:teuthology.orchestra.run.smithi071.stdout: "index_type": 0, 2023-05-24T21:15:12.700 INFO:teuthology.orchestra.run.smithi071.stdout: "inline_data": true 2023-05-24T21:15:12.701 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:15:12.701 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:15:12.701 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:15:12.701 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T21:15:12.701 INFO:teuthology.orchestra.run.smithi071.stdout: "notif_pool": "a2.rgw.log:notif" 2023-05-24T21:15:12.701 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:15:12.702 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:15:12.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:15:12.703 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-24T21:15:12.750 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:15:12.751 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:15:12.804 INFO:teuthology.orchestra.run.smithi071.stderr:Sending period to new master zone 299a3349-fae0-459e-8632-431fc36beb1b 2023-05-24T21:15:12.842 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:15:12.843 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:15:12.843 INFO:teuthology.orchestra.run.smithi071.stdout: "epoch": 5, 2023-05-24T21:15:12.843 INFO:teuthology.orchestra.run.smithi071.stdout: "predecessor_uuid": "1cea3a6e-eb52-480e-95b7-3f6e43317e60", 2023-05-24T21:15:12.843 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": [], 2023-05-24T21:15:12.843 INFO:teuthology.orchestra.run.smithi071.stdout: "period_map": { 2023-05-24T21:15:12.843 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:15:12.844 INFO:teuthology.orchestra.run.smithi071.stdout: "zonegroups": [ 2023-05-24T21:15:12.844 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:15:12.844 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "a", 2023-05-24T21:15:12.844 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a", 2023-05-24T21:15:12.844 INFO:teuthology.orchestra.run.smithi071.stdout: "api_name": "", 2023-05-24T21:15:12.844 INFO:teuthology.orchestra.run.smithi071.stdout: "is_master": true, 2023-05-24T21:15:12.844 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:15:12.845 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-05-24T21:15:12.845 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:15:12.845 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames": [], 2023-05-24T21:15:12.845 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames_s3website": [], 2023-05-24T21:15:12.845 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zone": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T21:15:12.845 INFO:teuthology.orchestra.run.smithi071.stdout: "zones": [ 2023-05-24T21:15:12.845 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:15:12.846 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T21:15:12.846 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a1", 2023-05-24T21:15:12.846 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:15:12.846 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-05-24T21:15:12.846 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:15:12.846 INFO:teuthology.orchestra.run.smithi071.stdout: "log_meta": false, 2023-05-24T21:15:12.846 INFO:teuthology.orchestra.run.smithi071.stdout: "log_data": true, 2023-05-24T21:15:12.847 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_index_max_shards": 11, 2023-05-24T21:15:12.847 INFO:teuthology.orchestra.run.smithi071.stdout: "read_only": false, 2023-05-24T21:15:12.847 INFO:teuthology.orchestra.run.smithi071.stdout: "tier_type": "", 2023-05-24T21:15:12.847 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from_all": true, 2023-05-24T21:15:12.847 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from": [], 2023-05-24T21:15:12.847 INFO:teuthology.orchestra.run.smithi071.stdout: "redirect_zone": "", 2023-05-24T21:15:12.847 INFO:teuthology.orchestra.run.smithi071.stdout: "supported_features": [ 2023-05-24T21:15:12.848 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:15:12.848 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:15:12.848 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:15:12.848 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:15:12.848 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "5af1588c-f9b9-4687-a040-ce6e0cc808d9", 2023-05-24T21:15:12.848 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a2", 2023-05-24T21:15:12.848 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:15:12.849 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi071.front.sepia.ceph.com:8001" 2023-05-24T21:15:12.849 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:15:12.849 INFO:teuthology.orchestra.run.smithi071.stdout: "log_meta": false, 2023-05-24T21:15:12.849 INFO:teuthology.orchestra.run.smithi071.stdout: "log_data": true, 2023-05-24T21:15:12.849 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_index_max_shards": 11, 2023-05-24T21:15:12.849 INFO:teuthology.orchestra.run.smithi071.stdout: "read_only": false, 2023-05-24T21:15:12.850 INFO:teuthology.orchestra.run.smithi071.stdout: "tier_type": "", 2023-05-24T21:15:12.850 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from_all": false, 2023-05-24T21:15:12.850 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from": [], 2023-05-24T21:15:12.850 INFO:teuthology.orchestra.run.smithi071.stdout: "redirect_zone": "", 2023-05-24T21:15:12.850 INFO:teuthology.orchestra.run.smithi071.stdout: "supported_features": [ 2023-05-24T21:15:12.850 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:15:12.850 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:15:12.851 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:15:12.851 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:15:12.851 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_targets": [ 2023-05-24T21:15:12.851 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:15:12.851 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "default-placement", 2023-05-24T21:15:12.851 INFO:teuthology.orchestra.run.smithi071.stdout: "tags": [], 2023-05-24T21:15:12.851 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": [ 2023-05-24T21:15:12.851 INFO:teuthology.orchestra.run.smithi071.stdout: "STANDARD" 2023-05-24T21:15:12.852 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:15:12.852 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:15:12.852 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:15:12.852 INFO:teuthology.orchestra.run.smithi071.stdout: "default_placement": "default-placement", 2023-05-24T21:15:12.852 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T21:15:12.852 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_policy": { 2023-05-24T21:15:12.853 INFO:teuthology.orchestra.run.smithi071.stdout: "groups": [] 2023-05-24T21:15:12.853 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:15:12.853 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled_features": [ 2023-05-24T21:15:12.853 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:15:12.853 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:15:12.853 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:15:12.853 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:15:12.854 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "b", 2023-05-24T21:15:12.854 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "b", 2023-05-24T21:15:12.854 INFO:teuthology.orchestra.run.smithi071.stdout: "api_name": "", 2023-05-24T21:15:12.854 INFO:teuthology.orchestra.run.smithi071.stdout: "is_master": false, 2023-05-24T21:15:12.854 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:15:12.854 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi157.front.sepia.ceph.com:8000" 2023-05-24T21:15:12.854 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:15:12.855 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames": [], 2023-05-24T21:15:12.855 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames_s3website": [], 2023-05-24T21:15:12.855 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zone": "ee18cd86-4684-4b3d-8de6-0075d3d925f1", 2023-05-24T21:15:12.855 INFO:teuthology.orchestra.run.smithi071.stdout: "zones": [ 2023-05-24T21:15:12.855 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:15:12.855 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "7b8c3b69-c8ed-4621-b2ef-b89d3bbb7973", 2023-05-24T21:15:12.855 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "b2", 2023-05-24T21:15:12.855 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:15:12.856 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi157.front.sepia.ceph.com:8001" 2023-05-24T21:15:12.856 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:15:12.856 INFO:teuthology.orchestra.run.smithi071.stdout: "log_meta": false, 2023-05-24T21:15:12.856 INFO:teuthology.orchestra.run.smithi071.stdout: "log_data": true, 2023-05-24T21:15:12.856 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_index_max_shards": 11, 2023-05-24T21:15:12.856 INFO:teuthology.orchestra.run.smithi071.stdout: "read_only": false, 2023-05-24T21:15:12.857 INFO:teuthology.orchestra.run.smithi071.stdout: "tier_type": "", 2023-05-24T21:15:12.857 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from_all": true, 2023-05-24T21:15:12.857 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from": [], 2023-05-24T21:15:12.857 INFO:teuthology.orchestra.run.smithi071.stdout: "redirect_zone": "", 2023-05-24T21:15:12.857 INFO:teuthology.orchestra.run.smithi071.stdout: "supported_features": [ 2023-05-24T21:15:12.857 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:15:12.857 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:15:12.858 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:15:12.858 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:15:12.862 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "ee18cd86-4684-4b3d-8de6-0075d3d925f1", 2023-05-24T21:15:12.862 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "b1", 2023-05-24T21:15:12.862 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:15:12.862 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi157.front.sepia.ceph.com:8000" 2023-05-24T21:15:12.863 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:15:12.863 INFO:teuthology.orchestra.run.smithi071.stdout: "log_meta": false, 2023-05-24T21:15:12.863 INFO:teuthology.orchestra.run.smithi071.stdout: "log_data": true, 2023-05-24T21:15:12.863 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_index_max_shards": 11, 2023-05-24T21:15:12.863 INFO:teuthology.orchestra.run.smithi071.stdout: "read_only": false, 2023-05-24T21:15:12.863 INFO:teuthology.orchestra.run.smithi071.stdout: "tier_type": "", 2023-05-24T21:15:12.864 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from_all": true, 2023-05-24T21:15:12.864 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from": [], 2023-05-24T21:15:12.864 INFO:teuthology.orchestra.run.smithi071.stdout: "redirect_zone": "", 2023-05-24T21:15:12.864 INFO:teuthology.orchestra.run.smithi071.stdout: "supported_features": [ 2023-05-24T21:15:12.864 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:15:12.864 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:15:12.864 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:15:12.865 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:15:12.865 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_targets": [ 2023-05-24T21:15:12.865 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:15:12.865 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "default-placement", 2023-05-24T21:15:12.865 INFO:teuthology.orchestra.run.smithi071.stdout: "tags": [], 2023-05-24T21:15:12.865 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": [ 2023-05-24T21:15:12.865 INFO:teuthology.orchestra.run.smithi071.stdout: "STANDARD" 2023-05-24T21:15:12.866 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:15:12.866 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:15:12.866 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:15:12.866 INFO:teuthology.orchestra.run.smithi071.stdout: "default_placement": "default-placement", 2023-05-24T21:15:12.866 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T21:15:12.866 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_policy": { 2023-05-24T21:15:12.866 INFO:teuthology.orchestra.run.smithi071.stdout: "groups": [] 2023-05-24T21:15:12.867 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:15:12.867 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled_features": [ 2023-05-24T21:15:12.867 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:15:12.867 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:15:12.867 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:15:12.867 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:15:12.867 INFO:teuthology.orchestra.run.smithi071.stdout: "short_zone_ids": [ 2023-05-24T21:15:12.868 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:15:12.868 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T21:15:12.868 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 3764661814 2023-05-24T21:15:12.868 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:15:12.868 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:15:12.868 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "5af1588c-f9b9-4687-a040-ce6e0cc808d9", 2023-05-24T21:15:12.869 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 2266179370 2023-05-24T21:15:12.869 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:15:12.869 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:15:12.869 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "7b8c3b69-c8ed-4621-b2ef-b89d3bbb7973", 2023-05-24T21:15:12.869 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 1540450555 2023-05-24T21:15:12.869 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:15:12.869 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:15:12.869 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "ee18cd86-4684-4b3d-8de6-0075d3d925f1", 2023-05-24T21:15:12.870 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 489875403 2023-05-24T21:15:12.870 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:15:12.870 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:15:12.870 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:15:12.870 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zonegroup": "a", 2023-05-24T21:15:12.870 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zone": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T21:15:12.870 INFO:teuthology.orchestra.run.smithi071.stdout: "period_config": { 2023-05-24T21:15:12.871 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_quota": { 2023-05-24T21:15:12.871 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false, 2023-05-24T21:15:12.871 INFO:teuthology.orchestra.run.smithi071.stdout: "check_on_raw": false, 2023-05-24T21:15:12.871 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size": -1, 2023-05-24T21:15:12.871 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size_kb": 0, 2023-05-24T21:15:12.871 INFO:teuthology.orchestra.run.smithi071.stdout: "max_objects": -1 2023-05-24T21:15:12.872 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:15:12.872 INFO:teuthology.orchestra.run.smithi071.stdout: "user_quota": { 2023-05-24T21:15:12.872 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false, 2023-05-24T21:15:12.872 INFO:teuthology.orchestra.run.smithi071.stdout: "check_on_raw": false, 2023-05-24T21:15:12.872 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size": -1, 2023-05-24T21:15:12.872 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size_kb": 0, 2023-05-24T21:15:12.872 INFO:teuthology.orchestra.run.smithi071.stdout: "max_objects": -1 2023-05-24T21:15:12.873 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:15:12.873 INFO:teuthology.orchestra.run.smithi071.stdout: "user_ratelimit": { 2023-05-24T21:15:12.873 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_ops": 0, 2023-05-24T21:15:12.873 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_ops": 0, 2023-05-24T21:15:12.873 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_bytes": 0, 2023-05-24T21:15:12.873 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_bytes": 0, 2023-05-24T21:15:12.873 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false 2023-05-24T21:15:12.874 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:15:12.874 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_ratelimit": { 2023-05-24T21:15:12.874 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_ops": 0, 2023-05-24T21:15:12.874 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_ops": 0, 2023-05-24T21:15:12.874 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_bytes": 0, 2023-05-24T21:15:12.874 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_bytes": 0, 2023-05-24T21:15:12.874 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false 2023-05-24T21:15:12.875 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:15:12.875 INFO:teuthology.orchestra.run.smithi071.stdout: "anonymous_ratelimit": { 2023-05-24T21:15:12.875 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_ops": 0, 2023-05-24T21:15:12.875 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_ops": 0, 2023-05-24T21:15:12.875 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_bytes": 0, 2023-05-24T21:15:12.875 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_bytes": 0, 2023-05-24T21:15:12.875 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false 2023-05-24T21:15:12.876 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:15:12.876 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:15:12.876 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T21:15:12.876 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_name": "test-realm", 2023-05-24T21:15:12.876 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:15:12.876 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:15:12.878 INFO:rgw_multi.tests:Set sync_from_all flag on zone a2 to false 2023-05-24T21:16:12.878 INFO:rgw_multi.tests:create bucket zone=a1 name=qskrps-35 2023-05-24T21:16:12.880 DEBUG:boto:path=/qskrps-35/ 2023-05-24T21:16:12.880 DEBUG:boto:auth_path=/qskrps-35/ 2023-05-24T21:16:12.880 DEBUG:boto:Method: PUT 2023-05-24T21:16:12.881 DEBUG:boto:Path: /qskrps-35/ 2023-05-24T21:16:12.881 DEBUG:boto:Data: 2023-05-24T21:16:12.881 DEBUG:boto:Headers: {} 2023-05-24T21:16:12.882 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:16:12.882 DEBUG:boto:Port: 8000 2023-05-24T21:16:12.882 DEBUG:boto:Params: {} 2023-05-24T21:16:12.883 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T21:16:12.883 DEBUG:boto:Token: None 2023-05-24T21:16:12.884 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:16:12 GMT /qskrps-35/ 2023-05-24T21:16:12.884 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:iwvxK5Uj45KlUJeK/N87ycGNTBY= 2023-05-24T21:16:12.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:16:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:iwvxK5Uj45KlUJeK/N87ycGNTBY=', 'Content-Length': '0'} 2023-05-24T21:16:12.916 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e39a88c75ead1a1e-00646e7e9c-14e1-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:16:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:16:12.917 DEBUG:boto:path=/qskrps-35/testredirect 2023-05-24T21:16:12.917 DEBUG:boto:auth_path=/qskrps-35/testredirect 2023-05-24T21:16:12.918 DEBUG:boto:Method: PUT 2023-05-24T21:16:12.918 DEBUG:boto:Path: /qskrps-35/testredirect 2023-05-24T21:16:12.918 DEBUG:boto:Data: 2023-05-24T21:16:12.919 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-05-24T21:16:12.919 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:16:12.919 DEBUG:boto:Port: 8000 2023-05-24T21:16:12.920 DEBUG:boto:Params: {} 2023-05-24T21:16:12.920 DEBUG:boto:Token: None 2023-05-24T21:16:12.921 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Wed, 24 May 2023 21:16:12 GMT /qskrps-35/testredirect 2023-05-24T21:16:12.921 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:8M5CDg/pLAnMlf+qzjSh/3HcMmc= 2023-05-24T21:16:12.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:16:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:8M5CDg/pLAnMlf+qzjSh/3HcMmc='} 2023-05-24T21:16:12.933 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c88bf03d146606d4-00646e7e9c-14e1-a1'), ('Date', 'Wed, 24 May 2023 21:16:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:16:12.933 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:16:12.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:16:12.934 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:16:12.986 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:16:12.986 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:16:13.043 INFO:teuthology.orchestra.run.smithi071.stderr:No --period given, using current period=5c04936b-70ea-4415-b4eb-3aad3dc612e5 2023-05-24T21:16:13.068 INFO:teuthology.orchestra.run.smithi071.stdout:[ 2023-05-24T21:16:13.069 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:16:13.069 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962972.907627_296.1", 2023-05-24T21:16:13.069 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:16:12.907627Z" 2023-05-24T21:16:13.069 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:13.069 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:16:13.070 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:16:13.070 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:06:32.333844Z" 2023-05-24T21:16:13.070 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:13.070 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:16:13.070 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:16:13.070 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:16:13.071 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:13.071 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:16:13.071 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:16:13.071 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:16:13.072 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:13.072 INFO:teuthology.orchestra.run.smithi071.stdout:] 2023-05-24T21:16:13.073 DEBUG:rgw_multi.tests:master meta markers={0: '1_1684962972.907627_296.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:16:13.074 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-24T21:16:13.074 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:16:13.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:16:13.075 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:16:13.124 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:16:13.124 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:16:13.154 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:16:13.148+0000 7f389b517780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:16:13.155 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:16:13.149+0000 7f389b517780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:16:13.167 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:16:13.167 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:16:13.167 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:16:13.167 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:16:13.168 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:16:13.168 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:16:13.168 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:13.168 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:13.168 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:16:13.168 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:16:13.168 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:16:13.169 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:16:13.169 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:16:13.169 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962880.273629_269.1", 2023-05-24T21:16:13.169 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:16:13.169 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:16:13.169 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:16:13.170 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:14:40.273629Z", 2023-05-24T21:16:13.170 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:13.170 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:13.170 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:13.170 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:16:13.170 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:16:13.170 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:16:13.171 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:16:13.171 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:16:13.171 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:16:13.171 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:16:13.171 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:16:13.171 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:16:13.171 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:13.172 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:13.172 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:13.172 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:16:13.172 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:16:13.172 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:16:13.172 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:16:13.173 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:16:13.173 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:16:13.173 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:16:13.173 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:16:13.173 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:16:13.173 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:13.173 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:13.174 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:13.174 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:16:13.174 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:16:13.174 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:16:13.174 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:16:13.174 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:16:13.174 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:16:13.175 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:16:13.175 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:16:13.175 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:16:13.175 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:13.175 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:13.175 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:13.175 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:16:13.176 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:13.176 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:16:13.176 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:16:13.176 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:16:13.176 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:13.176 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:16:13Z" 2023-05-24T21:16:13.177 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:16:13.177 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962880.273629_269.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:14:40.273629Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:16:13Z" } 2023-05-24T21:16:13.178 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962880.273629_269.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:14:40.273629Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:16:13.178 DEBUG:rgw_multi.tests:log_status={0: '1_1684962972.907627_296.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:16:13.179 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962880.273629_269.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:16:13.179 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684962972.907627_296.1 target=1_1684962880.273629_269.1 2023-05-24T21:16:18.180 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:16:18.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:16:18.181 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:16:18.239 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:16:18.239 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:16:18.286 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:16:18.279+0000 7f422d102780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:16:18.286 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:16:18.280+0000 7f422d102780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:16:18.309 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:16:18.309 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:16:18.309 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:16:18.309 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:16:18.309 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:16:18.310 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:16:18.310 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:18.310 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:18.310 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:16:18.310 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:16:18.310 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:16:18.311 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:16:18.311 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:16:18.311 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962880.273629_269.1", 2023-05-24T21:16:18.311 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:16:18.311 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:16:18.311 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:16:18.311 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:14:40.273629Z", 2023-05-24T21:16:18.312 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:18.312 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:18.312 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:18.312 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:16:18.312 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:16:18.312 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:16:18.312 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:16:18.313 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:16:18.313 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:16:18.313 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:16:18.313 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:16:18.313 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:16:18.313 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:18.313 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:18.314 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:18.314 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:16:18.314 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:16:18.314 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:16:18.314 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:16:18.314 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:16:18.315 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:16:18.315 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:16:18.315 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:16:18.315 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:16:18.315 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:18.315 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:18.315 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:18.316 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:16:18.316 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:16:18.316 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:16:18.316 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:16:18.316 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:16:18.316 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:16:18.316 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:16:18.317 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:16:18.317 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:16:18.317 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:18.317 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:18.317 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:18.317 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:16:18.317 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:18.318 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:16:18.318 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:16:18.318 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:16:18.318 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:18.318 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:16:18Z" 2023-05-24T21:16:18.318 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:16:18.319 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962880.273629_269.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:14:40.273629Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:16:18Z" } 2023-05-24T21:16:18.320 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962880.273629_269.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:14:40.273629Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:16:18.320 DEBUG:rgw_multi.tests:log_status={0: '1_1684962972.907627_296.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:16:18.320 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962880.273629_269.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:16:18.321 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684962972.907627_296.1 target=1_1684962880.273629_269.1 2023-05-24T21:16:23.322 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:16:23.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:16:23.323 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:16:23.380 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:16:23.381 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:16:23.426 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:16:23.419+0000 7f1133992780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:16:23.427 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:16:23.420+0000 7f1133992780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:16:23.440 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:16:23.440 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:16:23.440 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:16:23.440 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:16:23.441 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:16:23.441 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:16:23.441 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:23.441 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:23.441 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:16:23.441 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:16:23.442 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:16:23.442 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:16:23.442 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:16:23.442 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962880.273629_269.1", 2023-05-24T21:16:23.442 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:16:23.442 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:16:23.442 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:16:23.443 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:14:40.273629Z", 2023-05-24T21:16:23.443 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:23.443 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:23.443 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:23.443 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:16:23.443 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:16:23.444 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:16:23.444 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:16:23.444 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:16:23.444 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:16:23.445 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:16:23.445 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:16:23.445 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:16:23.445 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:23.445 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:23.446 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:23.446 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:16:23.446 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:16:23.446 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:16:23.447 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:16:23.447 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:16:23.447 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:16:23.447 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:16:23.447 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:16:23.447 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:16:23.448 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:23.448 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:23.448 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:23.448 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:16:23.448 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:16:23.448 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:16:23.449 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:16:23.449 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:16:23.449 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:16:23.449 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:16:23.449 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:16:23.450 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:16:23.450 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:23.450 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:23.450 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:23.450 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:16:23.450 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:23.450 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:16:23.451 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:16:23.451 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:16:23.451 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:23.451 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:16:23Z" 2023-05-24T21:16:23.451 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:16:23.452 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962880.273629_269.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:14:40.273629Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:16:23Z" } 2023-05-24T21:16:23.453 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962880.273629_269.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:14:40.273629Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:16:23.453 DEBUG:rgw_multi.tests:log_status={0: '1_1684962972.907627_296.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:16:23.453 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962880.273629_269.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:16:23.454 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684962972.907627_296.1 target=1_1684962880.273629_269.1 2023-05-24T21:16:28.455 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:16:28.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:16:28.456 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:16:28.513 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:16:28.513 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:16:28.548 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:16:28.541+0000 7fbff192a780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:16:28.548 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:16:28.542+0000 7fbff192a780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:16:28.566 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:16:28.567 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:16:28.567 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:16:28.567 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:16:28.567 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:16:28.567 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:16:28.568 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:28.568 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:28.568 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:16:28.568 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:16:28.568 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:16:28.568 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:16:28.568 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:16:28.569 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962880.273629_269.1", 2023-05-24T21:16:28.569 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:16:28.569 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:16:28.569 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:16:28.569 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:14:40.273629Z", 2023-05-24T21:16:28.569 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:28.569 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:28.570 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:28.570 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:16:28.570 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:16:28.570 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:16:28.570 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:16:28.570 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:16:28.570 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:16:28.571 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:16:28.571 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:16:28.571 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:16:28.571 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:28.571 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:28.571 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:28.572 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:16:28.572 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:16:28.572 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:16:28.572 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:16:28.572 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:16:28.572 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:16:28.572 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:16:28.573 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:16:28.573 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:16:28.573 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:28.573 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:28.573 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:28.573 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:16:28.573 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:16:28.574 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:16:28.574 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:16:28.574 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:16:28.574 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:16:28.574 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:16:28.574 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:16:28.574 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:16:28.575 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:28.575 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:28.575 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:28.575 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:16:28.575 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:28.575 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:16:28.575 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:16:28.576 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:16:28.576 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:28.576 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:16:28Z" 2023-05-24T21:16:28.576 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:16:28.577 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962880.273629_269.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:14:40.273629Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:16:28Z" } 2023-05-24T21:16:28.577 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962880.273629_269.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:14:40.273629Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:16:28.578 DEBUG:rgw_multi.tests:log_status={0: '1_1684962972.907627_296.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:16:28.578 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962880.273629_269.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:16:28.579 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684962972.907627_296.1 target=1_1684962880.273629_269.1 2023-05-24T21:16:33.579 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:16:33.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:16:33.580 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:16:33.638 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:16:33.638 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:16:33.693 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:16:33.687+0000 7f60fff73780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:16:33.693 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:16:33.688+0000 7f60fff73780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:16:33.715 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:16:33.715 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:16:33.715 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:16:33.715 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:16:33.715 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:16:33.716 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:16:33.716 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:33.716 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:33.716 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:16:33.716 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:16:33.716 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:16:33.716 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:16:33.717 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:16:33.717 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962880.273629_269.1", 2023-05-24T21:16:33.717 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:16:33.717 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:16:33.717 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:16:33.717 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:14:40.273629Z", 2023-05-24T21:16:33.718 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:33.718 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:33.718 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:33.718 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:16:33.718 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:16:33.718 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:16:33.718 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:16:33.719 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:16:33.719 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:16:33.719 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:16:33.719 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:16:33.719 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:16:33.719 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:33.719 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:33.720 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:33.720 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:16:33.720 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:16:33.720 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:16:33.720 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:16:33.720 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:16:33.720 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:16:33.721 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:16:33.721 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:16:33.721 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:16:33.721 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:33.721 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:33.721 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:33.721 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:16:33.722 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:16:33.722 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:16:33.722 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:16:33.722 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:16:33.722 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:16:33.722 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:16:33.722 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:16:33.723 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:16:33.723 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:33.723 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:33.723 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:33.723 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:16:33.723 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:33.723 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:16:33.724 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:16:33.724 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:16:33.724 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:33.724 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:16:33Z" 2023-05-24T21:16:33.724 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:16:33.725 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962880.273629_269.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:14:40.273629Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:16:33Z" } 2023-05-24T21:16:33.726 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962880.273629_269.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:14:40.273629Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:16:33.726 DEBUG:rgw_multi.tests:log_status={0: '1_1684962972.907627_296.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:16:33.726 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962880.273629_269.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:16:33.727 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684962972.907627_296.1 target=1_1684962880.273629_269.1 2023-05-24T21:16:38.728 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:16:38.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:16:38.729 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:16:38.788 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:16:38.788 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:16:38.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:16:38.823+0000 7fec1f3b5780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:16:38.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:16:38.824+0000 7fec1f3b5780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:16:38.842 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:16:38.843 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:16:38.843 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:16:38.843 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:16:38.843 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:16:38.843 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:16:38.844 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:38.844 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:38.844 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:16:38.844 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:16:38.844 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:16:38.844 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:16:38.845 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:16:38.845 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962880.273629_269.1", 2023-05-24T21:16:38.845 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:16:38.845 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:16:38.845 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:16:38.845 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:14:40.273629Z", 2023-05-24T21:16:38.845 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:38.846 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:38.846 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:38.846 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:16:38.846 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:16:38.846 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:16:38.846 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:16:38.846 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:16:38.847 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:16:38.847 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:16:38.847 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:16:38.847 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:16:38.847 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:38.847 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:38.847 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:38.848 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:16:38.848 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:16:38.848 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:16:38.848 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:16:38.848 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:16:38.848 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:16:38.849 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:16:38.849 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:16:38.849 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:16:38.849 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:38.849 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:38.849 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:38.849 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:16:38.850 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:16:38.850 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:16:38.850 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:16:38.850 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:16:38.850 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:16:38.850 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:16:38.850 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:16:38.851 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:16:38.851 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:38.851 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:38.851 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:38.851 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:16:38.851 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:38.851 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:16:38.852 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:16:38.852 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:16:38.852 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:38.852 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:16:38Z" 2023-05-24T21:16:38.852 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:16:38.853 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962880.273629_269.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:14:40.273629Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:16:38Z" } 2023-05-24T21:16:38.854 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962880.273629_269.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:14:40.273629Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:16:38.854 DEBUG:rgw_multi.tests:log_status={0: '1_1684962972.907627_296.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:16:38.854 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962880.273629_269.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:16:38.855 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684962972.907627_296.1 target=1_1684962880.273629_269.1 2023-05-24T21:16:43.856 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:16:43.857 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:16:43.858 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:16:43.915 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:16:43.916 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:16:43.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:16:43.964+0000 7f8368045780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:16:43.970 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:16:43.965+0000 7f8368045780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:16:43.991 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:16:43.992 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:16:43.992 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:16:43.992 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:16:43.992 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:16:43.992 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:16:43.993 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:43.993 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:43.993 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:16:43.993 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:16:43.993 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:16:43.993 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:16:43.994 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:16:43.994 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962880.273629_269.1", 2023-05-24T21:16:43.994 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:16:43.994 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:16:43.994 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:16:43.994 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:14:40.273629Z", 2023-05-24T21:16:43.995 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:43.995 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:43.995 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:43.995 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:16:43.995 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:16:43.995 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:16:43.995 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:16:43.996 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:16:43.996 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:16:43.996 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:16:43.996 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:16:43.996 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:16:43.997 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:43.997 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:43.997 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:43.997 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:16:43.997 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:16:43.997 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:16:43.997 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:16:43.998 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:16:43.998 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:16:43.998 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:16:43.998 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:16:43.999 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:16:43.999 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:43.999 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:43.999 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:43.999 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:16:43.999 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:16:44.000 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:16:44.000 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:16:44.000 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:16:44.000 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:16:44.000 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:16:44.000 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:16:44.000 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:16:44.001 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:44.001 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:44.001 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:44.001 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:16:44.001 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:44.001 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:16:44.001 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:16:44.002 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:16:44.002 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:44.002 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:16:43Z" 2023-05-24T21:16:44.002 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:16:44.003 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962880.273629_269.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:14:40.273629Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:16:43Z" } 2023-05-24T21:16:44.003 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962880.273629_269.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:14:40.273629Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:16:44.004 DEBUG:rgw_multi.tests:log_status={0: '1_1684962972.907627_296.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:16:44.004 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962880.273629_269.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:16:44.005 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684962972.907627_296.1 target=1_1684962880.273629_269.1 2023-05-24T21:16:49.005 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:16:49.006 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:16:49.007 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:16:49.064 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:16:49.064 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:16:49.114 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:16:49.109+0000 7fd35b99c780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:16:49.115 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:16:49.110+0000 7fd35b99c780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:16:49.128 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:16:49.129 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:16:49.129 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:16:49.129 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:16:49.129 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:16:49.129 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:16:49.130 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:49.130 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:49.130 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:16:49.130 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:16:49.130 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:16:49.130 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:16:49.130 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:16:49.131 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962880.273629_269.1", 2023-05-24T21:16:49.131 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:16:49.131 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:16:49.131 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:16:49.131 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:14:40.273629Z", 2023-05-24T21:16:49.131 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:49.131 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:49.132 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:49.132 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:16:49.132 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:16:49.132 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:16:49.132 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:16:49.132 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:16:49.133 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:16:49.133 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:16:49.133 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:16:49.133 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:16:49.133 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:49.133 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:49.133 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:49.134 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:16:49.134 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:16:49.134 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:16:49.134 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:16:49.134 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:16:49.134 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:16:49.134 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:16:49.135 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:16:49.135 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:16:49.135 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:49.135 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:49.135 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:49.135 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:16:49.135 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:16:49.135 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:16:49.136 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:16:49.136 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:16:49.136 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:16:49.136 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:16:49.136 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:16:49.136 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:16:49.137 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:49.137 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:49.137 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:49.137 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:16:49.137 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:49.137 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:16:49.137 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:16:49.137 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:16:49.138 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:49.138 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:16:49Z" 2023-05-24T21:16:49.138 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:16:49.139 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962880.273629_269.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:14:40.273629Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:16:49Z" } 2023-05-24T21:16:49.139 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962880.273629_269.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:14:40.273629Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:16:49.140 DEBUG:rgw_multi.tests:log_status={0: '1_1684962972.907627_296.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:16:49.140 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962880.273629_269.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:16:49.140 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684962972.907627_296.1 target=1_1684962880.273629_269.1 2023-05-24T21:16:54.142 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:16:54.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:16:54.143 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:16:54.199 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:16:54.200 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:16:54.238 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:16:54.232+0000 7f5539785780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:16:54.239 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:16:54.233+0000 7f5539785780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:16:54.252 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:16:54.253 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:16:54.253 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:16:54.253 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:16:54.253 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:16:54.253 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:16:54.253 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:54.254 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:54.254 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:16:54.254 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:16:54.254 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:16:54.254 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:16:54.254 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:16:54.254 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962880.273629_269.1", 2023-05-24T21:16:54.255 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:16:54.255 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:16:54.255 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:16:54.255 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:14:40.273629Z", 2023-05-24T21:16:54.255 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:54.255 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:54.255 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:54.256 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:16:54.256 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:16:54.256 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:16:54.256 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:16:54.256 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:16:54.256 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:16:54.256 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:16:54.257 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:16:54.257 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:16:54.257 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:54.257 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:54.257 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:54.257 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:16:54.257 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:16:54.258 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:16:54.258 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:16:54.258 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:16:54.258 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:16:54.258 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:16:54.258 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:16:54.259 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:16:54.259 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:54.259 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:54.259 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:54.259 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:16:54.259 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:16:54.259 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:16:54.260 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:16:54.260 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:16:54.260 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:16:54.260 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:16:54.260 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:16:54.260 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:16:54.260 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:54.261 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:54.261 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:54.261 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:16:54.261 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:54.261 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:16:54.261 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:16:54.261 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:16:54.262 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:54.262 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:16:54Z" 2023-05-24T21:16:54.262 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:16:54.263 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962880.273629_269.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:14:40.273629Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:16:54Z" } 2023-05-24T21:16:54.263 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962880.273629_269.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:14:40.273629Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:16:54.264 DEBUG:rgw_multi.tests:log_status={0: '1_1684962972.907627_296.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:16:54.264 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962880.273629_269.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:16:54.264 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684962972.907627_296.1 target=1_1684962880.273629_269.1 2023-05-24T21:16:59.265 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:16:59.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:16:59.267 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:16:59.323 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:16:59.324 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:16:59.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:16:59.369+0000 7f603cd16780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:16:59.376 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:16:59.370+0000 7f603cd16780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:16:59.397 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:16:59.397 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:16:59.398 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:16:59.398 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:16:59.398 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:16:59.399 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:16:59.399 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:59.399 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:59.399 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:16:59.400 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:16:59.400 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:16:59.400 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:16:59.400 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:16:59.400 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962880.273629_269.1", 2023-05-24T21:16:59.400 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:16:59.401 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:16:59.401 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:16:59.401 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:14:40.273629Z", 2023-05-24T21:16:59.401 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:59.401 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:59.401 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:59.401 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:16:59.402 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:16:59.402 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:16:59.402 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:16:59.402 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:16:59.402 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:16:59.402 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:16:59.402 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:16:59.403 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:16:59.403 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:59.403 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:59.403 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:59.403 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:16:59.403 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:16:59.403 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:16:59.404 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:16:59.404 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:16:59.404 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:16:59.404 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:16:59.404 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:16:59.404 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:16:59.404 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:59.405 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:59.405 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:59.405 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:16:59.405 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:16:59.405 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:16:59.405 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:16:59.405 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:16:59.406 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:16:59.406 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:16:59.406 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:16:59.406 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:16:59.406 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:16:59.406 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:59.406 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:16:59.407 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:16:59.407 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:59.407 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:16:59.407 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:16:59.407 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:16:59.407 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:16:59.407 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:16:59Z" 2023-05-24T21:16:59.408 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:16:59.409 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962880.273629_269.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:14:40.273629Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:16:59Z" } 2023-05-24T21:16:59.409 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962880.273629_269.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:14:40.273629Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:16:59.410 DEBUG:rgw_multi.tests:log_status={0: '1_1684962972.907627_296.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:16:59.410 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962880.273629_269.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:16:59.411 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684962972.907627_296.1 target=1_1684962880.273629_269.1 2023-05-24T21:17:04.411 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:17:04.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:17:04.413 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:17:04.469 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:17:04.470 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:17:04.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:17:04.495+0000 7f20357d9780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:17:04.503 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:17:04.496+0000 7f20357d9780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:17:04.516 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:17:04.516 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:17:04.517 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:17:04.517 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:17:04.517 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:17:04.517 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:17:04.517 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:17:04.518 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:17:04.518 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:17:04.518 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:17:04.518 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:17:04.518 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:17:04.518 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:17:04.518 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962880.273629_269.1", 2023-05-24T21:17:04.519 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:17:04.519 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:17:04.519 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:17:04.519 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:14:40.273629Z", 2023-05-24T21:17:04.519 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:17:04.519 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:17:04.520 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:17:04.520 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:17:04.520 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:17:04.520 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:17:04.520 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:17:04.520 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:17:04.520 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:17:04.521 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:17:04.521 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:17:04.521 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:17:04.521 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:17:04.521 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:17:04.522 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:17:04.522 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:17:04.522 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:17:04.522 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:17:04.522 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:17:04.522 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:17:04.522 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:17:04.523 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:17:04.523 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:17:04.523 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:17:04.524 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:17:04.524 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:17:04.524 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:17:04.524 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:17:04.524 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:17:04.524 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:17:04.525 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:17:04.525 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:17:04.525 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:17:04.525 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:17:04.525 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:17:04.525 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:17:04.526 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:17:04.526 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:17:04.526 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:17:04.526 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:17:04.526 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:17:04.527 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:17:04.527 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:17:04.527 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:17:04.527 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:17:04.528 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:17:04Z" 2023-05-24T21:17:04.528 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:17:04.529 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962880.273629_269.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:14:40.273629Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:17:04Z" } 2023-05-24T21:17:04.529 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962880.273629_269.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:14:40.273629Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:17:04.530 DEBUG:rgw_multi.tests:log_status={0: '1_1684962972.907627_296.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:17:04.530 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962880.273629_269.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:17:04.530 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684962972.907627_296.1 target=1_1684962880.273629_269.1 2023-05-24T21:17:09.532 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:17:09.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:17:09.533 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:17:09.589 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:17:09.590 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:17:09.643 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:17:09.636+0000 7f349f08d780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:17:09.643 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:17:09.637+0000 7f349f08d780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:17:09.665 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:17:09.665 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:17:09.665 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:17:09.665 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:17:09.666 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:17:09.666 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:17:09.666 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:17:09.666 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:17:09.666 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:17:09.666 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:17:09.667 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:17:09.667 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:17:09.667 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:17:09.667 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962880.273629_269.1", 2023-05-24T21:17:09.667 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:17:09.667 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:17:09.667 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:17:09.668 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:14:40.273629Z", 2023-05-24T21:17:09.668 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:17:09.668 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:17:09.668 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:17:09.668 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:17:09.668 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:17:09.669 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:17:09.669 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:17:09.669 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:17:09.669 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:17:09.669 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:17:09.669 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:17:09.669 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:17:09.670 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:17:09.670 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:17:09.670 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:17:09.670 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:17:09.670 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:17:09.670 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:17:09.670 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:17:09.671 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:17:09.671 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:17:09.671 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:17:09.671 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:17:09.671 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:17:09.671 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:17:09.671 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:17:09.672 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:17:09.672 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:17:09.672 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:17:09.672 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:17:09.672 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:17:09.672 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:17:09.672 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:17:09.673 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:17:09.673 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:17:09.673 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:17:09.674 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:17:09.674 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:17:09.674 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:17:09.674 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:17:09.674 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:17:09.674 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:17:09.675 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:17:09.675 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:17:09.675 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:17:09.675 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:17:09Z" 2023-05-24T21:17:09.675 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:17:09.676 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962880.273629_269.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:14:40.273629Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:17:09Z" } 2023-05-24T21:17:09.677 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962880.273629_269.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:14:40.273629Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:17:09.677 DEBUG:rgw_multi.tests:log_status={0: '1_1684962972.907627_296.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:17:09.677 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962880.273629_269.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:17:09.678 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684962972.907627_296.1 target=1_1684962880.273629_269.1 2023-05-24T21:17:14.679 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:17:14.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:17:14.680 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:17:14.738 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:17:14.739 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:17:14.773 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:17:14.766+0000 7f2112ec8780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:17:14.774 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:17:14.767+0000 7f2112ec8780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:17:14.787 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:17:14.787 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:17:14.787 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:17:14.787 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:17:14.788 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:17:14.788 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:17:14.788 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:17:14.788 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:17:14.788 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:17:14.788 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:17:14.789 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:17:14.789 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:17:14.789 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:17:14.789 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962972.907627_296.1", 2023-05-24T21:17:14.789 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:17:14.789 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:17:14.789 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:17:14.790 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:16:12.907627Z", 2023-05-24T21:17:14.790 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:17:14.790 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:17:14.790 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:17:14.790 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:17:14.790 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:17:14.790 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:17:14.791 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:17:14.791 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:17:14.791 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:17:14.791 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:17:14.791 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:17:14.791 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:17:14.791 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:17:14.792 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:17:14.792 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:17:14.792 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:17:14.792 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:17:14.792 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:17:14.792 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:17:14.792 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:17:14.793 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:17:14.793 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:17:14.793 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:17:14.793 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:17:14.793 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:17:14.793 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:17:14.794 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:17:14.794 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:17:14.794 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:17:14.794 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:17:14.794 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:17:14.794 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:17:14.794 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:17:14.795 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:17:14.795 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:17:14.795 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:17:14.795 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:17:14.795 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:17:14.795 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:17:14.795 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:17:14.796 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:17:14.796 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:17:14.796 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:17:14.796 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:17:14.796 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:17:14.796 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:17:14Z" 2023-05-24T21:17:14.796 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:17:14.797 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962972.907627_296.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:16:12.907627Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:17:14Z" } 2023-05-24T21:17:14.798 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962972.907627_296.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:16:12.907627Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:17:14.798 DEBUG:rgw_multi.tests:log_status={0: '1_1684962972.907627_296.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:17:14.799 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962972.907627_296.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:17:14.799 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-24T21:17:14.799 DEBUG:boto:path=/qskrps-35/ 2023-05-24T21:17:14.800 DEBUG:boto:auth_path=/qskrps-35/ 2023-05-24T21:17:14.800 DEBUG:boto:Method: HEAD 2023-05-24T21:17:14.800 DEBUG:boto:Path: /qskrps-35/ 2023-05-24T21:17:14.801 DEBUG:boto:Data: 2023-05-24T21:17:14.801 DEBUG:boto:Headers: {} 2023-05-24T21:17:14.801 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:17:14.802 DEBUG:boto:Port: 8001 2023-05-24T21:17:14.802 DEBUG:boto:Params: {} 2023-05-24T21:17:14.803 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-24T21:17:14.803 DEBUG:boto:Token: None 2023-05-24T21:17:14.803 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:17:14 GMT /qskrps-35/ 2023-05-24T21:17:14.804 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:OgO3I7A9CHmrUSxwxIyYBX93ap8= 2023-05-24T21:17:14.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:17:14 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:OgO3I7A9CHmrUSxwxIyYBX93ap8=', 'Content-Length': '0'} 2023-05-24T21:17:14.809 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004d8e4b453ba76717-00646e7eda-14dd-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:17:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:17:14.809 DEBUG:boto:path=/qskrps-35/testredirect 2023-05-24T21:17:14.810 DEBUG:boto:auth_path=/qskrps-35/testredirect 2023-05-24T21:17:14.810 DEBUG:boto:Method: HEAD 2023-05-24T21:17:14.811 DEBUG:boto:Path: /qskrps-35/testredirect 2023-05-24T21:17:14.811 DEBUG:boto:Data: 2023-05-24T21:17:14.811 DEBUG:boto:Headers: {} 2023-05-24T21:17:14.812 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:17:14.812 DEBUG:boto:Port: 8001 2023-05-24T21:17:14.812 DEBUG:boto:Params: {} 2023-05-24T21:17:14.813 DEBUG:boto:Token: None 2023-05-24T21:17:14.813 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:17:14 GMT /qskrps-35/testredirect 2023-05-24T21:17:14.814 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:pGcAbv993Cm/g/CO+Ss1hukmhFU= 2023-05-24T21:17:14.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:17:14 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:pGcAbv993Cm/g/CO+Ss1hukmhFU=', 'Content-Length': '0'} 2023-05-24T21:17:14.815 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000e6ae5e2d689c4068-00646e7eda-14dd-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:17:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:17:14.816 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=299a3349-fae0-459e-8632-431fc36beb1b', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:17:14.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=299a3349-fae0-459e-8632-431fc36beb1b', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:17:14.817 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --redirect-zone=299a3349-fae0-459e-8632-431fc36beb1b --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-24T21:17:14.865 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:17:14.865 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:17:14.885 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:17:14.885 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "5af1588c-f9b9-4687-a040-ce6e0cc808d9", 2023-05-24T21:17:14.885 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a2", 2023-05-24T21:17:14.885 INFO:teuthology.orchestra.run.smithi071.stdout: "domain_root": "a2.rgw.meta:root", 2023-05-24T21:17:14.885 INFO:teuthology.orchestra.run.smithi071.stdout: "control_pool": "a2.rgw.control", 2023-05-24T21:17:14.886 INFO:teuthology.orchestra.run.smithi071.stdout: "gc_pool": "a2.rgw.log:gc", 2023-05-24T21:17:14.886 INFO:teuthology.orchestra.run.smithi071.stdout: "lc_pool": "a2.rgw.log:lc", 2023-05-24T21:17:14.886 INFO:teuthology.orchestra.run.smithi071.stdout: "log_pool": "a2.rgw.log", 2023-05-24T21:17:14.886 INFO:teuthology.orchestra.run.smithi071.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-05-24T21:17:14.886 INFO:teuthology.orchestra.run.smithi071.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-05-24T21:17:14.886 INFO:teuthology.orchestra.run.smithi071.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-05-24T21:17:14.886 INFO:teuthology.orchestra.run.smithi071.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-05-24T21:17:14.887 INFO:teuthology.orchestra.run.smithi071.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-05-24T21:17:14.887 INFO:teuthology.orchestra.run.smithi071.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-05-24T21:17:14.887 INFO:teuthology.orchestra.run.smithi071.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-05-24T21:17:14.887 INFO:teuthology.orchestra.run.smithi071.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-05-24T21:17:14.887 INFO:teuthology.orchestra.run.smithi071.stdout: "otp_pool": "a2.rgw.otp", 2023-05-24T21:17:14.887 INFO:teuthology.orchestra.run.smithi071.stdout: "system_key": { 2023-05-24T21:17:14.887 INFO:teuthology.orchestra.run.smithi071.stdout: "access_key": "Z6KKVUYPP0489K03", 2023-05-24T21:17:14.888 INFO:teuthology.orchestra.run.smithi071.stdout: "secret_key": "EX5IDzHTe9AVqa8b1PSD8B5Y1CKI3jBg" 2023-05-24T21:17:14.888 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:17:14.888 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_pools": [ 2023-05-24T21:17:14.888 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:17:14.888 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "default-placement", 2023-05-24T21:17:14.888 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:17:14.889 INFO:teuthology.orchestra.run.smithi071.stdout: "index_pool": "a2.rgw.buckets.index", 2023-05-24T21:17:14.889 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": { 2023-05-24T21:17:14.889 INFO:teuthology.orchestra.run.smithi071.stdout: "STANDARD": { 2023-05-24T21:17:14.889 INFO:teuthology.orchestra.run.smithi071.stdout: "data_pool": "a2.rgw.buckets.data", 2023-05-24T21:17:14.889 INFO:teuthology.orchestra.run.smithi071.stdout: "compression_type": "random" 2023-05-24T21:17:14.889 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:17:14.889 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:17:14.890 INFO:teuthology.orchestra.run.smithi071.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-05-24T21:17:14.890 INFO:teuthology.orchestra.run.smithi071.stdout: "index_type": 0, 2023-05-24T21:17:14.890 INFO:teuthology.orchestra.run.smithi071.stdout: "inline_data": true 2023-05-24T21:17:14.890 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:17:14.890 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:17:14.890 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:17:14.890 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T21:17:14.891 INFO:teuthology.orchestra.run.smithi071.stdout: "notif_pool": "a2.rgw.log:notif" 2023-05-24T21:17:14.891 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:17:14.892 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:17:14.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:17:14.892 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-24T21:17:14.940 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:17:14.940 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:17:14.983 INFO:teuthology.orchestra.run.smithi071.stderr:Sending period to new master zone 299a3349-fae0-459e-8632-431fc36beb1b 2023-05-24T21:17:15.018 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:17:15.019 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:17:15.019 INFO:teuthology.orchestra.run.smithi071.stdout: "epoch": 6, 2023-05-24T21:17:15.019 INFO:teuthology.orchestra.run.smithi071.stdout: "predecessor_uuid": "1cea3a6e-eb52-480e-95b7-3f6e43317e60", 2023-05-24T21:17:15.020 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": [], 2023-05-24T21:17:15.020 INFO:teuthology.orchestra.run.smithi071.stdout: "period_map": { 2023-05-24T21:17:15.020 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:17:15.020 INFO:teuthology.orchestra.run.smithi071.stdout: "zonegroups": [ 2023-05-24T21:17:15.020 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:17:15.021 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "a", 2023-05-24T21:17:15.021 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a", 2023-05-24T21:17:15.021 INFO:teuthology.orchestra.run.smithi071.stdout: "api_name": "", 2023-05-24T21:17:15.021 INFO:teuthology.orchestra.run.smithi071.stdout: "is_master": true, 2023-05-24T21:17:15.021 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:17:15.021 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-05-24T21:17:15.021 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:17:15.022 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames": [], 2023-05-24T21:17:15.022 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames_s3website": [], 2023-05-24T21:17:15.022 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zone": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T21:17:15.022 INFO:teuthology.orchestra.run.smithi071.stdout: "zones": [ 2023-05-24T21:17:15.022 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:17:15.022 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T21:17:15.022 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a1", 2023-05-24T21:17:15.023 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:17:15.023 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-05-24T21:17:15.023 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:17:15.023 INFO:teuthology.orchestra.run.smithi071.stdout: "log_meta": false, 2023-05-24T21:17:15.023 INFO:teuthology.orchestra.run.smithi071.stdout: "log_data": true, 2023-05-24T21:17:15.023 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_index_max_shards": 11, 2023-05-24T21:17:15.023 INFO:teuthology.orchestra.run.smithi071.stdout: "read_only": false, 2023-05-24T21:17:15.024 INFO:teuthology.orchestra.run.smithi071.stdout: "tier_type": "", 2023-05-24T21:17:15.024 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from_all": true, 2023-05-24T21:17:15.024 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from": [], 2023-05-24T21:17:15.024 INFO:teuthology.orchestra.run.smithi071.stdout: "redirect_zone": "", 2023-05-24T21:17:15.024 INFO:teuthology.orchestra.run.smithi071.stdout: "supported_features": [ 2023-05-24T21:17:15.024 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:17:15.025 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:17:15.025 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:17:15.025 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:17:15.025 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "5af1588c-f9b9-4687-a040-ce6e0cc808d9", 2023-05-24T21:17:15.025 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a2", 2023-05-24T21:17:15.025 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:17:15.025 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi071.front.sepia.ceph.com:8001" 2023-05-24T21:17:15.026 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:17:15.026 INFO:teuthology.orchestra.run.smithi071.stdout: "log_meta": false, 2023-05-24T21:17:15.026 INFO:teuthology.orchestra.run.smithi071.stdout: "log_data": true, 2023-05-24T21:17:15.026 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_index_max_shards": 11, 2023-05-24T21:17:15.026 INFO:teuthology.orchestra.run.smithi071.stdout: "read_only": false, 2023-05-24T21:17:15.026 INFO:teuthology.orchestra.run.smithi071.stdout: "tier_type": "", 2023-05-24T21:17:15.026 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from_all": false, 2023-05-24T21:17:15.027 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from": [], 2023-05-24T21:17:15.027 INFO:teuthology.orchestra.run.smithi071.stdout: "redirect_zone": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T21:17:15.027 INFO:teuthology.orchestra.run.smithi071.stdout: "supported_features": [ 2023-05-24T21:17:15.027 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:17:15.027 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:17:15.027 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:17:15.027 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:17:15.028 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_targets": [ 2023-05-24T21:17:15.028 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:17:15.028 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "default-placement", 2023-05-24T21:17:15.028 INFO:teuthology.orchestra.run.smithi071.stdout: "tags": [], 2023-05-24T21:17:15.028 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": [ 2023-05-24T21:17:15.028 INFO:teuthology.orchestra.run.smithi071.stdout: "STANDARD" 2023-05-24T21:17:15.028 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:17:15.029 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:17:15.029 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:17:15.029 INFO:teuthology.orchestra.run.smithi071.stdout: "default_placement": "default-placement", 2023-05-24T21:17:15.029 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T21:17:15.029 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_policy": { 2023-05-24T21:17:15.029 INFO:teuthology.orchestra.run.smithi071.stdout: "groups": [] 2023-05-24T21:17:15.029 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:17:15.030 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled_features": [ 2023-05-24T21:17:15.030 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:17:15.030 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:17:15.030 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:17:15.030 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:17:15.030 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "b", 2023-05-24T21:17:15.031 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "b", 2023-05-24T21:17:15.031 INFO:teuthology.orchestra.run.smithi071.stdout: "api_name": "", 2023-05-24T21:17:15.031 INFO:teuthology.orchestra.run.smithi071.stdout: "is_master": false, 2023-05-24T21:17:15.031 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:17:15.031 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi157.front.sepia.ceph.com:8000" 2023-05-24T21:17:15.031 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:17:15.031 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames": [], 2023-05-24T21:17:15.032 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames_s3website": [], 2023-05-24T21:17:15.032 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zone": "ee18cd86-4684-4b3d-8de6-0075d3d925f1", 2023-05-24T21:17:15.032 INFO:teuthology.orchestra.run.smithi071.stdout: "zones": [ 2023-05-24T21:17:15.032 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:17:15.032 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "7b8c3b69-c8ed-4621-b2ef-b89d3bbb7973", 2023-05-24T21:17:15.032 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "b2", 2023-05-24T21:17:15.032 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:17:15.033 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi157.front.sepia.ceph.com:8001" 2023-05-24T21:17:15.033 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:17:15.033 INFO:teuthology.orchestra.run.smithi071.stdout: "log_meta": false, 2023-05-24T21:17:15.033 INFO:teuthology.orchestra.run.smithi071.stdout: "log_data": true, 2023-05-24T21:17:15.033 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_index_max_shards": 11, 2023-05-24T21:17:15.033 INFO:teuthology.orchestra.run.smithi071.stdout: "read_only": false, 2023-05-24T21:17:15.033 INFO:teuthology.orchestra.run.smithi071.stdout: "tier_type": "", 2023-05-24T21:17:15.034 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from_all": true, 2023-05-24T21:17:15.034 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from": [], 2023-05-24T21:17:15.034 INFO:teuthology.orchestra.run.smithi071.stdout: "redirect_zone": "", 2023-05-24T21:17:15.034 INFO:teuthology.orchestra.run.smithi071.stdout: "supported_features": [ 2023-05-24T21:17:15.034 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:17:15.034 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:17:15.034 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:17:15.035 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:17:15.039 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "ee18cd86-4684-4b3d-8de6-0075d3d925f1", 2023-05-24T21:17:15.040 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "b1", 2023-05-24T21:17:15.040 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:17:15.040 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi157.front.sepia.ceph.com:8000" 2023-05-24T21:17:15.040 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:17:15.041 INFO:teuthology.orchestra.run.smithi071.stdout: "log_meta": false, 2023-05-24T21:17:15.041 INFO:teuthology.orchestra.run.smithi071.stdout: "log_data": true, 2023-05-24T21:17:15.041 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_index_max_shards": 11, 2023-05-24T21:17:15.042 INFO:teuthology.orchestra.run.smithi071.stdout: "read_only": false, 2023-05-24T21:17:15.042 INFO:teuthology.orchestra.run.smithi071.stdout: "tier_type": "", 2023-05-24T21:17:15.043 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from_all": true, 2023-05-24T21:17:15.043 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from": [], 2023-05-24T21:17:15.043 INFO:teuthology.orchestra.run.smithi071.stdout: "redirect_zone": "", 2023-05-24T21:17:15.043 INFO:teuthology.orchestra.run.smithi071.stdout: "supported_features": [ 2023-05-24T21:17:15.044 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:17:15.044 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:17:15.044 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:17:15.045 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:17:15.045 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_targets": [ 2023-05-24T21:17:15.045 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:17:15.046 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "default-placement", 2023-05-24T21:17:15.046 INFO:teuthology.orchestra.run.smithi071.stdout: "tags": [], 2023-05-24T21:17:15.046 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": [ 2023-05-24T21:17:15.046 INFO:teuthology.orchestra.run.smithi071.stdout: "STANDARD" 2023-05-24T21:17:15.046 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:17:15.047 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:17:15.047 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:17:15.047 INFO:teuthology.orchestra.run.smithi071.stdout: "default_placement": "default-placement", 2023-05-24T21:17:15.048 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T21:17:15.048 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_policy": { 2023-05-24T21:17:15.048 INFO:teuthology.orchestra.run.smithi071.stdout: "groups": [] 2023-05-24T21:17:15.048 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:17:15.049 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled_features": [ 2023-05-24T21:17:15.049 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:17:15.049 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:17:15.049 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:17:15.049 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:17:15.049 INFO:teuthology.orchestra.run.smithi071.stdout: "short_zone_ids": [ 2023-05-24T21:17:15.049 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:17:15.050 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T21:17:15.050 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 3764661814 2023-05-24T21:17:15.050 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:17:15.050 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:17:15.050 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "5af1588c-f9b9-4687-a040-ce6e0cc808d9", 2023-05-24T21:17:15.050 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 2266179370 2023-05-24T21:17:15.051 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:17:15.051 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:17:15.051 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "7b8c3b69-c8ed-4621-b2ef-b89d3bbb7973", 2023-05-24T21:17:15.051 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 1540450555 2023-05-24T21:17:15.051 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:17:15.052 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:17:15.052 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "ee18cd86-4684-4b3d-8de6-0075d3d925f1", 2023-05-24T21:17:15.052 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 489875403 2023-05-24T21:17:15.052 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:17:15.052 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:17:15.052 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:17:15.053 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zonegroup": "a", 2023-05-24T21:17:15.053 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zone": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T21:17:15.053 INFO:teuthology.orchestra.run.smithi071.stdout: "period_config": { 2023-05-24T21:17:15.053 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_quota": { 2023-05-24T21:17:15.054 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false, 2023-05-24T21:17:15.054 INFO:teuthology.orchestra.run.smithi071.stdout: "check_on_raw": false, 2023-05-24T21:17:15.054 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size": -1, 2023-05-24T21:17:15.054 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size_kb": 0, 2023-05-24T21:17:15.054 INFO:teuthology.orchestra.run.smithi071.stdout: "max_objects": -1 2023-05-24T21:17:15.054 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:17:15.055 INFO:teuthology.orchestra.run.smithi071.stdout: "user_quota": { 2023-05-24T21:17:15.055 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false, 2023-05-24T21:17:15.055 INFO:teuthology.orchestra.run.smithi071.stdout: "check_on_raw": false, 2023-05-24T21:17:15.055 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size": -1, 2023-05-24T21:17:15.055 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size_kb": 0, 2023-05-24T21:17:15.055 INFO:teuthology.orchestra.run.smithi071.stdout: "max_objects": -1 2023-05-24T21:17:15.055 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:17:15.056 INFO:teuthology.orchestra.run.smithi071.stdout: "user_ratelimit": { 2023-05-24T21:17:15.056 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_ops": 0, 2023-05-24T21:17:15.056 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_ops": 0, 2023-05-24T21:17:15.056 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_bytes": 0, 2023-05-24T21:17:15.056 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_bytes": 0, 2023-05-24T21:17:15.056 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false 2023-05-24T21:17:15.057 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:17:15.057 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_ratelimit": { 2023-05-24T21:17:15.057 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_ops": 0, 2023-05-24T21:17:15.057 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_ops": 0, 2023-05-24T21:17:15.057 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_bytes": 0, 2023-05-24T21:17:15.057 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_bytes": 0, 2023-05-24T21:17:15.057 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false 2023-05-24T21:17:15.059 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:17:15.059 INFO:teuthology.orchestra.run.smithi071.stdout: "anonymous_ratelimit": { 2023-05-24T21:17:15.059 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_ops": 0, 2023-05-24T21:17:15.059 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_ops": 0, 2023-05-24T21:17:15.060 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_bytes": 0, 2023-05-24T21:17:15.060 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_bytes": 0, 2023-05-24T21:17:15.060 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false 2023-05-24T21:17:15.060 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:17:15.060 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:17:15.060 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T21:17:15.061 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_name": "test-realm", 2023-05-24T21:17:15.061 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:17:15.061 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:17:15.062 INFO:rgw_multi.tests:Set redirect_zone zone a2 to "299a3349-fae0-459e-8632-431fc36beb1b" 2023-05-24T21:18:15.063 DEBUG:boto:path=/qskrps-35/testredirect 2023-05-24T21:18:15.064 DEBUG:boto:auth_path=/qskrps-35/testredirect 2023-05-24T21:18:15.064 DEBUG:boto:Method: HEAD 2023-05-24T21:18:15.065 DEBUG:boto:Path: /qskrps-35/testredirect 2023-05-24T21:18:15.065 DEBUG:boto:Data: 2023-05-24T21:18:15.065 DEBUG:boto:Headers: {} 2023-05-24T21:18:15.066 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:18:15.066 DEBUG:boto:Port: 8001 2023-05-24T21:18:15.067 DEBUG:boto:Params: {} 2023-05-24T21:18:15.067 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-24T21:18:15.068 DEBUG:boto:Token: None 2023-05-24T21:18:15.068 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:18:15 GMT /qskrps-35/testredirect 2023-05-24T21:18:15.069 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4sI6UQMLXTjynchVgMtUy7NKXWY= 2023-05-24T21:18:15.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:18:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4sI6UQMLXTjynchVgMtUy7NKXWY=', 'Content-Length': '0'} 2023-05-24T21:18:15.074 DEBUG:boto:Response headers: [('Location', 'http://smithi071.front.sepia.ceph.com:8000/qskrps-35/testredirect?'), ('x-amz-request-id', 'tx00000709548096f056af8-00646e7f17-1565-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:18:15.075 DEBUG:boto:Redirecting: http://smithi071.front.sepia.ceph.com/qskrps-35/testredirect 2023-05-24T21:18:15.075 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': '8000'} 2023-05-24T21:18:15.076 DEBUG:boto:Token: None 2023-05-24T21:18:15.076 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:18:15 GMT /qskrps-35/testredirect 2023-05-24T21:18:15.077 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4sI6UQMLXTjynchVgMtUy7NKXWY= 2023-05-24T21:18:15.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:18:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4sI6UQMLXTjynchVgMtUy7NKXWY=', 'Content-Length': '0'} 2023-05-24T21:18:15.082 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:16:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000b0f5b1d86109f4c9-00646e7f17-1567-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:18:15.082 DEBUG:boto:path=/qskrps-35/testredirect 2023-05-24T21:18:15.083 DEBUG:boto:auth_path=/qskrps-35/testredirect 2023-05-24T21:18:15.083 DEBUG:boto:Method: GET 2023-05-24T21:18:15.084 DEBUG:boto:Path: /qskrps-35/testredirect 2023-05-24T21:18:15.084 DEBUG:boto:Data: 2023-05-24T21:18:15.084 DEBUG:boto:Headers: {} 2023-05-24T21:18:15.085 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:18:15.085 DEBUG:boto:Port: 8001 2023-05-24T21:18:15.086 DEBUG:boto:Params: {} 2023-05-24T21:18:15.086 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-24T21:18:15.086 DEBUG:boto:Token: None 2023-05-24T21:18:15.087 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:18:15 GMT /qskrps-35/testredirect 2023-05-24T21:18:15.088 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:tdDl17uSdoeZBgx37VyCIQjZDRE= 2023-05-24T21:18:15.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:18:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:tdDl17uSdoeZBgx37VyCIQjZDRE=', 'Content-Length': '0'} 2023-05-24T21:18:15.090 DEBUG:boto:Response headers: [('Location', 'http://smithi071.front.sepia.ceph.com:8000/qskrps-35/testredirect?'), ('x-amz-request-id', 'tx00000404d23d70fa4a9b3-00646e7f17-1565-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:18:15.091 DEBUG:boto:Redirecting: http://smithi071.front.sepia.ceph.com/qskrps-35/testredirect 2023-05-24T21:18:15.091 DEBUG:boto:Token: None 2023-05-24T21:18:15.092 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:18:15 GMT /qskrps-35/testredirect 2023-05-24T21:18:15.092 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:tdDl17uSdoeZBgx37VyCIQjZDRE= 2023-05-24T21:18:15.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:18:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:tdDl17uSdoeZBgx37VyCIQjZDRE=', 'Content-Length': '0'} 2023-05-24T21:18:15.094 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:16:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000d79cd679151957a0-00646e7f17-1567-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:18:15.095 DEBUG:boto:path=/qskrps-35/testredirect 2023-05-24T21:18:15.096 DEBUG:boto:auth_path=/qskrps-35/testredirect 2023-05-24T21:18:15.096 DEBUG:boto:Method: PUT 2023-05-24T21:18:15.097 DEBUG:boto:Path: /qskrps-35/testredirect 2023-05-24T21:18:15.097 DEBUG:boto:Data: 2023-05-24T21:18:15.097 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'VpBzljOcorCZvRIkX5Nt3A==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-05-24T21:18:15.098 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:18:15.098 DEBUG:boto:Port: 8000 2023-05-24T21:18:15.099 DEBUG:boto:Params: {} 2023-05-24T21:18:15.099 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T21:18:15.099 DEBUG:boto:Token: None 2023-05-24T21:18:15.100 DEBUG:boto:StringToSign: PUT VpBzljOcorCZvRIkX5Nt3A== application/octet-stream Wed, 24 May 2023 21:18:15 GMT /qskrps-35/testredirect 2023-05-24T21:18:15.100 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:DnJMj16A3DJlgQKfBZOIkea0bYQ= 2023-05-24T21:18:15.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'VpBzljOcorCZvRIkX5Nt3A==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:18:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:DnJMj16A3DJlgQKfBZOIkea0bYQ='} 2023-05-24T21:18:15.113 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"56907396339ca2b099bd12245f936ddc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ca4ee6cb5aec3954-00646e7f17-1567-a1'), ('Date', 'Wed, 24 May 2023 21:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:18:15.113 DEBUG:boto:path=/qskrps-35/testredirect 2023-05-24T21:18:15.114 DEBUG:boto:auth_path=/qskrps-35/testredirect 2023-05-24T21:18:15.114 DEBUG:boto:Method: GET 2023-05-24T21:18:15.114 DEBUG:boto:Path: /qskrps-35/testredirect 2023-05-24T21:18:15.115 DEBUG:boto:Data: 2023-05-24T21:18:15.115 DEBUG:boto:Headers: {} 2023-05-24T21:18:15.115 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:18:15.116 DEBUG:boto:Port: 8001 2023-05-24T21:18:15.116 DEBUG:boto:Params: {} 2023-05-24T21:18:15.117 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-24T21:18:15.117 DEBUG:boto:Token: None 2023-05-24T21:18:15.117 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:18:15 GMT /qskrps-35/testredirect 2023-05-24T21:18:15.118 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:tdDl17uSdoeZBgx37VyCIQjZDRE= 2023-05-24T21:18:15.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:18:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:tdDl17uSdoeZBgx37VyCIQjZDRE=', 'Content-Length': '0'} 2023-05-24T21:18:15.121 DEBUG:boto:Response headers: [('Location', 'http://smithi071.front.sepia.ceph.com:8000/qskrps-35/testredirect?'), ('x-amz-request-id', 'tx00000e76a015b2359b4cf-00646e7f17-1565-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:18:15.121 DEBUG:boto:Redirecting: http://smithi071.front.sepia.ceph.com/qskrps-35/testredirect 2023-05-24T21:18:15.122 DEBUG:boto:Token: None 2023-05-24T21:18:15.122 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:18:15 GMT /qskrps-35/testredirect 2023-05-24T21:18:15.123 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:tdDl17uSdoeZBgx37VyCIQjZDRE= 2023-05-24T21:18:15.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:18:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:tdDl17uSdoeZBgx37VyCIQjZDRE=', 'Content-Length': '0'} 2023-05-24T21:18:15.125 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:18:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"56907396339ca2b099bd12245f936ddc"'), ('x-amz-request-id', 'tx000007b014a12b748e8e8-00646e7f17-1567-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:18:15.126 DEBUG:boto:path=/qskrps-35/testredirect 2023-05-24T21:18:15.126 DEBUG:boto:auth_path=/qskrps-35/testredirect 2023-05-24T21:18:15.126 DEBUG:boto:Method: PUT 2023-05-24T21:18:15.127 DEBUG:boto:Path: /qskrps-35/testredirect 2023-05-24T21:18:15.127 DEBUG:boto:Data: 2023-05-24T21:18:15.128 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'uk9S5NXZfBvPq4jGr+LM5g==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-05-24T21:18:15.128 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:18:15.128 DEBUG:boto:Port: 8000 2023-05-24T21:18:15.129 DEBUG:boto:Params: {} 2023-05-24T21:18:15.129 DEBUG:boto:Token: None 2023-05-24T21:18:15.130 DEBUG:boto:StringToSign: PUT uk9S5NXZfBvPq4jGr+LM5g== application/octet-stream Wed, 24 May 2023 21:18:15 GMT /qskrps-35/testredirect 2023-05-24T21:18:15.130 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:QuyOTMQ8x3IU+Ba40Bu8s1C4JlI= 2023-05-24T21:18:15.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'uk9S5NXZfBvPq4jGr+LM5g==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:18:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:QuyOTMQ8x3IU+Ba40Bu8s1C4JlI='} 2023-05-24T21:18:15.138 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"ba4f52e4d5d97c1bcfab88c6afe2cce6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003c00c2383b9bcead-00646e7f17-1567-a1'), ('Date', 'Wed, 24 May 2023 21:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:18:15.139 DEBUG:boto:path=/qskrps-35/testredirect 2023-05-24T21:18:15.139 DEBUG:boto:auth_path=/qskrps-35/testredirect 2023-05-24T21:18:15.140 DEBUG:boto:Method: GET 2023-05-24T21:18:15.140 DEBUG:boto:Path: /qskrps-35/testredirect 2023-05-24T21:18:15.140 DEBUG:boto:Data: 2023-05-24T21:18:15.141 DEBUG:boto:Headers: {} 2023-05-24T21:18:15.141 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:18:15.141 DEBUG:boto:Port: 8001 2023-05-24T21:18:15.142 DEBUG:boto:Params: {} 2023-05-24T21:18:15.142 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-24T21:18:15.143 DEBUG:boto:Token: None 2023-05-24T21:18:15.143 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:18:15 GMT /qskrps-35/testredirect 2023-05-24T21:18:15.144 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:tdDl17uSdoeZBgx37VyCIQjZDRE= 2023-05-24T21:18:15.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:18:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:tdDl17uSdoeZBgx37VyCIQjZDRE=', 'Content-Length': '0'} 2023-05-24T21:18:15.146 DEBUG:boto:Response headers: [('Location', 'http://smithi071.front.sepia.ceph.com:8000/qskrps-35/testredirect?'), ('x-amz-request-id', 'tx0000065b59c30556d6671-00646e7f17-1565-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:18:15.147 DEBUG:boto:Redirecting: http://smithi071.front.sepia.ceph.com/qskrps-35/testredirect 2023-05-24T21:18:15.147 DEBUG:boto:Token: None 2023-05-24T21:18:15.148 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:18:15 GMT /qskrps-35/testredirect 2023-05-24T21:18:15.148 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:tdDl17uSdoeZBgx37VyCIQjZDRE= 2023-05-24T21:18:15.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:18:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:tdDl17uSdoeZBgx37VyCIQjZDRE=', 'Content-Length': '0'} 2023-05-24T21:18:15.151 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:18:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"ba4f52e4d5d97c1bcfab88c6afe2cce6"'), ('x-amz-request-id', 'tx00000146ae7df9b8ebbb2-00646e7f17-1567-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:18:15.151 DEBUG:boto:path=/qskrps-35/testredirect 2023-05-24T21:18:15.152 DEBUG:boto:auth_path=/qskrps-35/testredirect 2023-05-24T21:18:15.152 DEBUG:boto:Method: PUT 2023-05-24T21:18:15.153 DEBUG:boto:Path: /qskrps-35/testredirect 2023-05-24T21:18:15.153 DEBUG:boto:Data: 2023-05-24T21:18:15.153 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '6aKmvaJMgaK7Z2rfDZvVeg==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-05-24T21:18:15.154 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:18:15.154 DEBUG:boto:Port: 8000 2023-05-24T21:18:15.154 DEBUG:boto:Params: {} 2023-05-24T21:18:15.155 DEBUG:boto:Token: None 2023-05-24T21:18:15.155 DEBUG:boto:StringToSign: PUT 6aKmvaJMgaK7Z2rfDZvVeg== application/octet-stream Wed, 24 May 2023 21:18:15 GMT /qskrps-35/testredirect 2023-05-24T21:18:15.156 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:qb5B0hkEHzW9xA55AnqvQjdHc2A= 2023-05-24T21:18:15.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '6aKmvaJMgaK7Z2rfDZvVeg==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:18:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qb5B0hkEHzW9xA55AnqvQjdHc2A='} 2023-05-24T21:18:15.160 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"e9a2a6bda24c81a2bb676adf0d9bd57a"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ee84868a432b48ae-00646e7f17-1567-a1'), ('Date', 'Wed, 24 May 2023 21:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:18:15.161 DEBUG:boto:path=/qskrps-35/testredirect 2023-05-24T21:18:15.161 DEBUG:boto:auth_path=/qskrps-35/testredirect 2023-05-24T21:18:15.162 DEBUG:boto:Method: GET 2023-05-24T21:18:15.162 DEBUG:boto:Path: /qskrps-35/testredirect 2023-05-24T21:18:15.162 DEBUG:boto:Data: 2023-05-24T21:18:15.163 DEBUG:boto:Headers: {} 2023-05-24T21:18:15.163 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:18:15.164 DEBUG:boto:Port: 8001 2023-05-24T21:18:15.164 DEBUG:boto:Params: {} 2023-05-24T21:18:15.164 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-24T21:18:15.165 DEBUG:boto:Token: None 2023-05-24T21:18:15.165 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:18:15 GMT /qskrps-35/testredirect 2023-05-24T21:18:15.166 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:tdDl17uSdoeZBgx37VyCIQjZDRE= 2023-05-24T21:18:15.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:18:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:tdDl17uSdoeZBgx37VyCIQjZDRE=', 'Content-Length': '0'} 2023-05-24T21:18:15.168 DEBUG:boto:Response headers: [('Location', 'http://smithi071.front.sepia.ceph.com:8000/qskrps-35/testredirect?'), ('x-amz-request-id', 'tx00000a5d07f4b1f6ae3f3-00646e7f17-1565-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:18:15.169 DEBUG:boto:Redirecting: http://smithi071.front.sepia.ceph.com/qskrps-35/testredirect 2023-05-24T21:18:15.169 DEBUG:boto:Token: None 2023-05-24T21:18:15.170 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:18:15 GMT /qskrps-35/testredirect 2023-05-24T21:18:15.170 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:tdDl17uSdoeZBgx37VyCIQjZDRE= 2023-05-24T21:18:15.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:18:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:tdDl17uSdoeZBgx37VyCIQjZDRE=', 'Content-Length': '0'} 2023-05-24T21:18:15.173 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:18:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"e9a2a6bda24c81a2bb676adf0d9bd57a"'), ('x-amz-request-id', 'tx0000008083c9e0966a98f-00646e7f17-1567-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:18:15.174 DEBUG:boto:path=/qskrps-35/testredirect 2023-05-24T21:18:15.174 DEBUG:boto:auth_path=/qskrps-35/testredirect 2023-05-24T21:18:15.174 DEBUG:boto:Method: PUT 2023-05-24T21:18:15.175 DEBUG:boto:Path: /qskrps-35/testredirect 2023-05-24T21:18:15.175 DEBUG:boto:Data: 2023-05-24T21:18:15.175 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'GbNgNQ5MAukCtgY+rsT5Xw==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-05-24T21:18:15.176 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:18:15.176 DEBUG:boto:Port: 8000 2023-05-24T21:18:15.177 DEBUG:boto:Params: {} 2023-05-24T21:18:15.177 DEBUG:boto:Token: None 2023-05-24T21:18:15.177 DEBUG:boto:StringToSign: PUT GbNgNQ5MAukCtgY+rsT5Xw== application/octet-stream Wed, 24 May 2023 21:18:15 GMT /qskrps-35/testredirect 2023-05-24T21:18:15.178 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:f+ICuYMa86DkxzRP6Okth4U0ldk= 2023-05-24T21:18:15.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'GbNgNQ5MAukCtgY+rsT5Xw==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:18:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:f+ICuYMa86DkxzRP6Okth4U0ldk='} 2023-05-24T21:18:15.220 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"19b360350e4c02e902b6063eaec4f95f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000074fafd654039f013-00646e7f17-1567-a1'), ('Date', 'Wed, 24 May 2023 21:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:18:15.221 DEBUG:boto:path=/qskrps-35/testredirect 2023-05-24T21:18:15.221 DEBUG:boto:auth_path=/qskrps-35/testredirect 2023-05-24T21:18:15.222 DEBUG:boto:Method: GET 2023-05-24T21:18:15.222 DEBUG:boto:Path: /qskrps-35/testredirect 2023-05-24T21:18:15.223 DEBUG:boto:Data: 2023-05-24T21:18:15.223 DEBUG:boto:Headers: {} 2023-05-24T21:18:15.223 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:18:15.224 DEBUG:boto:Port: 8001 2023-05-24T21:18:15.224 DEBUG:boto:Params: {} 2023-05-24T21:18:15.224 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-24T21:18:15.225 DEBUG:boto:Token: None 2023-05-24T21:18:15.225 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:18:15 GMT /qskrps-35/testredirect 2023-05-24T21:18:15.226 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:tdDl17uSdoeZBgx37VyCIQjZDRE= 2023-05-24T21:18:15.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:18:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:tdDl17uSdoeZBgx37VyCIQjZDRE=', 'Content-Length': '0'} 2023-05-24T21:18:15.228 DEBUG:boto:Response headers: [('Location', 'http://smithi071.front.sepia.ceph.com:8000/qskrps-35/testredirect?'), ('x-amz-request-id', 'tx00000a0e2c7cf0dcf610d-00646e7f17-1565-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:18:15.229 DEBUG:boto:Redirecting: http://smithi071.front.sepia.ceph.com/qskrps-35/testredirect 2023-05-24T21:18:15.229 DEBUG:boto:Token: None 2023-05-24T21:18:15.230 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:18:15 GMT /qskrps-35/testredirect 2023-05-24T21:18:15.230 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:tdDl17uSdoeZBgx37VyCIQjZDRE= 2023-05-24T21:18:15.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:18:15 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:tdDl17uSdoeZBgx37VyCIQjZDRE=', 'Content-Length': '0'} 2023-05-24T21:18:15.233 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:18:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"19b360350e4c02e902b6063eaec4f95f"'), ('x-amz-request-id', 'tx0000005ad15d8b636618d-00646e7f17-1567-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:18:15.233 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:18:15.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:18:15.234 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --sync-from-all=true --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-24T21:18:15.288 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:18:15.289 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:18:15.310 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:18:15.310 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "5af1588c-f9b9-4687-a040-ce6e0cc808d9", 2023-05-24T21:18:15.311 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a2", 2023-05-24T21:18:15.311 INFO:teuthology.orchestra.run.smithi071.stdout: "domain_root": "a2.rgw.meta:root", 2023-05-24T21:18:15.311 INFO:teuthology.orchestra.run.smithi071.stdout: "control_pool": "a2.rgw.control", 2023-05-24T21:18:15.311 INFO:teuthology.orchestra.run.smithi071.stdout: "gc_pool": "a2.rgw.log:gc", 2023-05-24T21:18:15.311 INFO:teuthology.orchestra.run.smithi071.stdout: "lc_pool": "a2.rgw.log:lc", 2023-05-24T21:18:15.311 INFO:teuthology.orchestra.run.smithi071.stdout: "log_pool": "a2.rgw.log", 2023-05-24T21:18:15.312 INFO:teuthology.orchestra.run.smithi071.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-05-24T21:18:15.312 INFO:teuthology.orchestra.run.smithi071.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-05-24T21:18:15.312 INFO:teuthology.orchestra.run.smithi071.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-05-24T21:18:15.312 INFO:teuthology.orchestra.run.smithi071.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-05-24T21:18:15.312 INFO:teuthology.orchestra.run.smithi071.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-05-24T21:18:15.312 INFO:teuthology.orchestra.run.smithi071.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-05-24T21:18:15.313 INFO:teuthology.orchestra.run.smithi071.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-05-24T21:18:15.313 INFO:teuthology.orchestra.run.smithi071.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-05-24T21:18:15.313 INFO:teuthology.orchestra.run.smithi071.stdout: "otp_pool": "a2.rgw.otp", 2023-05-24T21:18:15.313 INFO:teuthology.orchestra.run.smithi071.stdout: "system_key": { 2023-05-24T21:18:15.313 INFO:teuthology.orchestra.run.smithi071.stdout: "access_key": "Z6KKVUYPP0489K03", 2023-05-24T21:18:15.313 INFO:teuthology.orchestra.run.smithi071.stdout: "secret_key": "EX5IDzHTe9AVqa8b1PSD8B5Y1CKI3jBg" 2023-05-24T21:18:15.313 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:18:15.314 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_pools": [ 2023-05-24T21:18:15.314 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:18:15.314 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "default-placement", 2023-05-24T21:18:15.314 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:18:15.314 INFO:teuthology.orchestra.run.smithi071.stdout: "index_pool": "a2.rgw.buckets.index", 2023-05-24T21:18:15.314 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": { 2023-05-24T21:18:15.314 INFO:teuthology.orchestra.run.smithi071.stdout: "STANDARD": { 2023-05-24T21:18:15.315 INFO:teuthology.orchestra.run.smithi071.stdout: "data_pool": "a2.rgw.buckets.data", 2023-05-24T21:18:15.315 INFO:teuthology.orchestra.run.smithi071.stdout: "compression_type": "random" 2023-05-24T21:18:15.315 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:18:15.315 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:18:15.315 INFO:teuthology.orchestra.run.smithi071.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-05-24T21:18:15.315 INFO:teuthology.orchestra.run.smithi071.stdout: "index_type": 0, 2023-05-24T21:18:15.315 INFO:teuthology.orchestra.run.smithi071.stdout: "inline_data": true 2023-05-24T21:18:15.316 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:18:15.316 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:18:15.316 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:18:15.316 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T21:18:15.316 INFO:teuthology.orchestra.run.smithi071.stdout: "notif_pool": "a2.rgw.log:notif" 2023-05-24T21:18:15.316 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:18:15.317 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:18:15.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:18:15.318 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-24T21:18:15.368 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:18:15.369 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:18:15.411 INFO:teuthology.orchestra.run.smithi071.stderr:Sending period to new master zone 299a3349-fae0-459e-8632-431fc36beb1b 2023-05-24T21:18:15.448 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:18:15.449 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:18:15.449 INFO:teuthology.orchestra.run.smithi071.stdout: "epoch": 7, 2023-05-24T21:18:15.449 INFO:teuthology.orchestra.run.smithi071.stdout: "predecessor_uuid": "1cea3a6e-eb52-480e-95b7-3f6e43317e60", 2023-05-24T21:18:15.449 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": [], 2023-05-24T21:18:15.449 INFO:teuthology.orchestra.run.smithi071.stdout: "period_map": { 2023-05-24T21:18:15.449 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:18:15.450 INFO:teuthology.orchestra.run.smithi071.stdout: "zonegroups": [ 2023-05-24T21:18:15.450 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:18:15.450 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "a", 2023-05-24T21:18:15.450 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a", 2023-05-24T21:18:15.450 INFO:teuthology.orchestra.run.smithi071.stdout: "api_name": "", 2023-05-24T21:18:15.450 INFO:teuthology.orchestra.run.smithi071.stdout: "is_master": true, 2023-05-24T21:18:15.451 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:18:15.451 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-05-24T21:18:15.451 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:18:15.451 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames": [], 2023-05-24T21:18:15.451 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames_s3website": [], 2023-05-24T21:18:15.451 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zone": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T21:18:15.451 INFO:teuthology.orchestra.run.smithi071.stdout: "zones": [ 2023-05-24T21:18:15.452 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:18:15.452 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T21:18:15.452 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a1", 2023-05-24T21:18:15.452 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:18:15.452 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-05-24T21:18:15.452 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:18:15.452 INFO:teuthology.orchestra.run.smithi071.stdout: "log_meta": false, 2023-05-24T21:18:15.453 INFO:teuthology.orchestra.run.smithi071.stdout: "log_data": true, 2023-05-24T21:18:15.453 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_index_max_shards": 11, 2023-05-24T21:18:15.453 INFO:teuthology.orchestra.run.smithi071.stdout: "read_only": false, 2023-05-24T21:18:15.453 INFO:teuthology.orchestra.run.smithi071.stdout: "tier_type": "", 2023-05-24T21:18:15.453 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from_all": true, 2023-05-24T21:18:15.453 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from": [], 2023-05-24T21:18:15.453 INFO:teuthology.orchestra.run.smithi071.stdout: "redirect_zone": "", 2023-05-24T21:18:15.454 INFO:teuthology.orchestra.run.smithi071.stdout: "supported_features": [ 2023-05-24T21:18:15.454 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:18:15.454 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:18:15.454 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:18:15.454 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:18:15.454 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "5af1588c-f9b9-4687-a040-ce6e0cc808d9", 2023-05-24T21:18:15.454 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a2", 2023-05-24T21:18:15.455 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:18:15.455 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi071.front.sepia.ceph.com:8001" 2023-05-24T21:18:15.455 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:18:15.455 INFO:teuthology.orchestra.run.smithi071.stdout: "log_meta": false, 2023-05-24T21:18:15.455 INFO:teuthology.orchestra.run.smithi071.stdout: "log_data": true, 2023-05-24T21:18:15.455 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_index_max_shards": 11, 2023-05-24T21:18:15.455 INFO:teuthology.orchestra.run.smithi071.stdout: "read_only": false, 2023-05-24T21:18:15.456 INFO:teuthology.orchestra.run.smithi071.stdout: "tier_type": "", 2023-05-24T21:18:15.456 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from_all": true, 2023-05-24T21:18:15.456 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from": [], 2023-05-24T21:18:15.456 INFO:teuthology.orchestra.run.smithi071.stdout: "redirect_zone": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T21:18:15.456 INFO:teuthology.orchestra.run.smithi071.stdout: "supported_features": [ 2023-05-24T21:18:15.456 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:18:15.456 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:18:15.457 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:18:15.457 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:18:15.457 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_targets": [ 2023-05-24T21:18:15.457 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:18:15.457 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "default-placement", 2023-05-24T21:18:15.457 INFO:teuthology.orchestra.run.smithi071.stdout: "tags": [], 2023-05-24T21:18:15.457 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": [ 2023-05-24T21:18:15.458 INFO:teuthology.orchestra.run.smithi071.stdout: "STANDARD" 2023-05-24T21:18:15.458 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:18:15.458 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:18:15.458 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:18:15.458 INFO:teuthology.orchestra.run.smithi071.stdout: "default_placement": "default-placement", 2023-05-24T21:18:15.458 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T21:18:15.458 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_policy": { 2023-05-24T21:18:15.459 INFO:teuthology.orchestra.run.smithi071.stdout: "groups": [] 2023-05-24T21:18:15.459 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:18:15.459 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled_features": [ 2023-05-24T21:18:15.459 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:18:15.459 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:18:15.459 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:18:15.459 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:18:15.460 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "b", 2023-05-24T21:18:15.460 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "b", 2023-05-24T21:18:15.460 INFO:teuthology.orchestra.run.smithi071.stdout: "api_name": "", 2023-05-24T21:18:15.460 INFO:teuthology.orchestra.run.smithi071.stdout: "is_master": false, 2023-05-24T21:18:15.460 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:18:15.460 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi157.front.sepia.ceph.com:8000" 2023-05-24T21:18:15.461 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:18:15.461 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames": [], 2023-05-24T21:18:15.461 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames_s3website": [], 2023-05-24T21:18:15.461 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zone": "ee18cd86-4684-4b3d-8de6-0075d3d925f1", 2023-05-24T21:18:15.461 INFO:teuthology.orchestra.run.smithi071.stdout: "zones": [ 2023-05-24T21:18:15.461 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:18:15.461 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "7b8c3b69-c8ed-4621-b2ef-b89d3bbb7973", 2023-05-24T21:18:15.462 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "b2", 2023-05-24T21:18:15.462 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:18:15.462 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi157.front.sepia.ceph.com:8001" 2023-05-24T21:18:15.462 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:18:15.462 INFO:teuthology.orchestra.run.smithi071.stdout: "log_meta": false, 2023-05-24T21:18:15.462 INFO:teuthology.orchestra.run.smithi071.stdout: "log_data": true, 2023-05-24T21:18:15.462 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_index_max_shards": 11, 2023-05-24T21:18:15.463 INFO:teuthology.orchestra.run.smithi071.stdout: "read_only": false, 2023-05-24T21:18:15.463 INFO:teuthology.orchestra.run.smithi071.stdout: "tier_type": "", 2023-05-24T21:18:15.463 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from_all": true, 2023-05-24T21:18:15.463 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from": [], 2023-05-24T21:18:15.463 INFO:teuthology.orchestra.run.smithi071.stdout: "redirect_zone": "", 2023-05-24T21:18:15.463 INFO:teuthology.orchestra.run.smithi071.stdout: "supported_features": [ 2023-05-24T21:18:15.463 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:18:15.463 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:18:15.464 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:18:15.464 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:18:15.472 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "ee18cd86-4684-4b3d-8de6-0075d3d925f1", 2023-05-24T21:18:15.472 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "b1", 2023-05-24T21:18:15.473 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:18:15.473 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi157.front.sepia.ceph.com:8000" 2023-05-24T21:18:15.473 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:18:15.473 INFO:teuthology.orchestra.run.smithi071.stdout: "log_meta": false, 2023-05-24T21:18:15.473 INFO:teuthology.orchestra.run.smithi071.stdout: "log_data": true, 2023-05-24T21:18:15.473 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_index_max_shards": 11, 2023-05-24T21:18:15.473 INFO:teuthology.orchestra.run.smithi071.stdout: "read_only": false, 2023-05-24T21:18:15.474 INFO:teuthology.orchestra.run.smithi071.stdout: "tier_type": "", 2023-05-24T21:18:15.474 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from_all": true, 2023-05-24T21:18:15.474 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from": [], 2023-05-24T21:18:15.474 INFO:teuthology.orchestra.run.smithi071.stdout: "redirect_zone": "", 2023-05-24T21:18:15.474 INFO:teuthology.orchestra.run.smithi071.stdout: "supported_features": [ 2023-05-24T21:18:15.474 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:18:15.475 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:18:15.475 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:18:15.475 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:18:15.475 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_targets": [ 2023-05-24T21:18:15.475 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:18:15.475 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "default-placement", 2023-05-24T21:18:15.475 INFO:teuthology.orchestra.run.smithi071.stdout: "tags": [], 2023-05-24T21:18:15.476 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": [ 2023-05-24T21:18:15.476 INFO:teuthology.orchestra.run.smithi071.stdout: "STANDARD" 2023-05-24T21:18:15.476 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:18:15.476 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:18:15.476 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:18:15.476 INFO:teuthology.orchestra.run.smithi071.stdout: "default_placement": "default-placement", 2023-05-24T21:18:15.476 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T21:18:15.477 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_policy": { 2023-05-24T21:18:15.477 INFO:teuthology.orchestra.run.smithi071.stdout: "groups": [] 2023-05-24T21:18:15.477 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:18:15.477 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled_features": [ 2023-05-24T21:18:15.477 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:18:15.477 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:18:15.477 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:18:15.478 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:18:15.478 INFO:teuthology.orchestra.run.smithi071.stdout: "short_zone_ids": [ 2023-05-24T21:18:15.478 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:18:15.478 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T21:18:15.478 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 3764661814 2023-05-24T21:18:15.478 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:18:15.478 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:18:15.479 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "5af1588c-f9b9-4687-a040-ce6e0cc808d9", 2023-05-24T21:18:15.479 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 2266179370 2023-05-24T21:18:15.479 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:18:15.479 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:18:15.479 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "7b8c3b69-c8ed-4621-b2ef-b89d3bbb7973", 2023-05-24T21:18:15.479 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 1540450555 2023-05-24T21:18:15.479 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:18:15.480 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:18:15.480 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "ee18cd86-4684-4b3d-8de6-0075d3d925f1", 2023-05-24T21:18:15.480 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 489875403 2023-05-24T21:18:15.480 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:18:15.480 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:18:15.480 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:18:15.480 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zonegroup": "a", 2023-05-24T21:18:15.481 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zone": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T21:18:15.481 INFO:teuthology.orchestra.run.smithi071.stdout: "period_config": { 2023-05-24T21:18:15.481 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_quota": { 2023-05-24T21:18:15.481 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false, 2023-05-24T21:18:15.481 INFO:teuthology.orchestra.run.smithi071.stdout: "check_on_raw": false, 2023-05-24T21:18:15.481 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size": -1, 2023-05-24T21:18:15.482 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size_kb": 0, 2023-05-24T21:18:15.482 INFO:teuthology.orchestra.run.smithi071.stdout: "max_objects": -1 2023-05-24T21:18:15.482 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:18:15.482 INFO:teuthology.orchestra.run.smithi071.stdout: "user_quota": { 2023-05-24T21:18:15.482 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false, 2023-05-24T21:18:15.482 INFO:teuthology.orchestra.run.smithi071.stdout: "check_on_raw": false, 2023-05-24T21:18:15.482 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size": -1, 2023-05-24T21:18:15.483 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size_kb": 0, 2023-05-24T21:18:15.483 INFO:teuthology.orchestra.run.smithi071.stdout: "max_objects": -1 2023-05-24T21:18:15.483 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:18:15.483 INFO:teuthology.orchestra.run.smithi071.stdout: "user_ratelimit": { 2023-05-24T21:18:15.483 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_ops": 0, 2023-05-24T21:18:15.483 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_ops": 0, 2023-05-24T21:18:15.483 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_bytes": 0, 2023-05-24T21:18:15.484 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_bytes": 0, 2023-05-24T21:18:15.484 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false 2023-05-24T21:18:15.484 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:18:15.484 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_ratelimit": { 2023-05-24T21:18:15.484 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_ops": 0, 2023-05-24T21:18:15.484 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_ops": 0, 2023-05-24T21:18:15.484 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_bytes": 0, 2023-05-24T21:18:15.485 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_bytes": 0, 2023-05-24T21:18:15.485 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false 2023-05-24T21:18:15.485 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:18:15.485 INFO:teuthology.orchestra.run.smithi071.stdout: "anonymous_ratelimit": { 2023-05-24T21:18:15.485 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_ops": 0, 2023-05-24T21:18:15.485 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_ops": 0, 2023-05-24T21:18:15.485 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_bytes": 0, 2023-05-24T21:18:15.486 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_bytes": 0, 2023-05-24T21:18:15.486 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false 2023-05-24T21:18:15.486 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:18:15.486 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:18:15.486 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T21:18:15.486 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_name": "test-realm", 2023-05-24T21:18:15.486 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:18:15.487 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:18:15.488 INFO:rgw_multi.tests:Set sync_from_all flag on zone a2 to true 2023-05-24T21:19:15.488 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:19:15.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:19:15.490 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --redirect-zone= --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-24T21:19:15.547 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:19:15.547 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:19:15.569 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:19:15.569 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "5af1588c-f9b9-4687-a040-ce6e0cc808d9", 2023-05-24T21:19:15.569 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a2", 2023-05-24T21:19:15.570 INFO:teuthology.orchestra.run.smithi071.stdout: "domain_root": "a2.rgw.meta:root", 2023-05-24T21:19:15.570 INFO:teuthology.orchestra.run.smithi071.stdout: "control_pool": "a2.rgw.control", 2023-05-24T21:19:15.570 INFO:teuthology.orchestra.run.smithi071.stdout: "gc_pool": "a2.rgw.log:gc", 2023-05-24T21:19:15.570 INFO:teuthology.orchestra.run.smithi071.stdout: "lc_pool": "a2.rgw.log:lc", 2023-05-24T21:19:15.570 INFO:teuthology.orchestra.run.smithi071.stdout: "log_pool": "a2.rgw.log", 2023-05-24T21:19:15.570 INFO:teuthology.orchestra.run.smithi071.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-05-24T21:19:15.570 INFO:teuthology.orchestra.run.smithi071.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-05-24T21:19:15.571 INFO:teuthology.orchestra.run.smithi071.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-05-24T21:19:15.571 INFO:teuthology.orchestra.run.smithi071.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-05-24T21:19:15.571 INFO:teuthology.orchestra.run.smithi071.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-05-24T21:19:15.571 INFO:teuthology.orchestra.run.smithi071.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-05-24T21:19:15.571 INFO:teuthology.orchestra.run.smithi071.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-05-24T21:19:15.571 INFO:teuthology.orchestra.run.smithi071.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-05-24T21:19:15.572 INFO:teuthology.orchestra.run.smithi071.stdout: "otp_pool": "a2.rgw.otp", 2023-05-24T21:19:15.572 INFO:teuthology.orchestra.run.smithi071.stdout: "system_key": { 2023-05-24T21:19:15.572 INFO:teuthology.orchestra.run.smithi071.stdout: "access_key": "Z6KKVUYPP0489K03", 2023-05-24T21:19:15.572 INFO:teuthology.orchestra.run.smithi071.stdout: "secret_key": "EX5IDzHTe9AVqa8b1PSD8B5Y1CKI3jBg" 2023-05-24T21:19:15.572 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:19:15.572 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_pools": [ 2023-05-24T21:19:15.572 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:19:15.573 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "default-placement", 2023-05-24T21:19:15.573 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:19:15.573 INFO:teuthology.orchestra.run.smithi071.stdout: "index_pool": "a2.rgw.buckets.index", 2023-05-24T21:19:15.573 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": { 2023-05-24T21:19:15.573 INFO:teuthology.orchestra.run.smithi071.stdout: "STANDARD": { 2023-05-24T21:19:15.573 INFO:teuthology.orchestra.run.smithi071.stdout: "data_pool": "a2.rgw.buckets.data", 2023-05-24T21:19:15.574 INFO:teuthology.orchestra.run.smithi071.stdout: "compression_type": "random" 2023-05-24T21:19:15.574 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:19:15.574 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:19:15.574 INFO:teuthology.orchestra.run.smithi071.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-05-24T21:19:15.574 INFO:teuthology.orchestra.run.smithi071.stdout: "index_type": 0, 2023-05-24T21:19:15.574 INFO:teuthology.orchestra.run.smithi071.stdout: "inline_data": true 2023-05-24T21:19:15.574 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:19:15.575 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:19:15.575 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:19:15.575 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T21:19:15.575 INFO:teuthology.orchestra.run.smithi071.stdout: "notif_pool": "a2.rgw.log:notif" 2023-05-24T21:19:15.575 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:19:15.576 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:19:15.577 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:19:15.577 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-24T21:19:15.627 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:19:15.627 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:19:15.694 INFO:teuthology.orchestra.run.smithi071.stderr:Sending period to new master zone 299a3349-fae0-459e-8632-431fc36beb1b 2023-05-24T21:19:15.742 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:19:15.743 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:19:15.743 INFO:teuthology.orchestra.run.smithi071.stdout: "epoch": 8, 2023-05-24T21:19:15.743 INFO:teuthology.orchestra.run.smithi071.stdout: "predecessor_uuid": "1cea3a6e-eb52-480e-95b7-3f6e43317e60", 2023-05-24T21:19:15.743 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": [], 2023-05-24T21:19:15.743 INFO:teuthology.orchestra.run.smithi071.stdout: "period_map": { 2023-05-24T21:19:15.743 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:19:15.744 INFO:teuthology.orchestra.run.smithi071.stdout: "zonegroups": [ 2023-05-24T21:19:15.744 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:19:15.744 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "a", 2023-05-24T21:19:15.744 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a", 2023-05-24T21:19:15.745 INFO:teuthology.orchestra.run.smithi071.stdout: "api_name": "", 2023-05-24T21:19:15.745 INFO:teuthology.orchestra.run.smithi071.stdout: "is_master": true, 2023-05-24T21:19:15.745 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:19:15.745 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-05-24T21:19:15.745 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:19:15.745 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames": [], 2023-05-24T21:19:15.745 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames_s3website": [], 2023-05-24T21:19:15.746 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zone": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T21:19:15.746 INFO:teuthology.orchestra.run.smithi071.stdout: "zones": [ 2023-05-24T21:19:15.746 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:19:15.746 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T21:19:15.746 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a1", 2023-05-24T21:19:15.746 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:19:15.747 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-05-24T21:19:15.747 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:19:15.747 INFO:teuthology.orchestra.run.smithi071.stdout: "log_meta": false, 2023-05-24T21:19:15.747 INFO:teuthology.orchestra.run.smithi071.stdout: "log_data": true, 2023-05-24T21:19:15.747 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_index_max_shards": 11, 2023-05-24T21:19:15.747 INFO:teuthology.orchestra.run.smithi071.stdout: "read_only": false, 2023-05-24T21:19:15.747 INFO:teuthology.orchestra.run.smithi071.stdout: "tier_type": "", 2023-05-24T21:19:15.748 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from_all": true, 2023-05-24T21:19:15.748 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from": [], 2023-05-24T21:19:15.748 INFO:teuthology.orchestra.run.smithi071.stdout: "redirect_zone": "", 2023-05-24T21:19:15.748 INFO:teuthology.orchestra.run.smithi071.stdout: "supported_features": [ 2023-05-24T21:19:15.748 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:19:15.748 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:19:15.748 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:19:15.749 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:19:15.749 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "5af1588c-f9b9-4687-a040-ce6e0cc808d9", 2023-05-24T21:19:15.749 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a2", 2023-05-24T21:19:15.749 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:19:15.749 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi071.front.sepia.ceph.com:8001" 2023-05-24T21:19:15.749 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:19:15.749 INFO:teuthology.orchestra.run.smithi071.stdout: "log_meta": false, 2023-05-24T21:19:15.750 INFO:teuthology.orchestra.run.smithi071.stdout: "log_data": true, 2023-05-24T21:19:15.750 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_index_max_shards": 11, 2023-05-24T21:19:15.750 INFO:teuthology.orchestra.run.smithi071.stdout: "read_only": false, 2023-05-24T21:19:15.750 INFO:teuthology.orchestra.run.smithi071.stdout: "tier_type": "", 2023-05-24T21:19:15.750 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from_all": true, 2023-05-24T21:19:15.750 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from": [], 2023-05-24T21:19:15.750 INFO:teuthology.orchestra.run.smithi071.stdout: "redirect_zone": "", 2023-05-24T21:19:15.751 INFO:teuthology.orchestra.run.smithi071.stdout: "supported_features": [ 2023-05-24T21:19:15.751 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:19:15.751 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:19:15.751 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:19:15.751 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:19:15.751 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_targets": [ 2023-05-24T21:19:15.751 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:19:15.752 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "default-placement", 2023-05-24T21:19:15.752 INFO:teuthology.orchestra.run.smithi071.stdout: "tags": [], 2023-05-24T21:19:15.752 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": [ 2023-05-24T21:19:15.752 INFO:teuthology.orchestra.run.smithi071.stdout: "STANDARD" 2023-05-24T21:19:15.752 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:19:15.752 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:19:15.752 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:19:15.753 INFO:teuthology.orchestra.run.smithi071.stdout: "default_placement": "default-placement", 2023-05-24T21:19:15.753 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T21:19:15.753 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_policy": { 2023-05-24T21:19:15.753 INFO:teuthology.orchestra.run.smithi071.stdout: "groups": [] 2023-05-24T21:19:15.753 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:19:15.753 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled_features": [ 2023-05-24T21:19:15.753 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:19:15.754 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:19:15.754 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:19:15.754 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:19:15.754 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "b", 2023-05-24T21:19:15.754 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "b", 2023-05-24T21:19:15.754 INFO:teuthology.orchestra.run.smithi071.stdout: "api_name": "", 2023-05-24T21:19:15.755 INFO:teuthology.orchestra.run.smithi071.stdout: "is_master": false, 2023-05-24T21:19:15.755 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:19:15.755 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi157.front.sepia.ceph.com:8000" 2023-05-24T21:19:15.755 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:19:15.755 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames": [], 2023-05-24T21:19:15.755 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames_s3website": [], 2023-05-24T21:19:15.755 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zone": "ee18cd86-4684-4b3d-8de6-0075d3d925f1", 2023-05-24T21:19:15.756 INFO:teuthology.orchestra.run.smithi071.stdout: "zones": [ 2023-05-24T21:19:15.756 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:19:15.756 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "7b8c3b69-c8ed-4621-b2ef-b89d3bbb7973", 2023-05-24T21:19:15.756 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "b2", 2023-05-24T21:19:15.756 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:19:15.756 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi157.front.sepia.ceph.com:8001" 2023-05-24T21:19:15.756 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:19:15.757 INFO:teuthology.orchestra.run.smithi071.stdout: "log_meta": false, 2023-05-24T21:19:15.757 INFO:teuthology.orchestra.run.smithi071.stdout: "log_data": true, 2023-05-24T21:19:15.757 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_index_max_shards": 11, 2023-05-24T21:19:15.757 INFO:teuthology.orchestra.run.smithi071.stdout: "read_only": false, 2023-05-24T21:19:15.757 INFO:teuthology.orchestra.run.smithi071.stdout: "tier_type": "", 2023-05-24T21:19:15.757 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from_all": true, 2023-05-24T21:19:15.757 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from": [], 2023-05-24T21:19:15.758 INFO:teuthology.orchestra.run.smithi071.stdout: "redirect_zone": "", 2023-05-24T21:19:15.758 INFO:teuthology.orchestra.run.smithi071.stdout: "supported_features": [ 2023-05-24T21:19:15.758 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:19:15.758 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:19:15.758 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:19:15.758 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:19:15.759 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "ee18cd86-4684-4b3d-8de6-0075d3d925f1", 2023-05-24T21:19:15.759 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "b1", 2023-05-24T21:19:15.760 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:19:15.760 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi157.front.sepia.ceph.com:8000" 2023-05-24T21:19:15.760 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:19:15.760 INFO:teuthology.orchestra.run.smithi071.stdout: "log_meta": false, 2023-05-24T21:19:15.760 INFO:teuthology.orchestra.run.smithi071.stdout: "log_data": true, 2023-05-24T21:19:15.760 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_index_max_shards": 11, 2023-05-24T21:19:15.761 INFO:teuthology.orchestra.run.smithi071.stdout: "read_only": false, 2023-05-24T21:19:15.761 INFO:teuthology.orchestra.run.smithi071.stdout: "tier_type": "", 2023-05-24T21:19:15.761 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from_all": true, 2023-05-24T21:19:15.761 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from": [], 2023-05-24T21:19:15.761 INFO:teuthology.orchestra.run.smithi071.stdout: "redirect_zone": "", 2023-05-24T21:19:15.761 INFO:teuthology.orchestra.run.smithi071.stdout: "supported_features": [ 2023-05-24T21:19:15.761 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:19:15.762 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:19:15.762 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:19:15.762 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:19:15.762 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_targets": [ 2023-05-24T21:19:15.762 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:19:15.762 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "default-placement", 2023-05-24T21:19:15.762 INFO:teuthology.orchestra.run.smithi071.stdout: "tags": [], 2023-05-24T21:19:15.763 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": [ 2023-05-24T21:19:15.763 INFO:teuthology.orchestra.run.smithi071.stdout: "STANDARD" 2023-05-24T21:19:15.763 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:19:15.763 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:19:15.763 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:19:15.763 INFO:teuthology.orchestra.run.smithi071.stdout: "default_placement": "default-placement", 2023-05-24T21:19:15.763 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T21:19:15.764 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_policy": { 2023-05-24T21:19:15.764 INFO:teuthology.orchestra.run.smithi071.stdout: "groups": [] 2023-05-24T21:19:15.764 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:19:15.764 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled_features": [ 2023-05-24T21:19:15.764 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:19:15.764 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:19:15.765 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:19:15.765 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:19:15.765 INFO:teuthology.orchestra.run.smithi071.stdout: "short_zone_ids": [ 2023-05-24T21:19:15.765 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:19:15.765 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T21:19:15.765 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 3764661814 2023-05-24T21:19:15.765 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:19:15.766 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:19:15.766 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "5af1588c-f9b9-4687-a040-ce6e0cc808d9", 2023-05-24T21:19:15.766 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 2266179370 2023-05-24T21:19:15.766 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:19:15.766 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:19:15.766 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "7b8c3b69-c8ed-4621-b2ef-b89d3bbb7973", 2023-05-24T21:19:15.766 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 1540450555 2023-05-24T21:19:15.767 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:19:15.767 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:19:15.767 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "ee18cd86-4684-4b3d-8de6-0075d3d925f1", 2023-05-24T21:19:15.767 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 489875403 2023-05-24T21:19:15.767 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:19:15.767 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:19:15.767 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:19:15.768 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zonegroup": "a", 2023-05-24T21:19:15.768 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zone": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T21:19:15.768 INFO:teuthology.orchestra.run.smithi071.stdout: "period_config": { 2023-05-24T21:19:15.768 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_quota": { 2023-05-24T21:19:15.768 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false, 2023-05-24T21:19:15.768 INFO:teuthology.orchestra.run.smithi071.stdout: "check_on_raw": false, 2023-05-24T21:19:15.768 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size": -1, 2023-05-24T21:19:15.769 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size_kb": 0, 2023-05-24T21:19:15.769 INFO:teuthology.orchestra.run.smithi071.stdout: "max_objects": -1 2023-05-24T21:19:15.769 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:19:15.769 INFO:teuthology.orchestra.run.smithi071.stdout: "user_quota": { 2023-05-24T21:19:15.769 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false, 2023-05-24T21:19:15.769 INFO:teuthology.orchestra.run.smithi071.stdout: "check_on_raw": false, 2023-05-24T21:19:15.769 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size": -1, 2023-05-24T21:19:15.770 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size_kb": 0, 2023-05-24T21:19:15.770 INFO:teuthology.orchestra.run.smithi071.stdout: "max_objects": -1 2023-05-24T21:19:15.770 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:19:15.770 INFO:teuthology.orchestra.run.smithi071.stdout: "user_ratelimit": { 2023-05-24T21:19:15.770 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_ops": 0, 2023-05-24T21:19:15.770 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_ops": 0, 2023-05-24T21:19:15.770 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_bytes": 0, 2023-05-24T21:19:15.771 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_bytes": 0, 2023-05-24T21:19:15.771 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false 2023-05-24T21:19:15.771 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:19:15.771 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_ratelimit": { 2023-05-24T21:19:15.771 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_ops": 0, 2023-05-24T21:19:15.771 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_ops": 0, 2023-05-24T21:19:15.771 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_bytes": 0, 2023-05-24T21:19:15.772 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_bytes": 0, 2023-05-24T21:19:15.772 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false 2023-05-24T21:19:15.772 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:19:15.772 INFO:teuthology.orchestra.run.smithi071.stdout: "anonymous_ratelimit": { 2023-05-24T21:19:15.772 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_ops": 0, 2023-05-24T21:19:15.772 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_ops": 0, 2023-05-24T21:19:15.773 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_bytes": 0, 2023-05-24T21:19:15.773 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_bytes": 0, 2023-05-24T21:19:15.773 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false 2023-05-24T21:19:15.773 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:19:15.773 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:19:15.773 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T21:19:15.773 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_name": "test-realm", 2023-05-24T21:19:15.774 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:19:15.774 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:19:15.774 INFO:rgw_multi.tests:Set redirect_zone zone a2 to "" 2023-05-24T21:19:16.108 INFO:tasks.rgw.c1.client.0.smithi071.stdout:2023-05-24T21:19:16.101+0000 7f4a397b6700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-05-24T21:20:15.776 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_zone_redirect ... ok 2023-05-24T21:20:15.777 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:20:15.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:20:15.778 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:20:15.834 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:20:15.835 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:20:15.875 INFO:teuthology.orchestra.run.smithi071.stderr:No --period given, using current period=5c04936b-70ea-4415-b4eb-3aad3dc612e5 2023-05-24T21:20:15.890 INFO:teuthology.orchestra.run.smithi071.stdout:[ 2023-05-24T21:20:15.891 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:15.891 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962972.907627_296.1", 2023-05-24T21:20:15.891 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:16:12.907627Z" 2023-05-24T21:20:15.891 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:15.891 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:15.891 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:20:15.892 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:06:32.333844Z" 2023-05-24T21:20:15.892 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:15.892 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:15.892 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:20:15.892 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:20:15.892 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:15.892 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:15.893 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:20:15.893 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:20:15.893 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:15.893 INFO:teuthology.orchestra.run.smithi071.stdout:] 2023-05-24T21:20:15.894 DEBUG:rgw_multi.tests:master meta markers={0: '1_1684962972.907627_296.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:20:15.894 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-24T21:20:15.895 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:20:15.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:20:15.896 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:20:15.946 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:20:15.946 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:20:15.986 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:20:15.980+0000 7f5984d68780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:20:15.987 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:20:15.981+0000 7f5984d68780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:20:16.007 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:20:16.008 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:20:16.008 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:20:16.008 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:20:16.008 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:20:16.008 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:20:16.008 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:20:16.009 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:16.009 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:20:16.009 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:16.009 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:20:16.009 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:20:16.009 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:20:16.009 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962972.907627_296.1", 2023-05-24T21:20:16.010 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:20:16.010 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:20:16.010 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:20:16.010 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:16:12.907627Z", 2023-05-24T21:20:16.010 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:20:16.010 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:16.010 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:16.011 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:16.011 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:20:16.011 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:20:16.011 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:20:16.011 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:20:16.011 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:20:16.011 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:20:16.012 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:20:16.012 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:20:16.012 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:20:16.012 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:16.012 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:16.012 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:16.013 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:20:16.013 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:20:16.013 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:20:16.013 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:20:16.013 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:20:16.013 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:20:16.013 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:20:16.014 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:20:16.014 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:20:16.014 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:16.014 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:16.014 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:16.014 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:20:16.014 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:20:16.015 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:20:16.015 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:20:16.015 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:20:16.015 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:20:16.015 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:20:16.015 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:20:16.016 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:20:16.016 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:16.016 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:16.016 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:16.016 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:16.016 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:20:16.016 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:20:16.017 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:20:16.017 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:16.017 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:20:15Z" 2023-05-24T21:20:16.017 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:20:16.018 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962972.907627_296.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:16:12.907627Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:20:15Z" } 2023-05-24T21:20:16.018 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962972.907627_296.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:16:12.907627Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:20:16.019 DEBUG:rgw_multi.tests:log_status={0: '1_1684962972.907627_296.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:20:16.019 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962972.907627_296.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:20:16.019 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-24T21:20:16.020 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', 'Z6KKVUYPP0489K03', '--secret', 'EX5IDzHTe9AVqa8b1PSD8B5Y1CKI3jBg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:20:16.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', 'Z6KKVUYPP0489K03', '--secret', 'EX5IDzHTe9AVqa8b1PSD8B5Y1CKI3jBg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:20:16.021 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup a --rgw-realm test-realm --access-key Z6KKVUYPP0489K03 --secret EX5IDzHTe9AVqa8b1PSD8B5Y1CKI3jBg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-24T21:20:16.071 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:20:16.071 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:20:16.087 INFO:teuthology.orchestra.run.smithi071.stderr:NOTICE: set zone's realm_id=4a7754f9-b4ac-4d00-9827-3910f2de91d2 2023-05-24T21:20:16.105 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:20:16.105 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "d90c9e8e-14e5-48a3-918b-d942762d7094", 2023-05-24T21:20:16.105 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "remove", 2023-05-24T21:20:16.106 INFO:teuthology.orchestra.run.smithi071.stdout: "domain_root": "remove.rgw.meta:root", 2023-05-24T21:20:16.106 INFO:teuthology.orchestra.run.smithi071.stdout: "control_pool": "remove.rgw.control", 2023-05-24T21:20:16.106 INFO:teuthology.orchestra.run.smithi071.stdout: "gc_pool": "remove.rgw.log:gc", 2023-05-24T21:20:16.106 INFO:teuthology.orchestra.run.smithi071.stdout: "lc_pool": "remove.rgw.log:lc", 2023-05-24T21:20:16.106 INFO:teuthology.orchestra.run.smithi071.stdout: "log_pool": "remove.rgw.log", 2023-05-24T21:20:16.106 INFO:teuthology.orchestra.run.smithi071.stdout: "intent_log_pool": "remove.rgw.log:intent", 2023-05-24T21:20:16.106 INFO:teuthology.orchestra.run.smithi071.stdout: "usage_log_pool": "remove.rgw.log:usage", 2023-05-24T21:20:16.107 INFO:teuthology.orchestra.run.smithi071.stdout: "roles_pool": "remove.rgw.meta:roles", 2023-05-24T21:20:16.107 INFO:teuthology.orchestra.run.smithi071.stdout: "reshard_pool": "remove.rgw.log:reshard", 2023-05-24T21:20:16.107 INFO:teuthology.orchestra.run.smithi071.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2023-05-24T21:20:16.107 INFO:teuthology.orchestra.run.smithi071.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2023-05-24T21:20:16.107 INFO:teuthology.orchestra.run.smithi071.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2023-05-24T21:20:16.107 INFO:teuthology.orchestra.run.smithi071.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2023-05-24T21:20:16.107 INFO:teuthology.orchestra.run.smithi071.stdout: "otp_pool": "remove.rgw.otp", 2023-05-24T21:20:16.108 INFO:teuthology.orchestra.run.smithi071.stdout: "system_key": { 2023-05-24T21:20:16.108 INFO:teuthology.orchestra.run.smithi071.stdout: "access_key": "Z6KKVUYPP0489K03", 2023-05-24T21:20:16.108 INFO:teuthology.orchestra.run.smithi071.stdout: "secret_key": "EX5IDzHTe9AVqa8b1PSD8B5Y1CKI3jBg" 2023-05-24T21:20:16.108 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:16.108 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_pools": [ 2023-05-24T21:20:16.109 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:16.109 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "default-placement", 2023-05-24T21:20:16.109 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:20:16.109 INFO:teuthology.orchestra.run.smithi071.stdout: "index_pool": "remove.rgw.buckets.index", 2023-05-24T21:20:16.110 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": { 2023-05-24T21:20:16.110 INFO:teuthology.orchestra.run.smithi071.stdout: "STANDARD": { 2023-05-24T21:20:16.110 INFO:teuthology.orchestra.run.smithi071.stdout: "data_pool": "remove.rgw.buckets.data" 2023-05-24T21:20:16.110 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:16.110 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:16.110 INFO:teuthology.orchestra.run.smithi071.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2023-05-24T21:20:16.111 INFO:teuthology.orchestra.run.smithi071.stdout: "index_type": 0, 2023-05-24T21:20:16.111 INFO:teuthology.orchestra.run.smithi071.stdout: "inline_data": true 2023-05-24T21:20:16.111 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:16.111 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:16.111 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:16.111 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T21:20:16.111 INFO:teuthology.orchestra.run.smithi071.stdout: "notif_pool": "remove.rgw.log:notif" 2023-05-24T21:20:16.112 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:20:16.113 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:20:16.113 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:20:16.114 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone remove --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-24T21:20:16.162 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:20:16.162 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:20:16.174 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:20:16.168+0000 7f5896cf4780 0 period (5c04936b-70ea-4415-b4eb-3aad3dc612e5 does not have zone d90c9e8e-14e5-48a3-918b-d942762d7094 configured 2023-05-24T21:20:23.062 INFO:teuthology.orchestra.run.smithi071.stderr:Sending period to new master zone 299a3349-fae0-459e-8632-431fc36beb1b 2023-05-24T21:20:23.102 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:20:23.102 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:20:23.103 INFO:teuthology.orchestra.run.smithi071.stdout: "epoch": 9, 2023-05-24T21:20:23.103 INFO:teuthology.orchestra.run.smithi071.stdout: "predecessor_uuid": "1cea3a6e-eb52-480e-95b7-3f6e43317e60", 2023-05-24T21:20:23.103 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": [], 2023-05-24T21:20:23.103 INFO:teuthology.orchestra.run.smithi071.stdout: "period_map": { 2023-05-24T21:20:23.103 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:20:23.103 INFO:teuthology.orchestra.run.smithi071.stdout: "zonegroups": [ 2023-05-24T21:20:23.104 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:23.104 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "a", 2023-05-24T21:20:23.104 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a", 2023-05-24T21:20:23.104 INFO:teuthology.orchestra.run.smithi071.stdout: "api_name": "", 2023-05-24T21:20:23.104 INFO:teuthology.orchestra.run.smithi071.stdout: "is_master": true, 2023-05-24T21:20:23.104 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:20:23.105 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-05-24T21:20:23.105 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:23.105 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames": [], 2023-05-24T21:20:23.105 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames_s3website": [], 2023-05-24T21:20:23.105 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zone": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T21:20:23.105 INFO:teuthology.orchestra.run.smithi071.stdout: "zones": [ 2023-05-24T21:20:23.105 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:23.106 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T21:20:23.106 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a1", 2023-05-24T21:20:23.106 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:20:23.106 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-05-24T21:20:23.106 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:23.106 INFO:teuthology.orchestra.run.smithi071.stdout: "log_meta": false, 2023-05-24T21:20:23.106 INFO:teuthology.orchestra.run.smithi071.stdout: "log_data": true, 2023-05-24T21:20:23.107 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_index_max_shards": 11, 2023-05-24T21:20:23.107 INFO:teuthology.orchestra.run.smithi071.stdout: "read_only": false, 2023-05-24T21:20:23.107 INFO:teuthology.orchestra.run.smithi071.stdout: "tier_type": "", 2023-05-24T21:20:23.107 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from_all": true, 2023-05-24T21:20:23.107 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from": [], 2023-05-24T21:20:23.107 INFO:teuthology.orchestra.run.smithi071.stdout: "redirect_zone": "", 2023-05-24T21:20:23.108 INFO:teuthology.orchestra.run.smithi071.stdout: "supported_features": [ 2023-05-24T21:20:23.108 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:20:23.108 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:23.108 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:23.108 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:23.108 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "5af1588c-f9b9-4687-a040-ce6e0cc808d9", 2023-05-24T21:20:23.108 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a2", 2023-05-24T21:20:23.109 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:20:23.109 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi071.front.sepia.ceph.com:8001" 2023-05-24T21:20:23.109 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:23.109 INFO:teuthology.orchestra.run.smithi071.stdout: "log_meta": false, 2023-05-24T21:20:23.109 INFO:teuthology.orchestra.run.smithi071.stdout: "log_data": true, 2023-05-24T21:20:23.109 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_index_max_shards": 11, 2023-05-24T21:20:23.109 INFO:teuthology.orchestra.run.smithi071.stdout: "read_only": false, 2023-05-24T21:20:23.110 INFO:teuthology.orchestra.run.smithi071.stdout: "tier_type": "", 2023-05-24T21:20:23.110 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from_all": true, 2023-05-24T21:20:23.110 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from": [], 2023-05-24T21:20:23.110 INFO:teuthology.orchestra.run.smithi071.stdout: "redirect_zone": "", 2023-05-24T21:20:23.110 INFO:teuthology.orchestra.run.smithi071.stdout: "supported_features": [ 2023-05-24T21:20:23.110 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:20:23.110 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:23.111 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:23.111 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:23.111 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "d90c9e8e-14e5-48a3-918b-d942762d7094", 2023-05-24T21:20:23.111 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "remove", 2023-05-24T21:20:23.111 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [], 2023-05-24T21:20:23.111 INFO:teuthology.orchestra.run.smithi071.stdout: "log_meta": false, 2023-05-24T21:20:23.111 INFO:teuthology.orchestra.run.smithi071.stdout: "log_data": true, 2023-05-24T21:20:23.112 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_index_max_shards": 11, 2023-05-24T21:20:23.112 INFO:teuthology.orchestra.run.smithi071.stdout: "read_only": false, 2023-05-24T21:20:23.112 INFO:teuthology.orchestra.run.smithi071.stdout: "tier_type": "", 2023-05-24T21:20:23.112 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from_all": true, 2023-05-24T21:20:23.112 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from": [], 2023-05-24T21:20:23.112 INFO:teuthology.orchestra.run.smithi071.stdout: "redirect_zone": "", 2023-05-24T21:20:23.112 INFO:teuthology.orchestra.run.smithi071.stdout: "supported_features": [ 2023-05-24T21:20:23.113 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:20:23.113 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:23.113 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:23.113 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:23.113 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_targets": [ 2023-05-24T21:20:23.113 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:23.113 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "default-placement", 2023-05-24T21:20:23.114 INFO:teuthology.orchestra.run.smithi071.stdout: "tags": [], 2023-05-24T21:20:23.114 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": [ 2023-05-24T21:20:23.114 INFO:teuthology.orchestra.run.smithi071.stdout: "STANDARD" 2023-05-24T21:20:23.114 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:23.114 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:23.114 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:23.114 INFO:teuthology.orchestra.run.smithi071.stdout: "default_placement": "default-placement", 2023-05-24T21:20:23.115 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T21:20:23.115 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_policy": { 2023-05-24T21:20:23.115 INFO:teuthology.orchestra.run.smithi071.stdout: "groups": [] 2023-05-24T21:20:23.115 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:23.115 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled_features": [ 2023-05-24T21:20:23.115 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:20:23.115 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:23.116 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:23.116 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:23.116 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "b", 2023-05-24T21:20:23.116 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "b", 2023-05-24T21:20:23.116 INFO:teuthology.orchestra.run.smithi071.stdout: "api_name": "", 2023-05-24T21:20:23.116 INFO:teuthology.orchestra.run.smithi071.stdout: "is_master": false, 2023-05-24T21:20:23.116 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:20:23.117 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi157.front.sepia.ceph.com:8000" 2023-05-24T21:20:23.117 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:23.117 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames": [], 2023-05-24T21:20:23.117 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames_s3website": [], 2023-05-24T21:20:23.117 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zone": "ee18cd86-4684-4b3d-8de6-0075d3d925f1", 2023-05-24T21:20:23.117 INFO:teuthology.orchestra.run.smithi071.stdout: "zones": [ 2023-05-24T21:20:23.117 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:23.118 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "7b8c3b69-c8ed-4621-b2ef-b89d3bbb7973", 2023-05-24T21:20:23.118 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "b2", 2023-05-24T21:20:23.127 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:20:23.128 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi157.front.sepia.ceph.com:8001" 2023-05-24T21:20:23.128 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:23.128 INFO:teuthology.orchestra.run.smithi071.stdout: "log_meta": false, 2023-05-24T21:20:23.128 INFO:teuthology.orchestra.run.smithi071.stdout: "log_data": true, 2023-05-24T21:20:23.128 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_index_max_shards": 11, 2023-05-24T21:20:23.128 INFO:teuthology.orchestra.run.smithi071.stdout: "read_only": false, 2023-05-24T21:20:23.128 INFO:teuthology.orchestra.run.smithi071.stdout: "tier_type": "", 2023-05-24T21:20:23.129 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from_all": true, 2023-05-24T21:20:23.129 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from": [], 2023-05-24T21:20:23.129 INFO:teuthology.orchestra.run.smithi071.stdout: "redirect_zone": "", 2023-05-24T21:20:23.129 INFO:teuthology.orchestra.run.smithi071.stdout: "supported_features": [ 2023-05-24T21:20:23.129 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:20:23.129 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:23.129 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:23.130 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:23.130 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "ee18cd86-4684-4b3d-8de6-0075d3d925f1", 2023-05-24T21:20:23.130 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "b1", 2023-05-24T21:20:23.130 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:20:23.130 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi157.front.sepia.ceph.com:8000" 2023-05-24T21:20:23.130 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:23.130 INFO:teuthology.orchestra.run.smithi071.stdout: "log_meta": false, 2023-05-24T21:20:23.131 INFO:teuthology.orchestra.run.smithi071.stdout: "log_data": true, 2023-05-24T21:20:23.131 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_index_max_shards": 11, 2023-05-24T21:20:23.131 INFO:teuthology.orchestra.run.smithi071.stdout: "read_only": false, 2023-05-24T21:20:23.131 INFO:teuthology.orchestra.run.smithi071.stdout: "tier_type": "", 2023-05-24T21:20:23.131 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from_all": true, 2023-05-24T21:20:23.131 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from": [], 2023-05-24T21:20:23.131 INFO:teuthology.orchestra.run.smithi071.stdout: "redirect_zone": "", 2023-05-24T21:20:23.132 INFO:teuthology.orchestra.run.smithi071.stdout: "supported_features": [ 2023-05-24T21:20:23.132 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:20:23.132 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:23.132 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:23.132 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:23.132 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_targets": [ 2023-05-24T21:20:23.132 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:23.133 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "default-placement", 2023-05-24T21:20:23.133 INFO:teuthology.orchestra.run.smithi071.stdout: "tags": [], 2023-05-24T21:20:23.133 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": [ 2023-05-24T21:20:23.133 INFO:teuthology.orchestra.run.smithi071.stdout: "STANDARD" 2023-05-24T21:20:23.133 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:23.133 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:23.133 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:23.134 INFO:teuthology.orchestra.run.smithi071.stdout: "default_placement": "default-placement", 2023-05-24T21:20:23.134 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T21:20:23.134 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_policy": { 2023-05-24T21:20:23.134 INFO:teuthology.orchestra.run.smithi071.stdout: "groups": [] 2023-05-24T21:20:23.134 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:23.134 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled_features": [ 2023-05-24T21:20:23.134 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:20:23.135 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:23.135 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:23.135 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:23.135 INFO:teuthology.orchestra.run.smithi071.stdout: "short_zone_ids": [ 2023-05-24T21:20:23.135 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:23.135 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T21:20:23.135 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 3764661814 2023-05-24T21:20:23.136 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:23.136 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:23.136 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "5af1588c-f9b9-4687-a040-ce6e0cc808d9", 2023-05-24T21:20:23.136 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 2266179370 2023-05-24T21:20:23.136 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:23.136 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:23.136 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "7b8c3b69-c8ed-4621-b2ef-b89d3bbb7973", 2023-05-24T21:20:23.137 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 1540450555 2023-05-24T21:20:23.137 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:23.137 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:23.137 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "d90c9e8e-14e5-48a3-918b-d942762d7094", 2023-05-24T21:20:23.137 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 2068985338 2023-05-24T21:20:23.137 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:23.137 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:23.138 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "ee18cd86-4684-4b3d-8de6-0075d3d925f1", 2023-05-24T21:20:23.138 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 489875403 2023-05-24T21:20:23.138 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:23.138 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:23.138 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:23.138 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zonegroup": "a", 2023-05-24T21:20:23.138 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zone": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T21:20:23.139 INFO:teuthology.orchestra.run.smithi071.stdout: "period_config": { 2023-05-24T21:20:23.139 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_quota": { 2023-05-24T21:20:23.139 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false, 2023-05-24T21:20:23.139 INFO:teuthology.orchestra.run.smithi071.stdout: "check_on_raw": false, 2023-05-24T21:20:23.139 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size": -1, 2023-05-24T21:20:23.139 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size_kb": 0, 2023-05-24T21:20:23.139 INFO:teuthology.orchestra.run.smithi071.stdout: "max_objects": -1 2023-05-24T21:20:23.140 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:23.140 INFO:teuthology.orchestra.run.smithi071.stdout: "user_quota": { 2023-05-24T21:20:23.140 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false, 2023-05-24T21:20:23.140 INFO:teuthology.orchestra.run.smithi071.stdout: "check_on_raw": false, 2023-05-24T21:20:23.140 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size": -1, 2023-05-24T21:20:23.140 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size_kb": 0, 2023-05-24T21:20:23.140 INFO:teuthology.orchestra.run.smithi071.stdout: "max_objects": -1 2023-05-24T21:20:23.141 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:23.141 INFO:teuthology.orchestra.run.smithi071.stdout: "user_ratelimit": { 2023-05-24T21:20:23.141 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_ops": 0, 2023-05-24T21:20:23.141 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_ops": 0, 2023-05-24T21:20:23.141 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_bytes": 0, 2023-05-24T21:20:23.141 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_bytes": 0, 2023-05-24T21:20:23.141 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false 2023-05-24T21:20:23.142 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:23.142 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_ratelimit": { 2023-05-24T21:20:23.142 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_ops": 0, 2023-05-24T21:20:23.142 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_ops": 0, 2023-05-24T21:20:23.142 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_bytes": 0, 2023-05-24T21:20:23.142 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_bytes": 0, 2023-05-24T21:20:23.142 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false 2023-05-24T21:20:23.143 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:23.143 INFO:teuthology.orchestra.run.smithi071.stdout: "anonymous_ratelimit": { 2023-05-24T21:20:23.143 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_ops": 0, 2023-05-24T21:20:23.143 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_ops": 0, 2023-05-24T21:20:23.143 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_bytes": 0, 2023-05-24T21:20:23.143 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_bytes": 0, 2023-05-24T21:20:23.143 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false 2023-05-24T21:20:23.144 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:23.144 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:23.144 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T21:20:23.144 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_name": "test-realm", 2023-05-24T21:20:23.144 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:20:23.144 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:20:23.145 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:20:23.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:20:23.146 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup a --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-24T21:20:23.200 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:20:23.201 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:20:23.223 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:20:23.224 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "a", 2023-05-24T21:20:23.224 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a", 2023-05-24T21:20:23.224 INFO:teuthology.orchestra.run.smithi071.stdout: "api_name": "", 2023-05-24T21:20:23.224 INFO:teuthology.orchestra.run.smithi071.stdout: "is_master": true, 2023-05-24T21:20:23.224 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:20:23.224 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-05-24T21:20:23.225 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:23.225 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames": [], 2023-05-24T21:20:23.225 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames_s3website": [], 2023-05-24T21:20:23.225 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zone": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T21:20:23.225 INFO:teuthology.orchestra.run.smithi071.stdout: "zones": [ 2023-05-24T21:20:23.225 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:23.225 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T21:20:23.226 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a1", 2023-05-24T21:20:23.226 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:20:23.226 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-05-24T21:20:23.226 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:23.226 INFO:teuthology.orchestra.run.smithi071.stdout: "log_meta": false, 2023-05-24T21:20:23.226 INFO:teuthology.orchestra.run.smithi071.stdout: "log_data": true, 2023-05-24T21:20:23.227 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_index_max_shards": 11, 2023-05-24T21:20:23.227 INFO:teuthology.orchestra.run.smithi071.stdout: "read_only": false, 2023-05-24T21:20:23.227 INFO:teuthology.orchestra.run.smithi071.stdout: "tier_type": "", 2023-05-24T21:20:23.227 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from_all": true, 2023-05-24T21:20:23.227 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from": [], 2023-05-24T21:20:23.227 INFO:teuthology.orchestra.run.smithi071.stdout: "redirect_zone": "", 2023-05-24T21:20:23.227 INFO:teuthology.orchestra.run.smithi071.stdout: "supported_features": [ 2023-05-24T21:20:23.228 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:20:23.228 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:23.228 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:23.228 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:23.228 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "5af1588c-f9b9-4687-a040-ce6e0cc808d9", 2023-05-24T21:20:23.228 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a2", 2023-05-24T21:20:23.228 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:20:23.229 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi071.front.sepia.ceph.com:8001" 2023-05-24T21:20:23.229 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:23.229 INFO:teuthology.orchestra.run.smithi071.stdout: "log_meta": false, 2023-05-24T21:20:23.229 INFO:teuthology.orchestra.run.smithi071.stdout: "log_data": true, 2023-05-24T21:20:23.229 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_index_max_shards": 11, 2023-05-24T21:20:23.229 INFO:teuthology.orchestra.run.smithi071.stdout: "read_only": false, 2023-05-24T21:20:23.229 INFO:teuthology.orchestra.run.smithi071.stdout: "tier_type": "", 2023-05-24T21:20:23.230 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from_all": true, 2023-05-24T21:20:23.230 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from": [], 2023-05-24T21:20:23.230 INFO:teuthology.orchestra.run.smithi071.stdout: "redirect_zone": "", 2023-05-24T21:20:23.230 INFO:teuthology.orchestra.run.smithi071.stdout: "supported_features": [ 2023-05-24T21:20:23.230 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:20:23.230 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:23.231 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:23.231 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:23.231 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_targets": [ 2023-05-24T21:20:23.231 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:23.231 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "default-placement", 2023-05-24T21:20:23.231 INFO:teuthology.orchestra.run.smithi071.stdout: "tags": [], 2023-05-24T21:20:23.231 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": [ 2023-05-24T21:20:23.231 INFO:teuthology.orchestra.run.smithi071.stdout: "STANDARD" 2023-05-24T21:20:23.232 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:23.232 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:23.232 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:23.232 INFO:teuthology.orchestra.run.smithi071.stdout: "default_placement": "default-placement", 2023-05-24T21:20:23.232 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T21:20:23.232 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_policy": { 2023-05-24T21:20:23.233 INFO:teuthology.orchestra.run.smithi071.stdout: "groups": [] 2023-05-24T21:20:23.233 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:23.233 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled_features": [ 2023-05-24T21:20:23.233 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:20:23.233 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:23.233 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:20:23.234 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:20:23.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:20:23.235 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup a --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-24T21:20:23.304 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:20:23.305 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:20:23.323 INFO:teuthology.orchestra.run.smithi071.stderr:zone name remove not found in zonegroup a 2023-05-24T21:20:23.329 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-05-24T21:20:23.329 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:20:23.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:20:23.330 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-24T21:20:23.388 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:20:23.388 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:20:23.439 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:20:23.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:20:23.440 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-24T21:20:23.496 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:20:23.497 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:20:23.516 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:20:23.510+0000 7f566c1c7780 0 WARNING: can't generate connection for zone d90c9e8e-14e5-48a3-918b-d942762d7094 id remove: no endpoints defined 2023-05-24T21:20:23.609 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:20:23.609 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:20:23.610 INFO:teuthology.orchestra.run.smithi071.stdout: "epoch": 10, 2023-05-24T21:20:23.610 INFO:teuthology.orchestra.run.smithi071.stdout: "predecessor_uuid": "1cea3a6e-eb52-480e-95b7-3f6e43317e60", 2023-05-24T21:20:23.610 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": [], 2023-05-24T21:20:23.610 INFO:teuthology.orchestra.run.smithi071.stdout: "period_map": { 2023-05-24T21:20:23.610 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:20:23.610 INFO:teuthology.orchestra.run.smithi071.stdout: "zonegroups": [ 2023-05-24T21:20:23.611 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:23.611 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "a", 2023-05-24T21:20:23.611 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a", 2023-05-24T21:20:23.611 INFO:teuthology.orchestra.run.smithi071.stdout: "api_name": "", 2023-05-24T21:20:23.611 INFO:teuthology.orchestra.run.smithi071.stdout: "is_master": true, 2023-05-24T21:20:23.611 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:20:23.612 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-05-24T21:20:23.612 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:23.612 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames": [], 2023-05-24T21:20:23.612 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames_s3website": [], 2023-05-24T21:20:23.612 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zone": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T21:20:23.612 INFO:teuthology.orchestra.run.smithi071.stdout: "zones": [ 2023-05-24T21:20:23.613 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:23.613 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T21:20:23.613 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a1", 2023-05-24T21:20:23.613 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:20:23.613 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-05-24T21:20:23.613 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:23.614 INFO:teuthology.orchestra.run.smithi071.stdout: "log_meta": false, 2023-05-24T21:20:23.614 INFO:teuthology.orchestra.run.smithi071.stdout: "log_data": true, 2023-05-24T21:20:23.614 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_index_max_shards": 11, 2023-05-24T21:20:23.614 INFO:teuthology.orchestra.run.smithi071.stdout: "read_only": false, 2023-05-24T21:20:23.614 INFO:teuthology.orchestra.run.smithi071.stdout: "tier_type": "", 2023-05-24T21:20:23.614 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from_all": true, 2023-05-24T21:20:23.614 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from": [], 2023-05-24T21:20:23.615 INFO:teuthology.orchestra.run.smithi071.stdout: "redirect_zone": "", 2023-05-24T21:20:23.615 INFO:teuthology.orchestra.run.smithi071.stdout: "supported_features": [ 2023-05-24T21:20:23.615 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:20:23.615 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:23.615 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:23.615 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:23.616 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "5af1588c-f9b9-4687-a040-ce6e0cc808d9", 2023-05-24T21:20:23.616 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a2", 2023-05-24T21:20:23.616 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:20:23.616 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi071.front.sepia.ceph.com:8001" 2023-05-24T21:20:23.616 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:23.616 INFO:teuthology.orchestra.run.smithi071.stdout: "log_meta": false, 2023-05-24T21:20:23.616 INFO:teuthology.orchestra.run.smithi071.stdout: "log_data": true, 2023-05-24T21:20:23.617 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_index_max_shards": 11, 2023-05-24T21:20:23.617 INFO:teuthology.orchestra.run.smithi071.stdout: "read_only": false, 2023-05-24T21:20:23.617 INFO:teuthology.orchestra.run.smithi071.stdout: "tier_type": "", 2023-05-24T21:20:23.617 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from_all": true, 2023-05-24T21:20:23.617 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from": [], 2023-05-24T21:20:23.617 INFO:teuthology.orchestra.run.smithi071.stdout: "redirect_zone": "", 2023-05-24T21:20:23.617 INFO:teuthology.orchestra.run.smithi071.stdout: "supported_features": [ 2023-05-24T21:20:23.618 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:20:23.618 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:23.618 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:23.618 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:23.618 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_targets": [ 2023-05-24T21:20:23.618 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:23.619 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "default-placement", 2023-05-24T21:20:23.619 INFO:teuthology.orchestra.run.smithi071.stdout: "tags": [], 2023-05-24T21:20:23.619 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": [ 2023-05-24T21:20:23.619 INFO:teuthology.orchestra.run.smithi071.stdout: "STANDARD" 2023-05-24T21:20:23.619 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:23.619 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:23.619 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:23.620 INFO:teuthology.orchestra.run.smithi071.stdout: "default_placement": "default-placement", 2023-05-24T21:20:23.620 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T21:20:23.620 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_policy": { 2023-05-24T21:20:23.620 INFO:teuthology.orchestra.run.smithi071.stdout: "groups": [] 2023-05-24T21:20:23.620 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:23.620 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled_features": [ 2023-05-24T21:20:23.620 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:20:23.621 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:23.621 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:23.621 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:23.621 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "b", 2023-05-24T21:20:23.621 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "b", 2023-05-24T21:20:23.621 INFO:teuthology.orchestra.run.smithi071.stdout: "api_name": "", 2023-05-24T21:20:23.622 INFO:teuthology.orchestra.run.smithi071.stdout: "is_master": false, 2023-05-24T21:20:23.622 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:20:23.622 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi157.front.sepia.ceph.com:8000" 2023-05-24T21:20:23.622 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:23.622 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames": [], 2023-05-24T21:20:23.622 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames_s3website": [], 2023-05-24T21:20:23.623 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zone": "ee18cd86-4684-4b3d-8de6-0075d3d925f1", 2023-05-24T21:20:23.623 INFO:teuthology.orchestra.run.smithi071.stdout: "zones": [ 2023-05-24T21:20:23.623 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:23.623 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "7b8c3b69-c8ed-4621-b2ef-b89d3bbb7973", 2023-05-24T21:20:23.623 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "b2", 2023-05-24T21:20:23.623 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:20:23.623 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi157.front.sepia.ceph.com:8001" 2023-05-24T21:20:23.624 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:23.624 INFO:teuthology.orchestra.run.smithi071.stdout: "log_meta": false, 2023-05-24T21:20:23.624 INFO:teuthology.orchestra.run.smithi071.stdout: "log_data": true, 2023-05-24T21:20:23.624 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_index_max_shards": 11, 2023-05-24T21:20:23.624 INFO:teuthology.orchestra.run.smithi071.stdout: "read_only": false, 2023-05-24T21:20:23.624 INFO:teuthology.orchestra.run.smithi071.stdout: "tier_type": "", 2023-05-24T21:20:23.625 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from_all": true, 2023-05-24T21:20:23.625 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from": [], 2023-05-24T21:20:23.625 INFO:teuthology.orchestra.run.smithi071.stdout: "redirect_zone": "", 2023-05-24T21:20:23.625 INFO:teuthology.orchestra.run.smithi071.stdout: "supported_features": [ 2023-05-24T21:20:23.625 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:20:23.625 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:23.625 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:23.626 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:23.651 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "ee18cd86-4684-4b3d-8de6-0075d3d925f1", 2023-05-24T21:20:23.651 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "b1", 2023-05-24T21:20:23.651 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:20:23.651 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi157.front.sepia.ceph.com:8000" 2023-05-24T21:20:23.651 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:23.651 INFO:teuthology.orchestra.run.smithi071.stdout: "log_meta": false, 2023-05-24T21:20:23.652 INFO:teuthology.orchestra.run.smithi071.stdout: "log_data": true, 2023-05-24T21:20:23.652 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_index_max_shards": 11, 2023-05-24T21:20:23.652 INFO:teuthology.orchestra.run.smithi071.stdout: "read_only": false, 2023-05-24T21:20:23.652 INFO:teuthology.orchestra.run.smithi071.stdout: "tier_type": "", 2023-05-24T21:20:23.652 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from_all": true, 2023-05-24T21:20:23.652 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from": [], 2023-05-24T21:20:23.653 INFO:teuthology.orchestra.run.smithi071.stdout: "redirect_zone": "", 2023-05-24T21:20:23.653 INFO:teuthology.orchestra.run.smithi071.stdout: "supported_features": [ 2023-05-24T21:20:23.653 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:20:23.653 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:23.653 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:23.653 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:23.653 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_targets": [ 2023-05-24T21:20:23.654 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:23.654 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "default-placement", 2023-05-24T21:20:23.654 INFO:teuthology.orchestra.run.smithi071.stdout: "tags": [], 2023-05-24T21:20:23.654 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": [ 2023-05-24T21:20:23.654 INFO:teuthology.orchestra.run.smithi071.stdout: "STANDARD" 2023-05-24T21:20:23.654 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:23.654 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:23.655 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:23.655 INFO:teuthology.orchestra.run.smithi071.stdout: "default_placement": "default-placement", 2023-05-24T21:20:23.655 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T21:20:23.655 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_policy": { 2023-05-24T21:20:23.655 INFO:teuthology.orchestra.run.smithi071.stdout: "groups": [] 2023-05-24T21:20:23.655 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:23.656 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled_features": [ 2023-05-24T21:20:23.656 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:20:23.656 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:23.656 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:23.656 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:23.656 INFO:teuthology.orchestra.run.smithi071.stdout: "short_zone_ids": [ 2023-05-24T21:20:23.656 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:23.657 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T21:20:23.657 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 3764661814 2023-05-24T21:20:23.657 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:23.657 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:23.657 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "5af1588c-f9b9-4687-a040-ce6e0cc808d9", 2023-05-24T21:20:23.657 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 2266179370 2023-05-24T21:20:23.657 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:23.658 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:23.658 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "7b8c3b69-c8ed-4621-b2ef-b89d3bbb7973", 2023-05-24T21:20:23.658 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 1540450555 2023-05-24T21:20:23.658 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:23.658 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:23.658 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "ee18cd86-4684-4b3d-8de6-0075d3d925f1", 2023-05-24T21:20:23.659 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 489875403 2023-05-24T21:20:23.659 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:23.659 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:23.659 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:23.659 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zonegroup": "a", 2023-05-24T21:20:23.659 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zone": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T21:20:23.659 INFO:teuthology.orchestra.run.smithi071.stdout: "period_config": { 2023-05-24T21:20:23.660 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_quota": { 2023-05-24T21:20:23.660 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false, 2023-05-24T21:20:23.660 INFO:teuthology.orchestra.run.smithi071.stdout: "check_on_raw": false, 2023-05-24T21:20:23.660 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size": -1, 2023-05-24T21:20:23.660 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size_kb": 0, 2023-05-24T21:20:23.660 INFO:teuthology.orchestra.run.smithi071.stdout: "max_objects": -1 2023-05-24T21:20:23.661 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:23.661 INFO:teuthology.orchestra.run.smithi071.stdout: "user_quota": { 2023-05-24T21:20:23.661 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false, 2023-05-24T21:20:23.661 INFO:teuthology.orchestra.run.smithi071.stdout: "check_on_raw": false, 2023-05-24T21:20:23.661 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size": -1, 2023-05-24T21:20:23.661 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size_kb": 0, 2023-05-24T21:20:23.661 INFO:teuthology.orchestra.run.smithi071.stdout: "max_objects": -1 2023-05-24T21:20:23.662 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:23.662 INFO:teuthology.orchestra.run.smithi071.stdout: "user_ratelimit": { 2023-05-24T21:20:23.662 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_ops": 0, 2023-05-24T21:20:23.662 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_ops": 0, 2023-05-24T21:20:23.662 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_bytes": 0, 2023-05-24T21:20:23.662 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_bytes": 0, 2023-05-24T21:20:23.662 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false 2023-05-24T21:20:23.663 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:23.663 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_ratelimit": { 2023-05-24T21:20:23.663 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_ops": 0, 2023-05-24T21:20:23.663 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_ops": 0, 2023-05-24T21:20:23.663 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_bytes": 0, 2023-05-24T21:20:23.663 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_bytes": 0, 2023-05-24T21:20:23.663 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false 2023-05-24T21:20:23.664 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:23.664 INFO:teuthology.orchestra.run.smithi071.stdout: "anonymous_ratelimit": { 2023-05-24T21:20:23.664 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_ops": 0, 2023-05-24T21:20:23.664 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_ops": 0, 2023-05-24T21:20:23.664 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_bytes": 0, 2023-05-24T21:20:23.664 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_bytes": 0, 2023-05-24T21:20:23.665 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false 2023-05-24T21:20:23.665 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:23.665 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:23.665 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T21:20:23.665 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_name": "test-realm", 2023-05-24T21:20:23.665 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:20:23.665 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:20:23.666 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zonegroup_remove ... ok 2023-05-24T21:20:23.667 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:20:23.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:20:23.668 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-24T21:20:23.717 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:20:23.718 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:20:23.752 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:20:23.753 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "96f26f03-a59a-4e4d-9940-cdfce5ea6291", 2023-05-24T21:20:23.753 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "remove_zg", 2023-05-24T21:20:23.753 INFO:teuthology.orchestra.run.smithi071.stdout: "api_name": "remove_zg", 2023-05-24T21:20:23.753 INFO:teuthology.orchestra.run.smithi071.stdout: "is_master": false, 2023-05-24T21:20:23.753 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [], 2023-05-24T21:20:23.754 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames": [], 2023-05-24T21:20:23.754 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames_s3website": [], 2023-05-24T21:20:23.754 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zone": "", 2023-05-24T21:20:23.754 INFO:teuthology.orchestra.run.smithi071.stdout: "zones": [], 2023-05-24T21:20:23.754 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_targets": [ 2023-05-24T21:20:23.754 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:23.754 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "default-placement", 2023-05-24T21:20:23.755 INFO:teuthology.orchestra.run.smithi071.stdout: "tags": [], 2023-05-24T21:20:23.755 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": [] 2023-05-24T21:20:23.755 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:23.755 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:23.755 INFO:teuthology.orchestra.run.smithi071.stdout: "default_placement": "default-placement", 2023-05-24T21:20:23.755 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T21:20:23.755 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_policy": { 2023-05-24T21:20:23.756 INFO:teuthology.orchestra.run.smithi071.stdout: "groups": [] 2023-05-24T21:20:23.756 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:23.756 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled_features": [ 2023-05-24T21:20:23.756 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:20:23.756 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:23.756 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:20:23.757 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:20:23.758 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:20:23.758 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-24T21:20:23.813 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:20:23.813 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:20:23.854 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:20:23.848+0000 7f4be638d780 0 NOTICE: promoted remove as new master_zone of zonegroup remove_zg 2023-05-24T21:20:23.862 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:20:23.862 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "89aff914-54fe-4a43-9d1c-22f67cc33bb4", 2023-05-24T21:20:23.862 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "remove", 2023-05-24T21:20:23.862 INFO:teuthology.orchestra.run.smithi071.stdout: "domain_root": "remove.rgw.meta:root", 2023-05-24T21:20:23.863 INFO:teuthology.orchestra.run.smithi071.stdout: "control_pool": "remove.rgw.control", 2023-05-24T21:20:23.863 INFO:teuthology.orchestra.run.smithi071.stdout: "gc_pool": "remove.rgw.log:gc", 2023-05-24T21:20:23.863 INFO:teuthology.orchestra.run.smithi071.stdout: "lc_pool": "remove.rgw.log:lc", 2023-05-24T21:20:23.863 INFO:teuthology.orchestra.run.smithi071.stdout: "log_pool": "remove.rgw.log", 2023-05-24T21:20:23.863 INFO:teuthology.orchestra.run.smithi071.stdout: "intent_log_pool": "remove.rgw.log:intent", 2023-05-24T21:20:23.863 INFO:teuthology.orchestra.run.smithi071.stdout: "usage_log_pool": "remove.rgw.log:usage", 2023-05-24T21:20:23.864 INFO:teuthology.orchestra.run.smithi071.stdout: "roles_pool": "remove.rgw.meta:roles", 2023-05-24T21:20:23.864 INFO:teuthology.orchestra.run.smithi071.stdout: "reshard_pool": "remove.rgw.log:reshard", 2023-05-24T21:20:23.864 INFO:teuthology.orchestra.run.smithi071.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2023-05-24T21:20:23.864 INFO:teuthology.orchestra.run.smithi071.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2023-05-24T21:20:23.864 INFO:teuthology.orchestra.run.smithi071.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2023-05-24T21:20:23.864 INFO:teuthology.orchestra.run.smithi071.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2023-05-24T21:20:23.864 INFO:teuthology.orchestra.run.smithi071.stdout: "otp_pool": "remove.rgw.otp", 2023-05-24T21:20:23.865 INFO:teuthology.orchestra.run.smithi071.stdout: "system_key": { 2023-05-24T21:20:23.865 INFO:teuthology.orchestra.run.smithi071.stdout: "access_key": "", 2023-05-24T21:20:23.865 INFO:teuthology.orchestra.run.smithi071.stdout: "secret_key": "" 2023-05-24T21:20:23.865 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:23.865 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_pools": [ 2023-05-24T21:20:23.865 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:23.865 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "default-placement", 2023-05-24T21:20:23.866 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:20:23.866 INFO:teuthology.orchestra.run.smithi071.stdout: "index_pool": "remove.rgw.buckets.index", 2023-05-24T21:20:23.866 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": { 2023-05-24T21:20:23.866 INFO:teuthology.orchestra.run.smithi071.stdout: "STANDARD": { 2023-05-24T21:20:23.866 INFO:teuthology.orchestra.run.smithi071.stdout: "data_pool": "remove.rgw.buckets.data" 2023-05-24T21:20:23.866 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:23.866 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:23.867 INFO:teuthology.orchestra.run.smithi071.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2023-05-24T21:20:23.867 INFO:teuthology.orchestra.run.smithi071.stdout: "index_type": 0, 2023-05-24T21:20:23.867 INFO:teuthology.orchestra.run.smithi071.stdout: "inline_data": true 2023-05-24T21:20:23.867 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:23.867 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:23.867 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:23.867 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T21:20:23.868 INFO:teuthology.orchestra.run.smithi071.stdout: "notif_pool": "remove.rgw.log:notif" 2023-05-24T21:20:23.868 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:20:23.869 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:20:23.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:20:23.870 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-24T21:20:23.925 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:20:23.925 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:20:24.050 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:20:24.051 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:20:24.051 INFO:teuthology.orchestra.run.smithi071.stdout: "epoch": 11, 2023-05-24T21:20:24.051 INFO:teuthology.orchestra.run.smithi071.stdout: "predecessor_uuid": "1cea3a6e-eb52-480e-95b7-3f6e43317e60", 2023-05-24T21:20:24.051 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": [], 2023-05-24T21:20:24.051 INFO:teuthology.orchestra.run.smithi071.stdout: "period_map": { 2023-05-24T21:20:24.052 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:20:24.052 INFO:teuthology.orchestra.run.smithi071.stdout: "zonegroups": [ 2023-05-24T21:20:24.052 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:24.052 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "96f26f03-a59a-4e4d-9940-cdfce5ea6291", 2023-05-24T21:20:24.052 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "remove_zg", 2023-05-24T21:20:24.052 INFO:teuthology.orchestra.run.smithi071.stdout: "api_name": "remove_zg", 2023-05-24T21:20:24.053 INFO:teuthology.orchestra.run.smithi071.stdout: "is_master": false, 2023-05-24T21:20:24.053 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [], 2023-05-24T21:20:24.053 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames": [], 2023-05-24T21:20:24.053 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames_s3website": [], 2023-05-24T21:20:24.053 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zone": "89aff914-54fe-4a43-9d1c-22f67cc33bb4", 2023-05-24T21:20:24.053 INFO:teuthology.orchestra.run.smithi071.stdout: "zones": [ 2023-05-24T21:20:24.054 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:24.054 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "89aff914-54fe-4a43-9d1c-22f67cc33bb4", 2023-05-24T21:20:24.054 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "remove", 2023-05-24T21:20:24.054 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [], 2023-05-24T21:20:24.054 INFO:teuthology.orchestra.run.smithi071.stdout: "log_meta": false, 2023-05-24T21:20:24.054 INFO:teuthology.orchestra.run.smithi071.stdout: "log_data": false, 2023-05-24T21:20:24.054 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_index_max_shards": 11, 2023-05-24T21:20:24.055 INFO:teuthology.orchestra.run.smithi071.stdout: "read_only": false, 2023-05-24T21:20:24.055 INFO:teuthology.orchestra.run.smithi071.stdout: "tier_type": "", 2023-05-24T21:20:24.055 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from_all": true, 2023-05-24T21:20:24.055 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from": [], 2023-05-24T21:20:24.055 INFO:teuthology.orchestra.run.smithi071.stdout: "redirect_zone": "", 2023-05-24T21:20:24.055 INFO:teuthology.orchestra.run.smithi071.stdout: "supported_features": [ 2023-05-24T21:20:24.055 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:20:24.056 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:24.056 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:24.056 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:24.056 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_targets": [ 2023-05-24T21:20:24.056 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:24.056 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "default-placement", 2023-05-24T21:20:24.056 INFO:teuthology.orchestra.run.smithi071.stdout: "tags": [], 2023-05-24T21:20:24.057 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": [ 2023-05-24T21:20:24.057 INFO:teuthology.orchestra.run.smithi071.stdout: "STANDARD" 2023-05-24T21:20:24.057 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:24.057 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:24.057 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:24.057 INFO:teuthology.orchestra.run.smithi071.stdout: "default_placement": "default-placement", 2023-05-24T21:20:24.058 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T21:20:24.058 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_policy": { 2023-05-24T21:20:24.058 INFO:teuthology.orchestra.run.smithi071.stdout: "groups": [] 2023-05-24T21:20:24.058 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:24.058 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled_features": [ 2023-05-24T21:20:24.058 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:20:24.058 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:24.059 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:24.059 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:24.059 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "a", 2023-05-24T21:20:24.059 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a", 2023-05-24T21:20:24.059 INFO:teuthology.orchestra.run.smithi071.stdout: "api_name": "", 2023-05-24T21:20:24.059 INFO:teuthology.orchestra.run.smithi071.stdout: "is_master": true, 2023-05-24T21:20:24.059 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:20:24.060 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-05-24T21:20:24.060 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:24.060 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames": [], 2023-05-24T21:20:24.060 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames_s3website": [], 2023-05-24T21:20:24.060 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zone": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T21:20:24.060 INFO:teuthology.orchestra.run.smithi071.stdout: "zones": [ 2023-05-24T21:20:24.061 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:24.061 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T21:20:24.061 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a1", 2023-05-24T21:20:24.061 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:20:24.061 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-05-24T21:20:24.061 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:24.061 INFO:teuthology.orchestra.run.smithi071.stdout: "log_meta": false, 2023-05-24T21:20:24.062 INFO:teuthology.orchestra.run.smithi071.stdout: "log_data": true, 2023-05-24T21:20:24.062 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_index_max_shards": 11, 2023-05-24T21:20:24.062 INFO:teuthology.orchestra.run.smithi071.stdout: "read_only": false, 2023-05-24T21:20:24.062 INFO:teuthology.orchestra.run.smithi071.stdout: "tier_type": "", 2023-05-24T21:20:24.062 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from_all": true, 2023-05-24T21:20:24.062 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from": [], 2023-05-24T21:20:24.062 INFO:teuthology.orchestra.run.smithi071.stdout: "redirect_zone": "", 2023-05-24T21:20:24.063 INFO:teuthology.orchestra.run.smithi071.stdout: "supported_features": [ 2023-05-24T21:20:24.063 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:20:24.063 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:24.063 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:24.063 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:24.063 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "5af1588c-f9b9-4687-a040-ce6e0cc808d9", 2023-05-24T21:20:24.063 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a2", 2023-05-24T21:20:24.064 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:20:24.064 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi071.front.sepia.ceph.com:8001" 2023-05-24T21:20:24.064 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:24.064 INFO:teuthology.orchestra.run.smithi071.stdout: "log_meta": false, 2023-05-24T21:20:24.064 INFO:teuthology.orchestra.run.smithi071.stdout: "log_data": true, 2023-05-24T21:20:24.064 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_index_max_shards": 11, 2023-05-24T21:20:24.064 INFO:teuthology.orchestra.run.smithi071.stdout: "read_only": false, 2023-05-24T21:20:24.065 INFO:teuthology.orchestra.run.smithi071.stdout: "tier_type": "", 2023-05-24T21:20:24.065 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from_all": true, 2023-05-24T21:20:24.065 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from": [], 2023-05-24T21:20:24.065 INFO:teuthology.orchestra.run.smithi071.stdout: "redirect_zone": "", 2023-05-24T21:20:24.065 INFO:teuthology.orchestra.run.smithi071.stdout: "supported_features": [ 2023-05-24T21:20:24.065 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:20:24.065 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:24.066 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:24.066 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:24.066 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_targets": [ 2023-05-24T21:20:24.066 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:24.066 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "default-placement", 2023-05-24T21:20:24.066 INFO:teuthology.orchestra.run.smithi071.stdout: "tags": [], 2023-05-24T21:20:24.066 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": [ 2023-05-24T21:20:24.067 INFO:teuthology.orchestra.run.smithi071.stdout: "STANDARD" 2023-05-24T21:20:24.067 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:24.067 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:24.067 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:24.067 INFO:teuthology.orchestra.run.smithi071.stdout: "default_placement": "default-placement", 2023-05-24T21:20:24.067 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T21:20:24.067 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_policy": { 2023-05-24T21:20:24.068 INFO:teuthology.orchestra.run.smithi071.stdout: "groups": [] 2023-05-24T21:20:24.068 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:24.068 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled_features": [ 2023-05-24T21:20:24.068 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:20:24.068 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:24.068 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:24.069 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:24.069 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "b", 2023-05-24T21:20:24.069 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "b", 2023-05-24T21:20:24.069 INFO:teuthology.orchestra.run.smithi071.stdout: "api_name": "", 2023-05-24T21:20:24.069 INFO:teuthology.orchestra.run.smithi071.stdout: "is_master": false, 2023-05-24T21:20:24.069 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:20:24.069 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi157.front.sepia.ceph.com:8000" 2023-05-24T21:20:24.070 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:24.070 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames": [], 2023-05-24T21:20:24.070 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames_s3website": [], 2023-05-24T21:20:24.070 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zone": "ee18cd86-4684-4b3d-8de6-0075d3d925f1", 2023-05-24T21:20:24.070 INFO:teuthology.orchestra.run.smithi071.stdout: "zones": [ 2023-05-24T21:20:24.070 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:24.070 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "7b8c3b69-c8ed-4621-b2ef-b89d3bbb7973", 2023-05-24T21:20:24.071 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "b2", 2023-05-24T21:20:24.071 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:20:24.071 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi157.front.sepia.ceph.com:8001" 2023-05-24T21:20:24.071 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:24.071 INFO:teuthology.orchestra.run.smithi071.stdout: "log_meta": false, 2023-05-24T21:20:24.071 INFO:teuthology.orchestra.run.smithi071.stdout: "log_data": true, 2023-05-24T21:20:24.071 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_index_max_shards": 11, 2023-05-24T21:20:24.072 INFO:teuthology.orchestra.run.smithi071.stdout: "read_only": false, 2023-05-24T21:20:24.072 INFO:teuthology.orchestra.run.smithi071.stdout: "tier_type": "", 2023-05-24T21:20:24.072 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from_all": true, 2023-05-24T21:20:24.072 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from": [], 2023-05-24T21:20:24.072 INFO:teuthology.orchestra.run.smithi071.stdout: "redirect_zone": "", 2023-05-24T21:20:24.072 INFO:teuthology.orchestra.run.smithi071.stdout: "supported_features": [ 2023-05-24T21:20:24.072 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:20:24.073 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:24.073 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:24.073 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:24.073 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "ee18cd86-4684-4b3d-8de6-0075d3d925f1", 2023-05-24T21:20:24.073 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "b1", 2023-05-24T21:20:24.073 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:20:24.073 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi157.front.sepia.ceph.com:8000" 2023-05-24T21:20:24.074 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:24.074 INFO:teuthology.orchestra.run.smithi071.stdout: "log_meta": false, 2023-05-24T21:20:24.074 INFO:teuthology.orchestra.run.smithi071.stdout: "log_data": true, 2023-05-24T21:20:24.074 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_index_max_shards": 11, 2023-05-24T21:20:24.074 INFO:teuthology.orchestra.run.smithi071.stdout: "read_only": false, 2023-05-24T21:20:24.074 INFO:teuthology.orchestra.run.smithi071.stdout: "tier_type": "", 2023-05-24T21:20:24.075 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from_all": true, 2023-05-24T21:20:24.075 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from": [], 2023-05-24T21:20:24.075 INFO:teuthology.orchestra.run.smithi071.stdout: "redirect_zone": "", 2023-05-24T21:20:24.075 INFO:teuthology.orchestra.run.smithi071.stdout: "supported_features": [ 2023-05-24T21:20:24.075 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:20:24.075 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:24.075 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:24.075 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:24.076 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_targets": [ 2023-05-24T21:20:24.076 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:24.076 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "default-placement", 2023-05-24T21:20:24.076 INFO:teuthology.orchestra.run.smithi071.stdout: "tags": [], 2023-05-24T21:20:24.076 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": [ 2023-05-24T21:20:24.076 INFO:teuthology.orchestra.run.smithi071.stdout: "STANDARD" 2023-05-24T21:20:24.077 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:24.077 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:24.077 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:24.077 INFO:teuthology.orchestra.run.smithi071.stdout: "default_placement": "default-placement", 2023-05-24T21:20:24.077 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T21:20:24.077 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_policy": { 2023-05-24T21:20:24.077 INFO:teuthology.orchestra.run.smithi071.stdout: "groups": [] 2023-05-24T21:20:24.078 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:24.078 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled_features": [ 2023-05-24T21:20:24.078 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:20:24.078 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:24.078 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:24.078 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:24.078 INFO:teuthology.orchestra.run.smithi071.stdout: "short_zone_ids": [ 2023-05-24T21:20:24.079 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:24.079 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T21:20:24.079 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 3764661814 2023-05-24T21:20:24.079 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:24.079 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:24.079 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "5af1588c-f9b9-4687-a040-ce6e0cc808d9", 2023-05-24T21:20:24.080 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 2266179370 2023-05-24T21:20:24.080 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:24.080 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:24.080 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "7b8c3b69-c8ed-4621-b2ef-b89d3bbb7973", 2023-05-24T21:20:24.080 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 1540450555 2023-05-24T21:20:24.080 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:24.080 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:24.081 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "89aff914-54fe-4a43-9d1c-22f67cc33bb4", 2023-05-24T21:20:24.081 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 2457252630 2023-05-24T21:20:24.081 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:24.081 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:24.081 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "ee18cd86-4684-4b3d-8de6-0075d3d925f1", 2023-05-24T21:20:24.081 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 489875403 2023-05-24T21:20:24.081 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:24.082 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:24.082 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:24.082 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zonegroup": "a", 2023-05-24T21:20:24.082 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zone": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T21:20:24.082 INFO:teuthology.orchestra.run.smithi071.stdout: "period_config": { 2023-05-24T21:20:24.082 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_quota": { 2023-05-24T21:20:24.082 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false, 2023-05-24T21:20:24.083 INFO:teuthology.orchestra.run.smithi071.stdout: "check_on_raw": false, 2023-05-24T21:20:24.083 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size": -1, 2023-05-24T21:20:24.083 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size_kb": 0, 2023-05-24T21:20:24.083 INFO:teuthology.orchestra.run.smithi071.stdout: "max_objects": -1 2023-05-24T21:20:24.083 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:24.083 INFO:teuthology.orchestra.run.smithi071.stdout: "user_quota": { 2023-05-24T21:20:24.083 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false, 2023-05-24T21:20:24.084 INFO:teuthology.orchestra.run.smithi071.stdout: "check_on_raw": false, 2023-05-24T21:20:24.085 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size": -1, 2023-05-24T21:20:24.085 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size_kb": 0, 2023-05-24T21:20:24.085 INFO:teuthology.orchestra.run.smithi071.stdout: "max_objects": -1 2023-05-24T21:20:24.085 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:24.085 INFO:teuthology.orchestra.run.smithi071.stdout: "user_ratelimit": { 2023-05-24T21:20:24.085 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_ops": 0, 2023-05-24T21:20:24.086 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_ops": 0, 2023-05-24T21:20:24.086 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_bytes": 0, 2023-05-24T21:20:24.086 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_bytes": 0, 2023-05-24T21:20:24.086 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false 2023-05-24T21:20:24.086 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:24.086 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_ratelimit": { 2023-05-24T21:20:24.087 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_ops": 0, 2023-05-24T21:20:24.087 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_ops": 0, 2023-05-24T21:20:24.087 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_bytes": 0, 2023-05-24T21:20:24.087 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_bytes": 0, 2023-05-24T21:20:24.087 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false 2023-05-24T21:20:24.087 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:24.087 INFO:teuthology.orchestra.run.smithi071.stdout: "anonymous_ratelimit": { 2023-05-24T21:20:24.088 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_ops": 0, 2023-05-24T21:20:24.088 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_ops": 0, 2023-05-24T21:20:24.088 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_bytes": 0, 2023-05-24T21:20:24.088 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_bytes": 0, 2023-05-24T21:20:24.088 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false 2023-05-24T21:20:24.088 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:24.088 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:24.089 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T21:20:24.089 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_name": "test-realm", 2023-05-24T21:20:24.089 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:20:24.089 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:20:24.089 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:20:24.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:20:24.090 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-24T21:20:24.140 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:20:24.141 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:20:24.171 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:20:24.165+0000 7f9f75e49780 0 NOTICE: removed master_zone of zonegroup remove_zg 2023-05-24T21:20:24.173 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:20:24.167+0000 7f9f75e49780 0 Removed zone from zonegroup remove_zg 2023-05-24T21:20:24.189 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:20:24.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:20:24.190 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-24T21:20:24.243 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:20:24.243 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:20:24.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:20:24.262+0000 7ff12c7e7780 0 ERROR: zonegroup remove_zg has a non existent master zone 2023-05-24T21:20:24.272 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-05-24T21:20:24.273 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:20:24.273 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:20:24.274 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-24T21:20:24.329 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:20:24.329 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:20:24.385 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:20:24.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:20:24.386 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-24T21:20:24.444 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:20:24.445 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:20:24.590 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:20:24.590 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:20:24.590 INFO:teuthology.orchestra.run.smithi071.stdout: "epoch": 12, 2023-05-24T21:20:24.590 INFO:teuthology.orchestra.run.smithi071.stdout: "predecessor_uuid": "1cea3a6e-eb52-480e-95b7-3f6e43317e60", 2023-05-24T21:20:24.591 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": [], 2023-05-24T21:20:24.591 INFO:teuthology.orchestra.run.smithi071.stdout: "period_map": { 2023-05-24T21:20:24.591 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:20:24.591 INFO:teuthology.orchestra.run.smithi071.stdout: "zonegroups": [ 2023-05-24T21:20:24.591 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:24.591 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "a", 2023-05-24T21:20:24.592 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a", 2023-05-24T21:20:24.592 INFO:teuthology.orchestra.run.smithi071.stdout: "api_name": "", 2023-05-24T21:20:24.592 INFO:teuthology.orchestra.run.smithi071.stdout: "is_master": true, 2023-05-24T21:20:24.592 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:20:24.592 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-05-24T21:20:24.592 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:24.592 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames": [], 2023-05-24T21:20:24.593 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames_s3website": [], 2023-05-24T21:20:24.593 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zone": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T21:20:24.593 INFO:teuthology.orchestra.run.smithi071.stdout: "zones": [ 2023-05-24T21:20:24.593 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:24.593 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T21:20:24.593 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a1", 2023-05-24T21:20:24.593 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:20:24.594 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-05-24T21:20:24.594 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:24.594 INFO:teuthology.orchestra.run.smithi071.stdout: "log_meta": false, 2023-05-24T21:20:24.594 INFO:teuthology.orchestra.run.smithi071.stdout: "log_data": true, 2023-05-24T21:20:24.594 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_index_max_shards": 11, 2023-05-24T21:20:24.594 INFO:teuthology.orchestra.run.smithi071.stdout: "read_only": false, 2023-05-24T21:20:24.594 INFO:teuthology.orchestra.run.smithi071.stdout: "tier_type": "", 2023-05-24T21:20:24.595 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from_all": true, 2023-05-24T21:20:24.595 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from": [], 2023-05-24T21:20:24.595 INFO:teuthology.orchestra.run.smithi071.stdout: "redirect_zone": "", 2023-05-24T21:20:24.595 INFO:teuthology.orchestra.run.smithi071.stdout: "supported_features": [ 2023-05-24T21:20:24.595 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:20:24.595 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:24.596 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:24.596 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:24.596 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "5af1588c-f9b9-4687-a040-ce6e0cc808d9", 2023-05-24T21:20:24.596 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a2", 2023-05-24T21:20:24.596 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:20:24.596 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi071.front.sepia.ceph.com:8001" 2023-05-24T21:20:24.596 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:24.597 INFO:teuthology.orchestra.run.smithi071.stdout: "log_meta": false, 2023-05-24T21:20:24.597 INFO:teuthology.orchestra.run.smithi071.stdout: "log_data": true, 2023-05-24T21:20:24.597 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_index_max_shards": 11, 2023-05-24T21:20:24.597 INFO:teuthology.orchestra.run.smithi071.stdout: "read_only": false, 2023-05-24T21:20:24.597 INFO:teuthology.orchestra.run.smithi071.stdout: "tier_type": "", 2023-05-24T21:20:24.597 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from_all": true, 2023-05-24T21:20:24.597 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from": [], 2023-05-24T21:20:24.598 INFO:teuthology.orchestra.run.smithi071.stdout: "redirect_zone": "", 2023-05-24T21:20:24.598 INFO:teuthology.orchestra.run.smithi071.stdout: "supported_features": [ 2023-05-24T21:20:24.598 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:20:24.598 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:24.598 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:24.598 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:24.598 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_targets": [ 2023-05-24T21:20:24.598 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:24.599 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "default-placement", 2023-05-24T21:20:24.599 INFO:teuthology.orchestra.run.smithi071.stdout: "tags": [], 2023-05-24T21:20:24.599 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": [ 2023-05-24T21:20:24.599 INFO:teuthology.orchestra.run.smithi071.stdout: "STANDARD" 2023-05-24T21:20:24.599 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:24.599 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:24.600 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:24.600 INFO:teuthology.orchestra.run.smithi071.stdout: "default_placement": "default-placement", 2023-05-24T21:20:24.600 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T21:20:24.600 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_policy": { 2023-05-24T21:20:24.600 INFO:teuthology.orchestra.run.smithi071.stdout: "groups": [] 2023-05-24T21:20:24.600 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:24.600 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled_features": [ 2023-05-24T21:20:24.601 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:20:24.601 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:24.601 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:24.601 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:24.601 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "b", 2023-05-24T21:20:24.601 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "b", 2023-05-24T21:20:24.601 INFO:teuthology.orchestra.run.smithi071.stdout: "api_name": "", 2023-05-24T21:20:24.602 INFO:teuthology.orchestra.run.smithi071.stdout: "is_master": false, 2023-05-24T21:20:24.602 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:20:24.602 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi157.front.sepia.ceph.com:8000" 2023-05-24T21:20:24.602 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:24.602 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames": [], 2023-05-24T21:20:24.602 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames_s3website": [], 2023-05-24T21:20:24.602 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zone": "ee18cd86-4684-4b3d-8de6-0075d3d925f1", 2023-05-24T21:20:24.603 INFO:teuthology.orchestra.run.smithi071.stdout: "zones": [ 2023-05-24T21:20:24.603 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:24.603 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "7b8c3b69-c8ed-4621-b2ef-b89d3bbb7973", 2023-05-24T21:20:24.603 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "b2", 2023-05-24T21:20:24.603 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:20:24.603 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi157.front.sepia.ceph.com:8001" 2023-05-24T21:20:24.603 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:24.604 INFO:teuthology.orchestra.run.smithi071.stdout: "log_meta": false, 2023-05-24T21:20:24.604 INFO:teuthology.orchestra.run.smithi071.stdout: "log_data": true, 2023-05-24T21:20:24.604 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_index_max_shards": 11, 2023-05-24T21:20:24.604 INFO:teuthology.orchestra.run.smithi071.stdout: "read_only": false, 2023-05-24T21:20:24.604 INFO:teuthology.orchestra.run.smithi071.stdout: "tier_type": "", 2023-05-24T21:20:24.604 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from_all": true, 2023-05-24T21:20:24.604 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from": [], 2023-05-24T21:20:24.605 INFO:teuthology.orchestra.run.smithi071.stdout: "redirect_zone": "", 2023-05-24T21:20:24.605 INFO:teuthology.orchestra.run.smithi071.stdout: "supported_features": [ 2023-05-24T21:20:24.605 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:20:24.605 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:24.605 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:24.605 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:24.626 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "ee18cd86-4684-4b3d-8de6-0075d3d925f1", 2023-05-24T21:20:24.626 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "b1", 2023-05-24T21:20:24.626 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-05-24T21:20:24.626 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi157.front.sepia.ceph.com:8000" 2023-05-24T21:20:24.626 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:24.627 INFO:teuthology.orchestra.run.smithi071.stdout: "log_meta": false, 2023-05-24T21:20:24.627 INFO:teuthology.orchestra.run.smithi071.stdout: "log_data": true, 2023-05-24T21:20:24.627 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_index_max_shards": 11, 2023-05-24T21:20:24.627 INFO:teuthology.orchestra.run.smithi071.stdout: "read_only": false, 2023-05-24T21:20:24.627 INFO:teuthology.orchestra.run.smithi071.stdout: "tier_type": "", 2023-05-24T21:20:24.627 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from_all": true, 2023-05-24T21:20:24.628 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from": [], 2023-05-24T21:20:24.628 INFO:teuthology.orchestra.run.smithi071.stdout: "redirect_zone": "", 2023-05-24T21:20:24.628 INFO:teuthology.orchestra.run.smithi071.stdout: "supported_features": [ 2023-05-24T21:20:24.628 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:20:24.628 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:24.628 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:24.628 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:24.629 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_targets": [ 2023-05-24T21:20:24.629 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:24.629 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "default-placement", 2023-05-24T21:20:24.629 INFO:teuthology.orchestra.run.smithi071.stdout: "tags": [], 2023-05-24T21:20:24.629 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": [ 2023-05-24T21:20:24.629 INFO:teuthology.orchestra.run.smithi071.stdout: "STANDARD" 2023-05-24T21:20:24.629 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:24.630 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:24.630 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:24.630 INFO:teuthology.orchestra.run.smithi071.stdout: "default_placement": "default-placement", 2023-05-24T21:20:24.630 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T21:20:24.630 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_policy": { 2023-05-24T21:20:24.630 INFO:teuthology.orchestra.run.smithi071.stdout: "groups": [] 2023-05-24T21:20:24.630 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:24.631 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled_features": [ 2023-05-24T21:20:24.631 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-05-24T21:20:24.631 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:24.631 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:24.631 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-05-24T21:20:24.631 INFO:teuthology.orchestra.run.smithi071.stdout: "short_zone_ids": [ 2023-05-24T21:20:24.631 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:24.632 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T21:20:24.632 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 3764661814 2023-05-24T21:20:24.632 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:24.632 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:24.632 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "5af1588c-f9b9-4687-a040-ce6e0cc808d9", 2023-05-24T21:20:24.632 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 2266179370 2023-05-24T21:20:24.632 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:24.633 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:24.633 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "7b8c3b69-c8ed-4621-b2ef-b89d3bbb7973", 2023-05-24T21:20:24.633 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 1540450555 2023-05-24T21:20:24.633 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:24.633 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:24.633 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "ee18cd86-4684-4b3d-8de6-0075d3d925f1", 2023-05-24T21:20:24.633 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 489875403 2023-05-24T21:20:24.634 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:24.634 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:24.634 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:24.634 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zonegroup": "a", 2023-05-24T21:20:24.634 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zone": "299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T21:20:24.634 INFO:teuthology.orchestra.run.smithi071.stdout: "period_config": { 2023-05-24T21:20:24.634 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_quota": { 2023-05-24T21:20:24.635 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false, 2023-05-24T21:20:24.635 INFO:teuthology.orchestra.run.smithi071.stdout: "check_on_raw": false, 2023-05-24T21:20:24.635 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size": -1, 2023-05-24T21:20:24.635 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size_kb": 0, 2023-05-24T21:20:24.635 INFO:teuthology.orchestra.run.smithi071.stdout: "max_objects": -1 2023-05-24T21:20:24.635 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:24.635 INFO:teuthology.orchestra.run.smithi071.stdout: "user_quota": { 2023-05-24T21:20:24.636 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false, 2023-05-24T21:20:24.636 INFO:teuthology.orchestra.run.smithi071.stdout: "check_on_raw": false, 2023-05-24T21:20:24.636 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size": -1, 2023-05-24T21:20:24.636 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size_kb": 0, 2023-05-24T21:20:24.636 INFO:teuthology.orchestra.run.smithi071.stdout: "max_objects": -1 2023-05-24T21:20:24.636 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:24.636 INFO:teuthology.orchestra.run.smithi071.stdout: "user_ratelimit": { 2023-05-24T21:20:24.637 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_ops": 0, 2023-05-24T21:20:24.637 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_ops": 0, 2023-05-24T21:20:24.637 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_bytes": 0, 2023-05-24T21:20:24.637 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_bytes": 0, 2023-05-24T21:20:24.637 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false 2023-05-24T21:20:24.637 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:24.637 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_ratelimit": { 2023-05-24T21:20:24.638 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_ops": 0, 2023-05-24T21:20:24.638 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_ops": 0, 2023-05-24T21:20:24.638 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_bytes": 0, 2023-05-24T21:20:24.638 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_bytes": 0, 2023-05-24T21:20:24.638 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false 2023-05-24T21:20:24.638 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:24.638 INFO:teuthology.orchestra.run.smithi071.stdout: "anonymous_ratelimit": { 2023-05-24T21:20:24.639 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_ops": 0, 2023-05-24T21:20:24.639 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_ops": 0, 2023-05-24T21:20:24.639 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_bytes": 0, 2023-05-24T21:20:24.639 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_bytes": 0, 2023-05-24T21:20:24.639 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false 2023-05-24T21:20:24.639 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:24.639 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:24.640 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "4a7754f9-b4ac-4d00-9827-3910f2de91d2", 2023-05-24T21:20:24.640 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_name": "test-realm", 2023-05-24T21:20:24.640 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:20:24.640 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:20:24.641 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zg_master_zone_delete ... ok 2023-05-24T21:20:24.642 INFO:rgw_multi.tests:create bucket zone=a1 name=qskrps-36 2023-05-24T21:20:24.642 DEBUG:boto:path=/qskrps-36/ 2023-05-24T21:20:24.643 DEBUG:boto:auth_path=/qskrps-36/ 2023-05-24T21:20:24.643 DEBUG:boto:Method: PUT 2023-05-24T21:20:24.643 DEBUG:boto:Path: /qskrps-36/ 2023-05-24T21:20:24.644 DEBUG:boto:Data: 2023-05-24T21:20:24.644 DEBUG:boto:Headers: {} 2023-05-24T21:20:24.644 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:20:24.645 DEBUG:boto:Port: 8000 2023-05-24T21:20:24.645 DEBUG:boto:Params: {} 2023-05-24T21:20:24.646 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T21:20:24.646 DEBUG:boto:Token: None 2023-05-24T21:20:24.646 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:20:24 GMT /qskrps-36/ 2023-05-24T21:20:24.647 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:mRQQ6osHyvjrIbjSkgIHCY8Lpaw= 2023-05-24T21:20:24.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:20:24 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:mRQQ6osHyvjrIbjSkgIHCY8Lpaw=', 'Content-Length': '0'} 2023-05-24T21:20:25.445 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000344916c583c85e84-00646e7f99-1627-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:20:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:20:25.446 INFO:rgw_multi.tests:create bucket zone=a2 name=qskrps-37 2023-05-24T21:20:25.446 DEBUG:boto:path=/qskrps-37/ 2023-05-24T21:20:25.447 DEBUG:boto:auth_path=/qskrps-37/ 2023-05-24T21:20:25.447 DEBUG:boto:Method: PUT 2023-05-24T21:20:25.447 DEBUG:boto:Path: /qskrps-37/ 2023-05-24T21:20:25.448 DEBUG:boto:Data: 2023-05-24T21:20:25.448 DEBUG:boto:Headers: {} 2023-05-24T21:20:25.449 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:20:25.449 DEBUG:boto:Port: 8001 2023-05-24T21:20:25.449 DEBUG:boto:Params: {} 2023-05-24T21:20:25.450 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-24T21:20:25.450 DEBUG:boto:Token: None 2023-05-24T21:20:25.450 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:20:25 GMT /qskrps-37/ 2023-05-24T21:20:25.451 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:uQBKlR+c2yEpbZjBOyd9UfAaOSg= 2023-05-24T21:20:25.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:20:25 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:uQBKlR+c2yEpbZjBOyd9UfAaOSg=', 'Content-Length': '0'} 2023-05-24T21:20:25.512 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008c3253c4c76c1fa5-00646e7f99-1625-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:20:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:20:25.513 INFO:rgw_multi.tests:create bucket zone=b1 name=qskrps-38 2023-05-24T21:20:25.513 DEBUG:boto:path=/qskrps-38/ 2023-05-24T21:20:25.514 DEBUG:boto:auth_path=/qskrps-38/ 2023-05-24T21:20:25.514 DEBUG:boto:Method: PUT 2023-05-24T21:20:25.514 DEBUG:boto:Path: /qskrps-38/ 2023-05-24T21:20:25.515 DEBUG:boto:Data: 2023-05-24T21:20:25.515 DEBUG:boto:Headers: {} 2023-05-24T21:20:25.515 DEBUG:boto:Host: smithi157.front.sepia.ceph.com:8000 2023-05-24T21:20:25.516 DEBUG:boto:Port: 8000 2023-05-24T21:20:25.516 DEBUG:boto:Params: {} 2023-05-24T21:20:25.517 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T21:20:25.517 DEBUG:boto:Token: None 2023-05-24T21:20:25.517 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:20:25 GMT /qskrps-38/ 2023-05-24T21:20:25.518 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:eRhFXf+9IUED6628jdivuwXuCqE= 2023-05-24T21:20:25.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:20:25 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:eRhFXf+9IUED6628jdivuwXuCqE=', 'Content-Length': '0'} 2023-05-24T21:20:25.765 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004f772cbef45a0451-00646e7f99-1166-b1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:20:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:20:25.766 INFO:rgw_multi.tests:create bucket zone=b2 name=qskrps-39 2023-05-24T21:20:25.766 DEBUG:boto:path=/qskrps-39/ 2023-05-24T21:20:25.767 DEBUG:boto:auth_path=/qskrps-39/ 2023-05-24T21:20:25.767 DEBUG:boto:Method: PUT 2023-05-24T21:20:25.767 DEBUG:boto:Path: /qskrps-39/ 2023-05-24T21:20:25.768 DEBUG:boto:Data: 2023-05-24T21:20:25.768 DEBUG:boto:Headers: {} 2023-05-24T21:20:25.768 DEBUG:boto:Host: smithi157.front.sepia.ceph.com:8001 2023-05-24T21:20:25.769 DEBUG:boto:Port: 8001 2023-05-24T21:20:25.769 DEBUG:boto:Params: {} 2023-05-24T21:20:25.769 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-24T21:20:25.770 DEBUG:boto:Token: None 2023-05-24T21:20:25.770 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:20:25 GMT /qskrps-39/ 2023-05-24T21:20:25.771 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:PMXMZQT9xD1OpNDipEXFOELtUio= 2023-05-24T21:20:25.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:20:25 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PMXMZQT9xD1OpNDipEXFOELtUio=', 'Content-Length': '0'} 2023-05-24T21:20:25.804 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d3fdb64a9ba23af8-00646e7f99-1162-b2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:20:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:20:25.804 DEBUG:boto:path=/qskrps-36/ 2023-05-24T21:20:25.805 DEBUG:boto:auth_path=/qskrps-36/ 2023-05-24T21:20:25.805 DEBUG:boto:path=/qskrps-36/?website 2023-05-24T21:20:25.805 DEBUG:boto:auth_path=/qskrps-36/?website 2023-05-24T21:20:25.806 DEBUG:boto:Method: PUT 2023-05-24T21:20:25.806 DEBUG:boto:Path: /qskrps-36/?website 2023-05-24T21:20:25.806 DEBUG:boto:Data: index.htmlerror.html 2023-05-24T21:20:25.807 DEBUG:boto:Headers: {} 2023-05-24T21:20:25.807 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:20:25.807 DEBUG:boto:Port: 8000 2023-05-24T21:20:25.808 DEBUG:boto:Params: {} 2023-05-24T21:20:25.808 DEBUG:boto:Token: None 2023-05-24T21:20:25.808 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:20:25 GMT /qskrps-36/?website 2023-05-24T21:20:25.809 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:r/Rl9ml1k4Aryi8cHVR3Oy1VfIE= 2023-05-24T21:20:25.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:20:25 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:r/Rl9ml1k4Aryi8cHVR3Oy1VfIE=', 'Content-Length': '241'} 2023-05-24T21:20:25.810 DEBUG:boto:Response headers: [('Content-Length', '192'), ('x-amz-request-id', 'tx00000fe25eecee02ecba9-00646e7f99-1627-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:20:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:20:25.811 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_website ... SKIP: test_set_bucket_website skipped. Requires rgw_enable_static_website = 1. 2023-05-24T21:20:25.812 INFO:rgw_multi.tests:create bucket zone=a1 name=qskrps-40 2023-05-24T21:20:25.812 DEBUG:boto:path=/qskrps-40/ 2023-05-24T21:20:25.812 DEBUG:boto:auth_path=/qskrps-40/ 2023-05-24T21:20:25.813 DEBUG:boto:Method: PUT 2023-05-24T21:20:25.813 DEBUG:boto:Path: /qskrps-40/ 2023-05-24T21:20:25.813 DEBUG:boto:Data: 2023-05-24T21:20:25.814 DEBUG:boto:Headers: {} 2023-05-24T21:20:25.814 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:20:25.814 DEBUG:boto:Port: 8000 2023-05-24T21:20:25.815 DEBUG:boto:Params: {} 2023-05-24T21:20:25.815 DEBUG:boto:Token: None 2023-05-24T21:20:25.816 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:20:25 GMT /qskrps-40/ 2023-05-24T21:20:25.816 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ve/QYNp2aWNZDGT60Bv4FApRWqs= 2023-05-24T21:20:25.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:20:25 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ve/QYNp2aWNZDGT60Bv4FApRWqs=', 'Content-Length': '0'} 2023-05-24T21:20:25.862 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b5a3670f9efb807b-00646e7f99-1627-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:20:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:20:25.863 INFO:rgw_multi.tests:create bucket zone=a2 name=qskrps-41 2023-05-24T21:20:25.863 DEBUG:boto:path=/qskrps-41/ 2023-05-24T21:20:25.864 DEBUG:boto:auth_path=/qskrps-41/ 2023-05-24T21:20:25.864 DEBUG:boto:Method: PUT 2023-05-24T21:20:25.864 DEBUG:boto:Path: /qskrps-41/ 2023-05-24T21:20:25.865 DEBUG:boto:Data: 2023-05-24T21:20:25.865 DEBUG:boto:Headers: {} 2023-05-24T21:20:25.865 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:20:25.866 DEBUG:boto:Port: 8001 2023-05-24T21:20:25.866 DEBUG:boto:Params: {} 2023-05-24T21:20:25.866 DEBUG:boto:Token: None 2023-05-24T21:20:25.867 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:20:25 GMT /qskrps-41/ 2023-05-24T21:20:25.867 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:eZExURW0+j2VkNOlRGW0vQJQ/aY= 2023-05-24T21:20:25.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:20:25 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:eZExURW0+j2VkNOlRGW0vQJQ/aY=', 'Content-Length': '0'} 2023-05-24T21:20:25.917 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001d852f1e2f9c9a3f-00646e7f99-1625-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:20:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:20:25.918 INFO:rgw_multi.tests:create bucket zone=b1 name=qskrps-42 2023-05-24T21:20:25.918 DEBUG:boto:path=/qskrps-42/ 2023-05-24T21:20:25.919 DEBUG:boto:auth_path=/qskrps-42/ 2023-05-24T21:20:25.919 DEBUG:boto:Method: PUT 2023-05-24T21:20:25.919 DEBUG:boto:Path: /qskrps-42/ 2023-05-24T21:20:25.920 DEBUG:boto:Data: 2023-05-24T21:20:25.920 DEBUG:boto:Headers: {} 2023-05-24T21:20:25.920 DEBUG:boto:Host: smithi157.front.sepia.ceph.com:8000 2023-05-24T21:20:25.921 DEBUG:boto:Port: 8000 2023-05-24T21:20:25.921 DEBUG:boto:Params: {} 2023-05-24T21:20:25.921 DEBUG:boto:Token: None 2023-05-24T21:20:25.922 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:20:25 GMT /qskrps-42/ 2023-05-24T21:20:25.922 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:qYJifdcKpvBfqVhr2j4h/zo1i4A= 2023-05-24T21:20:25.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:20:25 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qYJifdcKpvBfqVhr2j4h/zo1i4A=', 'Content-Length': '0'} 2023-05-24T21:20:25.960 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c3b96f96f7b7dfe6-00646e7f99-1166-b1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:20:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:20:25.961 INFO:rgw_multi.tests:create bucket zone=b2 name=qskrps-43 2023-05-24T21:20:25.961 DEBUG:boto:path=/qskrps-43/ 2023-05-24T21:20:25.961 DEBUG:boto:auth_path=/qskrps-43/ 2023-05-24T21:20:25.962 DEBUG:boto:Method: PUT 2023-05-24T21:20:25.962 DEBUG:boto:Path: /qskrps-43/ 2023-05-24T21:20:25.962 DEBUG:boto:Data: 2023-05-24T21:20:25.963 DEBUG:boto:Headers: {} 2023-05-24T21:20:25.963 DEBUG:boto:Host: smithi157.front.sepia.ceph.com:8001 2023-05-24T21:20:25.963 DEBUG:boto:Port: 8001 2023-05-24T21:20:25.964 DEBUG:boto:Params: {} 2023-05-24T21:20:25.964 DEBUG:boto:Token: None 2023-05-24T21:20:25.965 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:20:25 GMT /qskrps-43/ 2023-05-24T21:20:25.965 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:b8bVaiGmrFHcV+JAwQK25/F/Oos= 2023-05-24T21:20:25.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:20:25 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:b8bVaiGmrFHcV+JAwQK25/F/Oos=', 'Content-Length': '0'} 2023-05-24T21:20:26.019 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000895597c5ff6a89a5-00646e7f99-1162-b2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:20:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:20:26.019 DEBUG:boto:path=/qskrps-40/ 2023-05-24T21:20:26.019 DEBUG:boto:auth_path=/qskrps-40/ 2023-05-24T21:20:26.020 DEBUG:boto:path=/qskrps-40/?policy 2023-05-24T21:20:26.020 DEBUG:boto:auth_path=/qskrps-40/?policy 2023-05-24T21:20:26.020 DEBUG:boto:Method: PUT 2023-05-24T21:20:26.021 DEBUG:boto:Path: /qskrps-40/?policy 2023-05-24T21:20:26.021 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2023-05-24T21:20:26.021 DEBUG:boto:Headers: {} 2023-05-24T21:20:26.022 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:20:26.022 DEBUG:boto:Port: 8000 2023-05-24T21:20:26.022 DEBUG:boto:Params: {} 2023-05-24T21:20:26.023 DEBUG:boto:Token: None 2023-05-24T21:20:26.023 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:20:26 GMT /qskrps-40/?policy 2023-05-24T21:20:26.023 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:szrpG02WKqnOtltMsahVtPOiI3I= 2023-05-24T21:20:26.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:20:26 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:szrpG02WKqnOtltMsahVtPOiI3I=', 'Content-Length': '97'} 2023-05-24T21:20:26.031 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000675b7aa3b034d005-00646e7f9a-1627-a1'), ('Date', 'Wed, 24 May 2023 21:20:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:20:26.031 DEBUG:boto:path=/qskrps-40/ 2023-05-24T21:20:26.032 DEBUG:boto:auth_path=/qskrps-40/ 2023-05-24T21:20:26.032 DEBUG:boto:path=/qskrps-40/?policy 2023-05-24T21:20:26.032 DEBUG:boto:auth_path=/qskrps-40/?policy 2023-05-24T21:20:26.033 DEBUG:boto:Method: GET 2023-05-24T21:20:26.033 DEBUG:boto:Path: /qskrps-40/?policy 2023-05-24T21:20:26.033 DEBUG:boto:Data: 2023-05-24T21:20:26.034 DEBUG:boto:Headers: {} 2023-05-24T21:20:26.034 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:20:26.034 DEBUG:boto:Port: 8000 2023-05-24T21:20:26.035 DEBUG:boto:Params: {} 2023-05-24T21:20:26.035 DEBUG:boto:Token: None 2023-05-24T21:20:26.035 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:20:26 GMT /qskrps-40/?policy 2023-05-24T21:20:26.036 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:OM2paEiJbglRsOJ2FUC2tsTsw+U= 2023-05-24T21:20:26.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:20:26 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:OM2paEiJbglRsOJ2FUC2tsTsw+U=', 'Content-Length': '0'} 2023-05-24T21:20:26.037 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000091bef65959456369-00646e7f9a-1627-a1'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Wed, 24 May 2023 21:20:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:20:26.038 DEBUG:boto:path=/qskrps-41/ 2023-05-24T21:20:26.038 DEBUG:boto:auth_path=/qskrps-41/ 2023-05-24T21:20:26.038 DEBUG:boto:path=/qskrps-41/?policy 2023-05-24T21:20:26.039 DEBUG:boto:auth_path=/qskrps-41/?policy 2023-05-24T21:20:26.039 DEBUG:boto:Method: PUT 2023-05-24T21:20:26.039 DEBUG:boto:Path: /qskrps-41/?policy 2023-05-24T21:20:26.040 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2023-05-24T21:20:26.040 DEBUG:boto:Headers: {} 2023-05-24T21:20:26.040 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:20:26.041 DEBUG:boto:Port: 8001 2023-05-24T21:20:26.041 DEBUG:boto:Params: {} 2023-05-24T21:20:26.041 DEBUG:boto:Token: None 2023-05-24T21:20:26.042 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:20:26 GMT /qskrps-41/?policy 2023-05-24T21:20:26.042 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:qC+650yLEKUueMZWd0JgvdjOD9U= 2023-05-24T21:20:26.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:20:26 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qC+650yLEKUueMZWd0JgvdjOD9U=', 'Content-Length': '97'} 2023-05-24T21:20:27.052 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005a96454f94e91afe-00646e7f9a-1625-a2'), ('Date', 'Wed, 24 May 2023 21:20:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:20:27.053 DEBUG:boto:path=/qskrps-41/ 2023-05-24T21:20:27.053 DEBUG:boto:auth_path=/qskrps-41/ 2023-05-24T21:20:27.054 DEBUG:boto:path=/qskrps-41/?policy 2023-05-24T21:20:27.054 DEBUG:boto:auth_path=/qskrps-41/?policy 2023-05-24T21:20:27.054 DEBUG:boto:Method: GET 2023-05-24T21:20:27.055 DEBUG:boto:Path: /qskrps-41/?policy 2023-05-24T21:20:27.055 DEBUG:boto:Data: 2023-05-24T21:20:27.055 DEBUG:boto:Headers: {} 2023-05-24T21:20:27.056 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:20:27.056 DEBUG:boto:Port: 8001 2023-05-24T21:20:27.056 DEBUG:boto:Params: {} 2023-05-24T21:20:27.057 DEBUG:boto:Token: None 2023-05-24T21:20:27.057 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:20:27 GMT /qskrps-41/?policy 2023-05-24T21:20:27.057 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:MMKovtVwnhK/qIeVNWD4NPokTLM= 2023-05-24T21:20:27.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:20:27 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:MMKovtVwnhK/qIeVNWD4NPokTLM=', 'Content-Length': '0'} 2023-05-24T21:20:27.059 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bbc86040512fa1f3-00646e7f9b-1625-a2'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Wed, 24 May 2023 21:20:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:20:27.060 DEBUG:boto:path=/qskrps-42/ 2023-05-24T21:20:27.060 DEBUG:boto:auth_path=/qskrps-42/ 2023-05-24T21:20:27.060 DEBUG:boto:path=/qskrps-42/?policy 2023-05-24T21:20:27.061 DEBUG:boto:auth_path=/qskrps-42/?policy 2023-05-24T21:20:27.061 DEBUG:boto:Method: PUT 2023-05-24T21:20:27.061 DEBUG:boto:Path: /qskrps-42/?policy 2023-05-24T21:20:27.061 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2023-05-24T21:20:27.062 DEBUG:boto:Headers: {} 2023-05-24T21:20:27.062 DEBUG:boto:Host: smithi157.front.sepia.ceph.com:8000 2023-05-24T21:20:27.062 DEBUG:boto:Port: 8000 2023-05-24T21:20:27.063 DEBUG:boto:Params: {} 2023-05-24T21:20:27.063 DEBUG:boto:Token: None 2023-05-24T21:20:27.063 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:20:27 GMT /qskrps-42/?policy 2023-05-24T21:20:27.064 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:p4LCLgNZmm4JWjshCLZCvl649gs= 2023-05-24T21:20:27.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:20:27 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:p4LCLgNZmm4JWjshCLZCvl649gs=', 'Content-Length': '97'} 2023-05-24T21:20:28.075 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000011fb1f29338950f1-00646e7f9b-1166-b1'), ('Date', 'Wed, 24 May 2023 21:20:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:20:28.076 DEBUG:boto:path=/qskrps-42/ 2023-05-24T21:20:28.077 DEBUG:boto:auth_path=/qskrps-42/ 2023-05-24T21:20:28.077 DEBUG:boto:path=/qskrps-42/?policy 2023-05-24T21:20:28.077 DEBUG:boto:auth_path=/qskrps-42/?policy 2023-05-24T21:20:28.078 DEBUG:boto:Method: GET 2023-05-24T21:20:28.078 DEBUG:boto:Path: /qskrps-42/?policy 2023-05-24T21:20:28.078 DEBUG:boto:Data: 2023-05-24T21:20:28.078 DEBUG:boto:Headers: {} 2023-05-24T21:20:28.079 DEBUG:boto:Host: smithi157.front.sepia.ceph.com:8000 2023-05-24T21:20:28.079 DEBUG:boto:Port: 8000 2023-05-24T21:20:28.079 DEBUG:boto:Params: {} 2023-05-24T21:20:28.080 DEBUG:boto:Token: None 2023-05-24T21:20:28.080 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:20:28 GMT /qskrps-42/?policy 2023-05-24T21:20:28.081 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:YSBKgtvGgMq/N1QddiuegB5yX5s= 2023-05-24T21:20:28.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:20:28 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:YSBKgtvGgMq/N1QddiuegB5yX5s=', 'Content-Length': '0'} 2023-05-24T21:20:28.082 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000029e742b273fb4d60-00646e7f9c-1166-b1'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Wed, 24 May 2023 21:20:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:20:28.083 DEBUG:boto:path=/qskrps-43/ 2023-05-24T21:20:28.083 DEBUG:boto:auth_path=/qskrps-43/ 2023-05-24T21:20:28.084 DEBUG:boto:path=/qskrps-43/?policy 2023-05-24T21:20:28.084 DEBUG:boto:auth_path=/qskrps-43/?policy 2023-05-24T21:20:28.084 DEBUG:boto:Method: PUT 2023-05-24T21:20:28.085 DEBUG:boto:Path: /qskrps-43/?policy 2023-05-24T21:20:28.085 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2023-05-24T21:20:28.085 DEBUG:boto:Headers: {} 2023-05-24T21:20:28.086 DEBUG:boto:Host: smithi157.front.sepia.ceph.com:8001 2023-05-24T21:20:28.086 DEBUG:boto:Port: 8001 2023-05-24T21:20:28.086 DEBUG:boto:Params: {} 2023-05-24T21:20:28.087 DEBUG:boto:Token: None 2023-05-24T21:20:28.087 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:20:28 GMT /qskrps-43/?policy 2023-05-24T21:20:28.088 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:WikZDnuXovHsMa1aUOBR934PzUs= 2023-05-24T21:20:28.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:20:28 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WikZDnuXovHsMa1aUOBR934PzUs=', 'Content-Length': '97'} 2023-05-24T21:20:29.099 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000895363e7de3cf177-00646e7f9c-1162-b2'), ('Date', 'Wed, 24 May 2023 21:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:20:29.100 DEBUG:boto:path=/qskrps-43/ 2023-05-24T21:20:29.100 DEBUG:boto:auth_path=/qskrps-43/ 2023-05-24T21:20:29.101 DEBUG:boto:path=/qskrps-43/?policy 2023-05-24T21:20:29.101 DEBUG:boto:auth_path=/qskrps-43/?policy 2023-05-24T21:20:29.101 DEBUG:boto:Method: GET 2023-05-24T21:20:29.102 DEBUG:boto:Path: /qskrps-43/?policy 2023-05-24T21:20:29.102 DEBUG:boto:Data: 2023-05-24T21:20:29.103 DEBUG:boto:Headers: {} 2023-05-24T21:20:29.103 DEBUG:boto:Host: smithi157.front.sepia.ceph.com:8001 2023-05-24T21:20:29.103 DEBUG:boto:Port: 8001 2023-05-24T21:20:29.103 DEBUG:boto:Params: {} 2023-05-24T21:20:29.104 DEBUG:boto:Token: None 2023-05-24T21:20:29.104 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:20:29 GMT /qskrps-43/?policy 2023-05-24T21:20:29.105 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:v1tF/FhVTRgXev/SDAaC7EDSlQg= 2023-05-24T21:20:29.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:20:29 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:v1tF/FhVTRgXev/SDAaC7EDSlQg=', 'Content-Length': '0'} 2023-05-24T21:20:29.106 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000122d81440a4d7df3-00646e7f9d-1162-b2'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Wed, 24 May 2023 21:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:20:29.107 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_policy ... ok 2023-05-24T21:20:29.108 INFO:rgw_multi.tests:create bucket zone=a1 name=qskrps-44 2023-05-24T21:20:29.108 DEBUG:boto:path=/qskrps-44/ 2023-05-24T21:20:29.109 DEBUG:boto:auth_path=/qskrps-44/ 2023-05-24T21:20:29.109 DEBUG:boto:Method: PUT 2023-05-24T21:20:29.109 DEBUG:boto:Path: /qskrps-44/ 2023-05-24T21:20:29.110 DEBUG:boto:Data: 2023-05-24T21:20:29.110 DEBUG:boto:Headers: {} 2023-05-24T21:20:29.110 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:20:29.111 DEBUG:boto:Port: 8000 2023-05-24T21:20:29.111 DEBUG:boto:Params: {} 2023-05-24T21:20:29.111 DEBUG:boto:Token: None 2023-05-24T21:20:29.112 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:20:29 GMT /qskrps-44/ 2023-05-24T21:20:29.112 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ZhMR5snXfrBDOgXOg/bNFpFdyNA= 2023-05-24T21:20:29.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:20:29 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ZhMR5snXfrBDOgXOg/bNFpFdyNA=', 'Content-Length': '0'} 2023-05-24T21:20:29.130 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000316bd1e0759497d-00646e7f9d-1627-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:20:29.131 INFO:rgw_multi.tests:create bucket zone=a2 name=qskrps-45 2023-05-24T21:20:29.131 DEBUG:boto:path=/qskrps-45/ 2023-05-24T21:20:29.132 DEBUG:boto:auth_path=/qskrps-45/ 2023-05-24T21:20:29.132 DEBUG:boto:Method: PUT 2023-05-24T21:20:29.132 DEBUG:boto:Path: /qskrps-45/ 2023-05-24T21:20:29.133 DEBUG:boto:Data: 2023-05-24T21:20:29.133 DEBUG:boto:Headers: {} 2023-05-24T21:20:29.133 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:20:29.134 DEBUG:boto:Port: 8001 2023-05-24T21:20:29.134 DEBUG:boto:Params: {} 2023-05-24T21:20:29.134 DEBUG:boto:Token: None 2023-05-24T21:20:29.135 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:20:29 GMT /qskrps-45/ 2023-05-24T21:20:29.135 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:oktYIqoy9jjTEqHMQpCQHPHGgv8= 2023-05-24T21:20:29.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:20:29 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:oktYIqoy9jjTEqHMQpCQHPHGgv8=', 'Content-Length': '0'} 2023-05-24T21:20:29.172 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000570e9b9da3dc359-00646e7f9d-1625-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:20:29.173 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:20:29.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:20:29.174 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:20:29.225 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:20:29.225 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:20:29.266 INFO:teuthology.orchestra.run.smithi071.stderr:No --period given, using current period=5c04936b-70ea-4415-b4eb-3aad3dc612e5 2023-05-24T21:20:29.280 INFO:teuthology.orchestra.run.smithi071.stdout:[ 2023-05-24T21:20:29.280 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:29.280 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963229.148253_462.1", 2023-05-24T21:20:29.280 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:20:29.148253Z" 2023-05-24T21:20:29.280 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:29.280 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:29.281 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:20:29.281 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:06:32.333844Z" 2023-05-24T21:20:29.281 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:29.281 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:29.281 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:20:29.281 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:20:29.282 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:29.282 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:29.282 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:20:29.282 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:20:29.282 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:29.282 INFO:teuthology.orchestra.run.smithi071.stdout:] 2023-05-24T21:20:29.283 DEBUG:rgw_multi.tests:master meta markers={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:20:29.284 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-24T21:20:29.284 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:20:29.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:20:29.285 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:20:29.333 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:20:29.333 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:20:29.363 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:20:29.357+0000 7f3023c58780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:20:29.364 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:20:29.358+0000 7f3023c58780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:20:29.377 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:20:29.377 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:20:29.378 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:20:29.378 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:20:29.378 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:20:29.378 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:20:29.378 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:20:29.378 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:29.379 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:20:29.379 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:29.379 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:20:29.379 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:20:29.379 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:20:29.379 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962972.907627_296.1", 2023-05-24T21:20:29.379 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:20:29.380 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:20:29.380 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:20:29.380 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:16:12.907627Z", 2023-05-24T21:20:29.380 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:20:29.380 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:29.380 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:29.380 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:29.381 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:20:29.381 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:20:29.381 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:20:29.381 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:20:29.381 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:20:29.381 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:20:29.381 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:20:29.382 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:20:29.382 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:20:29.382 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:29.382 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:29.382 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:29.382 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:20:29.383 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:20:29.383 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:20:29.383 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:20:29.383 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:20:29.383 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:20:29.383 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:20:29.383 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:20:29.384 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:20:29.384 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:29.384 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:29.384 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:29.384 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:20:29.384 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:20:29.385 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:20:29.385 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:20:29.385 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:20:29.385 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:20:29.385 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:20:29.385 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:20:29.385 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:20:29.386 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:29.386 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:29.386 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:29.386 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:29.386 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:20:29.386 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:20:29.386 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:20:29.387 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:29.387 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:20:29Z" 2023-05-24T21:20:29.387 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:20:29.388 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962972.907627_296.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:16:12.907627Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:20:29Z" } 2023-05-24T21:20:29.388 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962972.907627_296.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:16:12.907627Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:20:29.388 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:20:29.389 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962972.907627_296.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:20:29.389 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684962972.907627_296.1 2023-05-24T21:20:34.390 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:20:34.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:20:34.391 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:20:34.449 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:20:34.450 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:20:34.504 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:20:34.498+0000 7f22c3960780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:20:34.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:20:34.499+0000 7f22c3960780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:20:34.526 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:20:34.526 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:20:34.526 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:20:34.527 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:20:34.527 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:20:34.527 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:20:34.527 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:20:34.527 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:34.527 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:20:34.528 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:34.528 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:20:34.528 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:20:34.528 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:20:34.528 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962972.907627_296.1", 2023-05-24T21:20:34.528 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:20:34.528 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:20:34.529 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:20:34.529 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:16:12.907627Z", 2023-05-24T21:20:34.529 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:20:34.529 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:34.529 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:34.529 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:34.530 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:20:34.530 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:20:34.530 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:20:34.530 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:20:34.530 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:20:34.530 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:20:34.530 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:20:34.531 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:20:34.531 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:20:34.531 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:34.531 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:34.531 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:34.531 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:20:34.531 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:20:34.532 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:20:34.532 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:20:34.532 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:20:34.532 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:20:34.532 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:20:34.532 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:20:34.532 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:20:34.533 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:34.533 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:34.533 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:34.533 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:20:34.533 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:20:34.533 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:20:34.533 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:20:34.534 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:20:34.534 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:20:34.534 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:20:34.534 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:20:34.534 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:20:34.534 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:34.534 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:34.535 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:34.535 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:34.535 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:20:34.535 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:20:34.535 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:20:34.535 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:34.535 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:20:34Z" 2023-05-24T21:20:34.536 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:20:34.536 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962972.907627_296.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:16:12.907627Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:20:34Z" } 2023-05-24T21:20:34.537 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962972.907627_296.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:16:12.907627Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:20:34.537 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:20:34.538 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962972.907627_296.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:20:34.538 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684962972.907627_296.1 2023-05-24T21:20:39.539 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:20:39.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:20:39.540 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:20:39.598 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:20:39.598 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:20:39.650 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:20:39.644+0000 7fd72930c780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:20:39.651 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:20:39.645+0000 7fd72930c780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:20:39.663 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:20:39.664 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:20:39.664 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:20:39.664 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:20:39.664 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:20:39.664 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:20:39.664 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:20:39.665 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:39.665 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:20:39.665 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:39.665 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:20:39.665 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:20:39.665 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:20:39.665 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962972.907627_296.1", 2023-05-24T21:20:39.666 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:20:39.666 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:20:39.666 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:20:39.666 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:16:12.907627Z", 2023-05-24T21:20:39.666 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:20:39.666 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:39.667 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:39.667 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:39.667 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:20:39.667 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:20:39.667 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:20:39.667 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:20:39.667 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:20:39.668 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:20:39.668 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:20:39.668 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:20:39.668 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:20:39.668 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:39.668 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:39.669 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:39.669 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:20:39.669 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:20:39.669 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:20:39.669 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:20:39.669 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:20:39.669 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:20:39.670 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:20:39.670 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:20:39.670 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:20:39.670 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:39.670 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:39.670 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:39.670 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:20:39.671 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:20:39.671 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:20:39.671 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:20:39.671 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:20:39.671 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:20:39.671 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:20:39.671 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:20:39.672 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:20:39.672 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:39.672 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:39.672 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:39.672 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:39.672 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:20:39.672 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:20:39.673 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:20:39.673 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:39.673 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:20:39Z" 2023-05-24T21:20:39.673 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:20:39.674 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962972.907627_296.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:16:12.907627Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:20:39Z" } 2023-05-24T21:20:39.674 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962972.907627_296.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:16:12.907627Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:20:39.675 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:20:39.675 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962972.907627_296.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:20:39.675 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684962972.907627_296.1 2023-05-24T21:20:44.677 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:20:44.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:20:44.678 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:20:44.735 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:20:44.735 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:20:44.780 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:20:44.774+0000 7f3df0d41780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:20:44.781 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:20:44.775+0000 7f3df0d41780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:20:44.802 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:20:44.803 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:20:44.803 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:20:44.803 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:20:44.803 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:20:44.803 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:20:44.804 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:20:44.804 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:44.804 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:20:44.804 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:44.804 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:20:44.804 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:20:44.805 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:20:44.805 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962972.907627_296.1", 2023-05-24T21:20:44.805 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:20:44.805 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:20:44.805 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:20:44.805 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:16:12.907627Z", 2023-05-24T21:20:44.805 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:20:44.806 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:44.806 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:44.806 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:44.806 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:20:44.806 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:20:44.807 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:20:44.807 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:20:44.807 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:20:44.807 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:20:44.807 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:20:44.807 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:20:44.808 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:20:44.808 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:44.808 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:44.808 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:44.808 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:20:44.808 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:20:44.809 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:20:44.809 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:20:44.809 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:20:44.809 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:20:44.809 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:20:44.809 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:20:44.810 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:20:44.810 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:44.810 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:44.810 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:44.810 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:20:44.810 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:20:44.810 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:20:44.810 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:20:44.811 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:20:44.811 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:20:44.811 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:20:44.811 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:20:44.811 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:20:44.811 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:44.812 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:44.812 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:44.812 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:44.812 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:20:44.812 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:20:44.812 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:20:44.813 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:44.813 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:20:44Z" 2023-05-24T21:20:44.813 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:20:44.814 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962972.907627_296.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:16:12.907627Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:20:44Z" } 2023-05-24T21:20:44.814 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962972.907627_296.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:16:12.907627Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:20:44.815 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:20:44.815 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962972.907627_296.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:20:44.815 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684962972.907627_296.1 2023-05-24T21:20:49.817 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:20:49.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:20:49.818 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:20:49.874 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:20:49.874 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:20:49.911 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:20:49.904+0000 7f71b1c02780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:20:49.912 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:20:49.905+0000 7f71b1c02780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:20:49.925 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:20:49.925 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:20:49.926 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:20:49.926 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:20:49.926 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:20:49.926 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:20:49.926 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:20:49.926 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:49.927 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:20:49.927 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:49.927 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:20:49.927 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:20:49.927 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:20:49.927 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962972.907627_296.1", 2023-05-24T21:20:49.927 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:20:49.928 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:20:49.928 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:20:49.928 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:16:12.907627Z", 2023-05-24T21:20:49.928 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:20:49.929 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:49.929 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:49.929 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:49.929 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:20:49.929 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:20:49.929 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:20:49.930 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:20:49.930 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:20:49.930 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:20:49.930 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:20:49.930 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:20:49.930 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:20:49.931 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:49.931 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:49.931 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:49.931 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:20:49.931 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:20:49.931 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:20:49.932 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:20:49.932 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:20:49.932 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:20:49.932 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:20:49.932 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:20:49.932 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:20:49.932 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:49.933 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:49.933 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:49.933 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:20:49.933 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:20:49.933 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:20:49.934 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:20:49.934 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:20:49.934 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:20:49.934 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:20:49.934 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:20:49.934 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:20:49.935 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:49.935 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:49.935 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:49.935 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:49.935 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:20:49.935 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:20:49.935 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:20:49.936 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:49.936 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:20:49Z" 2023-05-24T21:20:49.936 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:20:49.937 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962972.907627_296.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:16:12.907627Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:20:49Z" } 2023-05-24T21:20:49.937 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962972.907627_296.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:16:12.907627Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:20:49.938 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:20:49.938 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962972.907627_296.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:20:49.939 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684962972.907627_296.1 2023-05-24T21:20:54.939 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:20:54.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:20:54.940 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:20:54.998 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:20:54.998 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:20:55.053 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:20:55.046+0000 7fcdd4d27780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:20:55.054 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:20:55.048+0000 7fcdd4d27780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:20:55.075 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:20:55.075 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:20:55.075 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:20:55.075 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:20:55.076 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:20:55.076 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:20:55.076 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:20:55.076 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:55.076 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:20:55.076 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:55.077 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:20:55.077 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:20:55.077 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:20:55.077 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962972.907627_296.1", 2023-05-24T21:20:55.077 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:20:55.077 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:20:55.077 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:20:55.078 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:16:12.907627Z", 2023-05-24T21:20:55.078 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:20:55.078 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:55.078 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:55.078 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:55.078 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:20:55.079 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:20:55.079 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:20:55.079 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:20:55.079 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:20:55.079 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:20:55.079 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:20:55.079 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:20:55.080 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:20:55.080 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:55.080 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:55.080 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:55.080 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:20:55.080 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:20:55.080 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:20:55.081 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:20:55.081 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:20:55.081 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:20:55.081 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:20:55.081 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:20:55.081 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:20:55.081 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:55.082 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:55.082 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:20:55.082 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:20:55.082 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:20:55.082 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:20:55.082 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:20:55.082 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:20:55.083 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:20:55.083 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:20:55.083 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:20:55.083 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:20:55.083 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:55.083 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:20:55.083 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:20:55.084 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:55.084 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:20:55.084 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:20:55.084 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:20:55.084 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:20:55.084 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:20:55Z" 2023-05-24T21:20:55.085 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:20:55.085 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962972.907627_296.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:16:12.907627Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:20:55Z" } 2023-05-24T21:20:55.086 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962972.907627_296.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:16:12.907627Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:20:55.086 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:20:55.086 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962972.907627_296.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:20:55.087 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684962972.907627_296.1 2023-05-24T21:21:00.088 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:21:00.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:21:00.089 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:21:00.146 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:21:00.146 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:21:00.180 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:21:00.173+0000 7f4fa07b8780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:21:00.181 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:21:00.174+0000 7f4fa07b8780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:21:00.193 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:21:00.194 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:21:00.194 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:21:00.194 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:21:00.194 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:21:00.194 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:21:00.194 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:00.194 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:00.195 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:21:00.195 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:00.195 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:21:00.195 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:00.195 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:00.195 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962972.907627_296.1", 2023-05-24T21:21:00.195 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:00.196 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:21:00.196 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:00.196 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:16:12.907627Z", 2023-05-24T21:21:00.196 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:00.196 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:00.196 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:00.197 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:00.197 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:21:00.197 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:00.197 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:00.197 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:21:00.197 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:00.197 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:21:00.198 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:00.198 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:21:00.198 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:00.198 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:00.198 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:00.198 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:00.198 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:21:00.199 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:00.199 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:00.199 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:21:00.199 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:00.199 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:21:00.199 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:00.199 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:21:00.200 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:00.200 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:00.200 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:00.200 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:00.200 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:21:00.200 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:00.200 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:00.201 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:21:00.201 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:00.201 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:21:00.201 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:00.201 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:21:00.201 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:00.201 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:00.202 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:00.202 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:21:00.202 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:00.202 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:21:00.202 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:21:00.202 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:21:00.202 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:00.203 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:21:00Z" 2023-05-24T21:21:00.203 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:21:00.203 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962972.907627_296.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:16:12.907627Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:21:00Z" } 2023-05-24T21:21:00.204 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962972.907627_296.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:16:12.907627Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:21:00.204 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:21:00.205 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962972.907627_296.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:21:00.205 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684962972.907627_296.1 2023-05-24T21:21:05.206 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:21:05.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:21:05.207 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:21:05.265 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:21:05.265 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:21:05.310 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:21:05.304+0000 7fb5dd3a6780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:21:05.311 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:21:05.305+0000 7fb5dd3a6780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:21:05.333 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:21:05.334 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:21:05.334 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:21:05.334 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:21:05.334 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:21:05.334 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:21:05.335 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:05.335 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:05.335 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:21:05.335 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:05.335 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:21:05.335 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:05.335 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:05.336 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962972.907627_296.1", 2023-05-24T21:21:05.336 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:05.336 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:21:05.336 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:05.336 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:16:12.907627Z", 2023-05-24T21:21:05.336 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:05.337 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:05.337 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:05.337 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:05.337 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:21:05.337 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:05.338 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:05.338 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:21:05.338 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:05.338 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:21:05.339 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:05.339 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:21:05.339 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:05.339 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:05.339 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:05.339 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:05.339 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:21:05.340 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:05.340 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:05.340 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:21:05.340 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:05.340 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:21:05.340 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:05.340 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:21:05.341 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:05.341 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:05.341 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:05.341 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:05.341 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:21:05.341 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:05.341 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:05.342 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:21:05.342 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:05.342 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:21:05.342 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:05.342 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:21:05.342 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:05.342 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:05.343 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:05.343 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:21:05.343 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:05.343 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:21:05.343 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:21:05.343 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:21:05.343 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:05.344 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:21:05Z" 2023-05-24T21:21:05.344 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:21:05.345 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962972.907627_296.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:16:12.907627Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:21:05Z" } 2023-05-24T21:21:05.345 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962972.907627_296.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:16:12.907627Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:21:05.346 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:21:05.346 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962972.907627_296.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:21:05.346 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684962972.907627_296.1 2023-05-24T21:21:10.348 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:21:10.348 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:21:10.349 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:21:10.406 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:21:10.406 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:21:10.458 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:21:10.452+0000 7fb4b0a32780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:21:10.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:21:10.453+0000 7fb4b0a32780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:21:10.476 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:21:10.476 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:21:10.476 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:21:10.477 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:21:10.477 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:21:10.477 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:21:10.477 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:10.477 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:10.477 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:21:10.478 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:10.478 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:21:10.478 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:10.478 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:10.478 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962972.907627_296.1", 2023-05-24T21:21:10.478 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:10.478 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:21:10.479 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:10.479 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:16:12.907627Z", 2023-05-24T21:21:10.479 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:10.479 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:10.479 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:10.479 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:10.479 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:21:10.480 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:10.480 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:10.480 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:21:10.480 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:10.480 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:21:10.480 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:10.480 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:21:10.481 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:10.481 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:10.481 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:10.481 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:10.481 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:21:10.481 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:10.481 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:10.482 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:21:10.482 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:10.482 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:21:10.482 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:10.482 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:21:10.482 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:10.482 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:10.483 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:10.483 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:10.483 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:21:10.483 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:10.483 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:10.483 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:21:10.483 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:10.484 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:21:10.484 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:10.484 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:21:10.484 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:10.484 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:10.484 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:10.485 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:21:10.485 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:10.485 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:21:10.485 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:21:10.485 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:21:10.485 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:10.485 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:21:10Z" 2023-05-24T21:21:10.486 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:21:10.486 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962972.907627_296.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:16:12.907627Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:21:10Z" } 2023-05-24T21:21:10.487 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962972.907627_296.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:16:12.907627Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:21:10.487 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:21:10.488 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962972.907627_296.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:21:10.488 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684962972.907627_296.1 2023-05-24T21:21:15.489 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:21:15.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:21:15.490 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:21:15.546 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:21:15.546 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:21:15.579 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:21:15.574+0000 7f7390335780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:21:15.580 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:21:15.575+0000 7f7390335780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:21:15.594 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:21:15.595 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:21:15.595 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:21:15.595 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:21:15.595 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:21:15.595 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:21:15.595 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:15.596 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:15.596 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:21:15.596 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:15.596 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:21:15.596 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:15.596 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:15.596 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962972.907627_296.1", 2023-05-24T21:21:15.597 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:15.597 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:21:15.597 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:15.597 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:16:12.907627Z", 2023-05-24T21:21:15.597 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:15.597 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:15.598 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:15.598 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:15.598 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:21:15.598 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:15.598 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:15.598 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:21:15.598 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:15.599 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:21:15.599 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:15.599 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:21:15.599 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:15.599 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:15.599 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:15.599 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:15.600 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:21:15.600 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:15.600 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:15.600 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:21:15.600 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:15.600 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:21:15.600 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:15.601 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:21:15.601 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:15.601 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:15.601 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:15.601 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:15.601 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:21:15.602 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:15.602 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:15.602 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:21:15.602 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:15.602 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:21:15.602 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:15.602 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:21:15.603 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:15.603 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:15.603 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:15.603 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:21:15.603 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:15.603 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:21:15.603 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:21:15.604 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:21:15.604 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:15.604 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:21:15Z" 2023-05-24T21:21:15.604 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:21:15.605 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962972.907627_296.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:16:12.907627Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:21:15Z" } 2023-05-24T21:21:15.605 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962972.907627_296.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:16:12.907627Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:21:15.606 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:21:15.606 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962972.907627_296.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:21:15.606 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684962972.907627_296.1 2023-05-24T21:21:20.608 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:21:20.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:21:20.609 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:21:20.666 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:21:20.666 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:21:20.719 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:21:20.714+0000 7fead54e2780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:21:20.720 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:21:20.715+0000 7fead54e2780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:21:20.735 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:21:20.736 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:21:20.736 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:21:20.736 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:21:20.736 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:21:20.736 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:21:20.736 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:20.737 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:20.737 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:21:20.737 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:20.737 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:21:20.737 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:20.737 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:20.738 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962972.907627_296.1", 2023-05-24T21:21:20.738 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:20.738 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:21:20.738 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:20.738 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:16:12.907627Z", 2023-05-24T21:21:20.738 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:20.738 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:20.739 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:20.739 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:20.739 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:21:20.739 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:20.739 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:20.739 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:21:20.739 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:20.740 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:21:20.740 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:20.740 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:21:20.740 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:20.740 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:20.740 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:20.740 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:20.741 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:21:20.741 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:20.741 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:20.741 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:21:20.741 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:20.741 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:21:20.741 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:20.742 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:21:20.742 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:20.742 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:20.742 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:20.742 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:20.742 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:21:20.742 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:20.743 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:20.743 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:21:20.743 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:20.743 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:21:20.743 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:20.743 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:21:20.743 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:20.744 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:20.744 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:20.744 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:21:20.744 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:20.744 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:21:20.744 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:21:20.744 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:21:20.745 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:20.745 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:21:20Z" 2023-05-24T21:21:20.745 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:21:20.746 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962972.907627_296.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:16:12.907627Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:21:20Z" } 2023-05-24T21:21:20.746 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962972.907627_296.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:16:12.907627Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:21:20.747 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:21:20.747 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962972.907627_296.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:21:20.747 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684962972.907627_296.1 2023-05-24T21:21:25.748 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:21:25.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:21:25.749 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:21:25.810 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:21:25.810 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:21:25.842 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:21:25.837+0000 7fcd9c901780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:21:25.843 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:21:25.838+0000 7fcd9c901780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:21:25.858 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:21:25.858 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:21:25.859 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:21:25.859 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:21:25.859 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:21:25.859 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:21:25.859 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:25.859 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:25.860 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:21:25.860 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:25.860 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:21:25.860 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:25.860 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:25.860 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962972.907627_296.1", 2023-05-24T21:21:25.860 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:25.861 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:21:25.861 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:25.861 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:16:12.907627Z", 2023-05-24T21:21:25.861 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:25.861 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:25.861 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:25.862 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:25.862 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:21:25.862 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:25.862 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:25.862 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:21:25.862 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:25.862 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:21:25.863 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:25.863 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:21:25.863 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:25.863 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:25.863 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:25.863 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:25.863 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:21:25.864 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:25.864 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:25.864 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:21:25.864 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:25.864 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:21:25.864 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:25.864 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:21:25.865 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:25.865 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:25.865 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:25.865 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:25.865 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:21:25.865 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:25.865 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:25.866 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:21:25.866 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:25.866 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:21:25.866 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:25.866 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:21:25.866 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:25.866 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:25.867 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:25.867 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:21:25.867 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:25.867 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:21:25.867 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:21:25.867 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:21:25.867 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:25.868 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:21:25Z" 2023-05-24T21:21:25.868 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:21:25.868 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962972.907627_296.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:16:12.907627Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:21:25Z" } 2023-05-24T21:21:25.869 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962972.907627_296.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:16:12.907627Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:21:25.869 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:21:25.870 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962972.907627_296.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:21:25.870 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684962972.907627_296.1 2023-05-24T21:21:30.871 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:21:30.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:21:30.872 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:21:30.929 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:21:30.929 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:21:30.978 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:21:30.973+0000 7f9119d99780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:21:30.979 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:21:30.974+0000 7f9119d99780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:21:30.996 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:21:30.997 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:21:30.997 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:21:30.997 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:21:30.997 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:21:30.998 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:21:30.998 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:30.998 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:30.998 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:21:30.998 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:30.999 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:21:30.999 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:30.999 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:30.999 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962972.907627_296.1", 2023-05-24T21:21:30.999 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:30.999 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:21:31.000 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:31.000 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:16:12.907627Z", 2023-05-24T21:21:31.000 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:31.000 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:31.000 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:31.001 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:31.001 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:21:31.001 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:31.001 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:31.001 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:21:31.001 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:31.002 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:21:31.002 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:31.002 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:21:31.002 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:31.002 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:31.002 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:31.002 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:31.003 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:21:31.003 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:31.003 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:31.003 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:21:31.003 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:31.003 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:21:31.003 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:31.004 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:21:31.004 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:31.004 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:31.004 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:31.004 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:31.004 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:21:31.004 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:31.005 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:31.005 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:21:31.005 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:31.005 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:21:31.005 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:31.005 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:21:31.005 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:31.006 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:31.006 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:31.006 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:21:31.006 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:31.006 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:21:31.006 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:21:31.006 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:21:31.007 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:31.007 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:21:30Z" 2023-05-24T21:21:31.007 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:21:31.008 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962972.907627_296.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:16:12.907627Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:21:30Z" } 2023-05-24T21:21:31.008 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962972.907627_296.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:16:12.907627Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:21:31.008 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:21:31.009 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962972.907627_296.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:21:31.009 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684962972.907627_296.1 2023-05-24T21:21:36.010 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:21:36.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:21:36.011 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:21:36.068 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:21:36.068 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:21:36.103 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:21:36.098+0000 7f52e6904780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:21:36.104 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:21:36.099+0000 7f52e6904780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:21:36.117 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:21:36.117 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:21:36.118 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:21:36.118 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:21:36.118 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:21:36.118 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:21:36.118 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:36.118 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:36.119 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:21:36.119 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:36.119 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:21:36.119 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:36.119 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:36.119 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962972.907627_296.1", 2023-05-24T21:21:36.120 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:36.120 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:21:36.120 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:36.120 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:16:12.907627Z", 2023-05-24T21:21:36.120 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:36.120 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:36.120 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:36.121 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:36.121 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:21:36.121 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:36.121 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:36.121 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:21:36.121 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:36.122 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:21:36.122 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:36.122 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:21:36.122 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:36.122 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:36.122 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:36.122 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:36.123 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:21:36.123 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:36.123 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:36.123 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:21:36.123 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:36.123 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:21:36.123 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:36.124 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:21:36.124 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:36.124 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:36.124 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:36.124 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:36.124 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:21:36.125 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:36.125 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:36.125 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:21:36.125 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:36.125 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:21:36.125 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:36.125 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:21:36.126 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:36.126 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:36.126 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:36.126 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:21:36.126 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:36.126 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:21:36.126 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:21:36.127 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:21:36.127 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:36.127 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:21:36Z" 2023-05-24T21:21:36.127 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:21:36.128 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962972.907627_296.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:16:12.907627Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:21:36Z" } 2023-05-24T21:21:36.128 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962972.907627_296.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:16:12.907627Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:21:36.129 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:21:36.129 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962972.907627_296.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:21:36.130 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684962972.907627_296.1 2023-05-24T21:21:41.131 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:21:41.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:21:41.132 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:21:41.189 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:21:41.189 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:21:41.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:21:41.236+0000 7f228d81b780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:21:41.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:21:41.237+0000 7f228d81b780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:21:41.265 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:21:41.266 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:21:41.266 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:21:41.266 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:21:41.266 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:21:41.266 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:21:41.267 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:41.267 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:41.267 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:21:41.267 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:41.267 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:21:41.267 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:41.267 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:41.268 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962972.907627_296.1", 2023-05-24T21:21:41.268 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:41.268 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:21:41.268 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:41.268 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:16:12.907627Z", 2023-05-24T21:21:41.268 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:41.268 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:41.269 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:41.269 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:41.269 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:21:41.269 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:41.269 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:41.269 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:21:41.269 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:41.270 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:21:41.270 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:41.270 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:21:41.270 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:41.270 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:41.270 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:41.270 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:41.271 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:21:41.271 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:41.271 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:41.271 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:21:41.271 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:41.271 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:21:41.271 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:41.272 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:21:41.272 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:41.272 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:41.272 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:41.272 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:41.272 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:21:41.273 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:41.273 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:41.273 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:21:41.273 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:41.273 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:21:41.273 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:41.273 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:21:41.273 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:41.274 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:41.274 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:41.274 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:21:41.274 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:41.274 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:21:41.274 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:21:41.274 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:21:41.275 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:41.275 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:21:41Z" 2023-05-24T21:21:41.275 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:21:41.276 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962972.907627_296.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:16:12.907627Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:21:41Z" } 2023-05-24T21:21:41.276 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962972.907627_296.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:16:12.907627Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:21:41.277 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:21:41.277 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962972.907627_296.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:21:41.277 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684962972.907627_296.1 2023-05-24T21:21:46.279 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:21:46.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:21:46.280 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:21:46.338 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:21:46.338 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:21:46.373 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:21:46.366+0000 7fbd0120c780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:21:46.374 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:21:46.367+0000 7fbd0120c780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:21:46.387 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:21:46.387 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:21:46.387 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:21:46.387 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:21:46.388 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:21:46.388 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:21:46.388 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:46.388 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:46.388 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:21:46.389 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:46.389 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:21:46.389 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:46.389 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:46.389 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962972.907627_296.1", 2023-05-24T21:21:46.389 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:46.390 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:21:46.390 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:46.390 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:16:12.907627Z", 2023-05-24T21:21:46.390 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:46.390 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:46.390 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:46.390 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:46.391 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:21:46.391 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:46.391 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:46.391 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:21:46.391 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:46.391 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:21:46.392 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:46.392 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:21:46.392 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:46.392 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:46.392 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:46.392 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:46.393 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:21:46.393 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:46.393 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:46.393 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:21:46.393 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:46.393 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:21:46.393 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:46.394 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:21:46.394 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:46.394 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:46.394 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:46.394 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:46.394 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:21:46.395 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:46.395 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:46.395 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:21:46.395 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:46.395 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:21:46.395 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:46.396 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:21:46.396 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:46.396 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:46.396 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:46.396 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:21:46.396 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:46.396 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:21:46.397 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:21:46.397 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:21:46.397 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:46.397 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:21:46Z" 2023-05-24T21:21:46.397 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:21:46.398 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962972.907627_296.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:16:12.907627Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:21:46Z" } 2023-05-24T21:21:46.398 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962972.907627_296.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:16:12.907627Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:21:46.399 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:21:46.399 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962972.907627_296.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:21:46.400 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684962972.907627_296.1 2023-05-24T21:21:51.401 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:21:51.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:21:51.402 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:21:51.459 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:21:51.459 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:21:51.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:21:51.499+0000 7f4a60c44780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:21:51.506 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:21:51.500+0000 7f4a60c44780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:21:51.528 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:21:51.528 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:21:51.528 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:21:51.529 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:21:51.529 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:21:51.529 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:21:51.529 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:51.529 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:51.529 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:21:51.530 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:51.530 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:21:51.530 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:51.530 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:51.530 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962972.907627_296.1", 2023-05-24T21:21:51.530 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:51.530 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:21:51.531 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:51.531 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:16:12.907627Z", 2023-05-24T21:21:51.531 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:51.531 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:51.531 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:51.531 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:51.532 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:21:51.532 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:51.532 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:51.532 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:21:51.532 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:51.532 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:21:51.532 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:51.533 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:21:51.533 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:51.533 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:51.533 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:51.533 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:51.533 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:21:51.534 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:51.534 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:51.534 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:21:51.534 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:51.534 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:21:51.534 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:51.534 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:21:51.535 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:51.535 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:51.535 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:51.535 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:51.535 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:21:51.535 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:51.535 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:51.536 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:21:51.536 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:51.536 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:21:51.536 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:51.536 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:21:51.536 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:51.537 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:51.537 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:51.537 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:21:51.537 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:51.537 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:21:51.537 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:21:51.537 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:21:51.538 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:51.538 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:21:51Z" 2023-05-24T21:21:51.538 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:21:51.539 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962972.907627_296.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:16:12.907627Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:21:51Z" } 2023-05-24T21:21:51.539 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962972.907627_296.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:16:12.907627Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:21:51.540 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:21:51.540 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962972.907627_296.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:21:51.540 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684962972.907627_296.1 2023-05-24T21:21:56.542 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:21:56.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:21:56.543 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:21:56.596 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:21:56.597 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:21:56.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:21:56.625+0000 7f7211c93780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:21:56.633 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:21:56.626+0000 7f7211c93780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:21:56.647 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:21:56.648 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:21:56.648 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:21:56.648 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:21:56.648 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:21:56.648 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:21:56.648 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:56.649 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:56.649 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:21:56.649 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:56.649 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:21:56.649 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:56.650 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:56.650 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962972.907627_296.1", 2023-05-24T21:21:56.650 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:56.650 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:21:56.650 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:56.651 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:16:12.907627Z", 2023-05-24T21:21:56.651 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:56.651 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:56.651 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:56.651 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:56.651 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:21:56.651 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:56.652 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:56.652 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:21:56.652 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:56.652 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:21:56.652 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:56.652 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:21:56.653 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:56.653 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:56.653 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:56.653 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:56.653 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:21:56.653 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:56.654 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:56.654 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:21:56.654 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:56.654 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:21:56.655 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:56.655 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:21:56.655 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:56.655 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:56.655 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:56.656 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:21:56.656 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:21:56.656 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:21:56.656 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:21:56.656 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:21:56.656 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:21:56.656 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:21:56.657 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:21:56.657 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:21:56.657 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:21:56.657 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:56.657 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:21:56.657 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:21:56.658 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:56.658 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:21:56.658 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:21:56.658 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:21:56.658 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:21:56.658 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:21:56Z" 2023-05-24T21:21:56.658 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:21:56.659 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962972.907627_296.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:16:12.907627Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:21:56Z" } 2023-05-24T21:21:56.660 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962972.907627_296.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:16:12.907627Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:21:56.660 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:21:56.661 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962972.907627_296.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:21:56.661 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684962972.907627_296.1 2023-05-24T21:22:01.662 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:22:01.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:22:01.663 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:22:01.720 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:22:01.759 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:22:01.768 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:22:01.762+0000 7f9dc62f7780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:22:01.769 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:22:01.763+0000 7f9dc62f7780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:22:01.790 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:22:01.790 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:22:01.790 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:22:01.790 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:22:01.790 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:22:01.791 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:22:01.791 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:01.791 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:01.791 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:22:01.791 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:01.791 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:22:01.792 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:01.792 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:01.792 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962972.907627_296.1", 2023-05-24T21:22:01.792 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:01.792 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:22:01.792 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:01.792 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:16:12.907627Z", 2023-05-24T21:22:01.793 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:01.793 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:01.793 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:01.793 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:01.793 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:22:01.793 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:01.793 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:01.794 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:22:01.794 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:01.794 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:22:01.794 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:01.794 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:22:01.794 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:01.794 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:01.795 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:01.795 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:01.795 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:22:01.795 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:01.795 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:01.795 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:22:01.795 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:01.796 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:22:01.796 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:01.796 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:22:01.796 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:01.796 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:01.796 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:01.797 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:01.797 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:22:01.797 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:01.797 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:01.797 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:22:01.797 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:01.797 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:22:01.798 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:01.798 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:22:01.798 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:01.798 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:01.798 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:01.798 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:22:01.798 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:01.799 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:22:01.799 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:22:01.799 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:22:01.799 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:01.799 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:22:01Z" 2023-05-24T21:22:01.799 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:22:01.800 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962972.907627_296.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:16:12.907627Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:22:01Z" } 2023-05-24T21:22:01.801 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962972.907627_296.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:16:12.907627Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:22:01.801 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:22:01.801 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962972.907627_296.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:22:01.802 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684962972.907627_296.1 2023-05-24T21:22:06.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:22:06.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:22:06.803 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:22:06.861 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:22:06.861 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:22:06.909 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:22:06.902+0000 7f4cbcb1e780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:22:06.910 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:22:06.903+0000 7f4cbcb1e780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:22:06.923 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:22:06.923 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:22:06.923 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:22:06.923 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:22:06.923 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:22:06.923 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:22:06.924 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:06.924 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:06.924 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:22:06.924 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:06.924 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:22:06.924 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:06.925 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:06.925 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962972.907627_296.1", 2023-05-24T21:22:06.925 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:06.925 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:22:06.925 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:06.925 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:16:12.907627Z", 2023-05-24T21:22:06.925 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:06.926 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:06.926 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:06.926 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:06.926 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:22:06.926 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:06.926 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:06.926 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:22:06.927 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:06.927 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:22:06.927 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:06.927 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:22:06.927 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:06.927 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:06.927 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:06.928 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:06.928 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:22:06.928 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:06.928 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:06.928 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:22:06.928 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:06.928 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:22:06.929 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:06.929 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:22:06.929 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:06.929 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:06.929 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:06.929 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:06.929 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:22:06.930 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:06.930 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:06.930 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:22:06.930 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:06.930 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:22:06.930 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:06.930 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:22:06.931 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:06.931 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:06.931 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:06.931 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:22:06.931 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:06.931 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:22:06.931 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:22:06.932 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:22:06.932 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:06.932 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:22:06Z" 2023-05-24T21:22:06.932 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:22:06.933 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962972.907627_296.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:16:12.907627Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:22:06Z" } 2023-05-24T21:22:06.933 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962972.907627_296.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:16:12.907627Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:22:06.934 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:22:06.934 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962972.907627_296.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:22:06.934 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684962972.907627_296.1 2023-05-24T21:22:11.935 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:22:11.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:22:11.937 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:22:11.993 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:22:11.993 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:22:12.027 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:22:12.021+0000 7f8cd0d9a780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:22:12.028 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:22:12.023+0000 7f8cd0d9a780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:22:12.049 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:22:12.049 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:22:12.049 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:22:12.049 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:22:12.050 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:22:12.050 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:22:12.050 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:12.050 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:12.050 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:22:12.050 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:12.051 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:22:12.051 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:12.051 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:12.051 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962972.907627_296.1", 2023-05-24T21:22:12.051 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:12.051 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:22:12.051 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:12.052 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:16:12.907627Z", 2023-05-24T21:22:12.052 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:12.052 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:12.052 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:12.052 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:12.052 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:22:12.053 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:12.053 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:12.053 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:22:12.053 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:12.053 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:22:12.053 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:12.053 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:22:12.054 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:12.054 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:12.054 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:12.054 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:12.054 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:22:12.054 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:12.054 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:12.055 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:22:12.055 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:12.055 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:22:12.055 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:12.055 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:22:12.055 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:12.055 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:12.056 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:12.056 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:12.056 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:22:12.056 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:12.056 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:12.056 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:22:12.057 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:12.057 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:22:12.057 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:12.057 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:22:12.057 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:12.057 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:12.057 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:12.058 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:22:12.058 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:12.058 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:22:12.058 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:22:12.058 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:22:12.058 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:12.058 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:22:12Z" 2023-05-24T21:22:12.059 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:22:12.059 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962972.907627_296.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:16:12.907627Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:22:12Z" } 2023-05-24T21:22:12.060 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962972.907627_296.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:16:12.907627Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:22:12.060 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:22:12.061 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962972.907627_296.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:22:12.061 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684962972.907627_296.1 2023-05-24T21:22:17.062 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:22:17.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:22:17.064 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:22:17.121 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:22:17.121 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:22:17.154 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:22:17.148+0000 7f7cb77d1780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:22:17.155 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:22:17.149+0000 7f7cb77d1780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:22:17.168 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:22:17.168 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:22:17.168 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:22:17.168 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:22:17.168 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:22:17.169 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:22:17.169 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:17.169 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:17.169 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:22:17.169 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:17.169 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:22:17.169 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:17.170 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:17.170 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962972.907627_296.1", 2023-05-24T21:22:17.170 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:17.170 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:22:17.170 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:17.170 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:16:12.907627Z", 2023-05-24T21:22:17.171 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:17.171 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:17.171 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:17.171 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:17.171 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:22:17.171 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:17.171 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:17.172 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:22:17.172 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:17.172 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:22:17.172 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:17.172 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:22:17.172 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:17.172 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:17.173 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:17.173 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:17.173 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:22:17.173 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:17.173 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:17.173 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:22:17.173 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:17.174 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:22:17.174 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:17.174 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:22:17.174 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:17.174 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:17.174 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:17.174 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:17.175 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:22:17.175 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:17.175 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:17.175 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:22:17.175 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:17.175 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:22:17.175 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:17.176 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:22:17.176 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:17.176 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:17.176 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:17.176 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:22:17.176 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:17.176 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:22:17.177 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:22:17.177 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:22:17.177 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:17.177 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:22:17Z" 2023-05-24T21:22:17.177 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:22:17.178 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962972.907627_296.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:16:12.907627Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:22:17Z" } 2023-05-24T21:22:17.178 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962972.907627_296.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:16:12.907627Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:22:17.179 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:22:17.179 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962972.907627_296.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:22:17.179 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684962972.907627_296.1 2023-05-24T21:22:22.180 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:22:22.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:22:22.181 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:22:22.238 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:22:22.239 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:22:22.292 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:22:22.286+0000 7f2c1f172780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:22:22.293 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:22:22.287+0000 7f2c1f172780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:22:22.310 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:22:22.310 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:22:22.311 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:22:22.311 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:22:22.311 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:22:22.311 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:22:22.311 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:22.311 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:22.311 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:22:22.312 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:22.312 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:22:22.312 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:22.312 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:22.312 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962972.907627_296.1", 2023-05-24T21:22:22.312 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:22.313 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:22:22.313 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:22.313 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:16:12.907627Z", 2023-05-24T21:22:22.313 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:22.313 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:22.313 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:22.313 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:22.314 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:22:22.314 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:22.314 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:22.314 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:22:22.314 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:22.314 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:22:22.314 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:22.315 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:22:22.315 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:22.315 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:22.315 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:22.315 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:22.315 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:22:22.316 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:22.316 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:22.316 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:22:22.316 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:22.316 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:22:22.316 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:22.316 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:22:22.317 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:22.317 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:22.317 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:22.317 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:22.317 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:22:22.317 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:22.318 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:22.318 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:22:22.318 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:22.318 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:22:22.318 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:22.318 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:22:22.318 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:22.318 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:22.319 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:22.319 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:22:22.319 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:22.319 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:22:22.319 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:22:22.319 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:22:22.320 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:22.320 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:22:22Z" 2023-05-24T21:22:22.320 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:22:22.321 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684962972.907627_296.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:16:12.907627Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:22:22Z" } 2023-05-24T21:22:22.321 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962972.907627_296.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:16:12.907627Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:22:22.322 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:22:22.322 DEBUG:rgw_multi.tests:sync_status={0: '1_1684962972.907627_296.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:22:22.322 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684962972.907627_296.1 2023-05-24T21:22:27.324 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:22:27.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:22:27.325 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:22:27.374 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:22:27.374 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:22:27.405 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:22:27.400+0000 7fe60f758780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:22:27.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:22:27.401+0000 7fe60f758780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:22:27.418 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:22:27.418 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:22:27.419 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:22:27.419 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:22:27.419 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:22:27.419 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:22:27.419 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:27.419 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:27.420 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:22:27.420 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:27.420 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:22:27.420 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:27.420 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:27.420 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963225.414960_403.1", 2023-05-24T21:22:27.420 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:27.421 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:22:27.421 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:27.421 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:20:25.414960Z", 2023-05-24T21:22:27.421 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:27.421 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:27.421 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:27.422 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:27.422 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:22:27.422 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:27.422 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:27.422 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:22:27.422 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:27.422 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:22:27.423 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:27.423 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:22:27.423 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:27.423 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:27.424 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:27.424 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:27.424 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:22:27.424 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:27.424 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:27.424 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:22:27.424 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:27.425 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:22:27.425 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:27.425 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:22:27.425 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:27.425 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:27.425 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:27.425 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:27.426 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:22:27.426 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:27.426 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:27.426 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:22:27.426 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:27.426 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:22:27.426 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:27.427 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:22:27.427 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:27.427 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:27.427 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:27.427 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:22:27.427 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:27.427 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:22:27.428 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:22:27.428 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:22:27.428 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:27.428 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:22:27Z" 2023-05-24T21:22:27.428 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:22:27.429 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684963225.414960_403.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:20:25.414960Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:22:27Z" } 2023-05-24T21:22:27.429 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963225.414960_403.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:20:25.414960Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:22:27.430 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:22:27.430 DEBUG:rgw_multi.tests:sync_status={0: '1_1684963225.414960_403.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:22:27.431 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684963225.414960_403.1 2023-05-24T21:22:32.432 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:22:32.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:22:32.539 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:22:32.590 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:22:32.590 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:22:32.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:22:32.626+0000 7ff6de95e780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:22:32.633 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:22:32.627+0000 7ff6de95e780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:22:32.645 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:22:32.646 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:22:32.646 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:22:32.646 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:22:32.646 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:22:32.646 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:22:32.646 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:32.646 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:32.647 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:22:32.647 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:32.647 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:22:32.647 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:32.647 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:32.647 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963225.680656_409.1", 2023-05-24T21:22:32.647 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:32.648 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:22:32.648 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:32.648 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:20:25.680656Z", 2023-05-24T21:22:32.648 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:32.648 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:32.648 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:32.649 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:32.649 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:22:32.649 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:32.649 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:32.649 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:22:32.649 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:32.649 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:22:32.650 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:32.650 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:22:32.650 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:32.650 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:32.650 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:32.650 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:32.650 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:22:32.651 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:32.651 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:32.651 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:22:32.651 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:32.651 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:22:32.651 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:32.651 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:22:32.652 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:32.652 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:32.652 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:32.652 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:32.652 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:22:32.652 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:32.652 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:32.653 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:22:32.653 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:32.653 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:22:32.653 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:32.653 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:22:32.653 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:32.653 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:32.654 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:32.654 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:22:32.654 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:32.654 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:22:32.654 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:22:32.654 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:22:32.654 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:32.655 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:22:32Z" 2023-05-24T21:22:32.655 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:22:32.655 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684963225.680656_409.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:20:25.680656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:22:32Z" } 2023-05-24T21:22:32.656 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963225.680656_409.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:20:25.680656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:22:32.656 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:22:32.657 DEBUG:rgw_multi.tests:sync_status={0: '1_1684963225.680656_409.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:22:32.657 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684963225.680656_409.1 2023-05-24T21:22:37.658 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:22:37.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:22:37.659 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:22:37.709 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:22:37.709 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:22:37.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:22:37.758+0000 7f19e4bc5780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:22:37.765 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:22:37.759+0000 7f19e4bc5780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:22:37.777 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:22:37.777 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:22:37.777 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:22:37.778 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:22:37.778 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:22:37.778 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:22:37.778 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:37.778 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:37.778 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:22:37.779 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:37.779 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:22:37.779 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:37.779 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:37.779 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963225.680656_409.1", 2023-05-24T21:22:37.779 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:37.780 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:22:37.780 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:37.780 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:20:25.680656Z", 2023-05-24T21:22:37.780 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:37.780 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:37.780 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:37.780 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:37.781 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:22:37.781 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:37.781 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:37.781 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:22:37.781 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:37.781 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:22:37.782 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:37.782 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:22:37.782 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:37.782 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:37.783 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:37.783 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:37.783 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:22:37.783 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:37.783 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:37.783 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:22:37.783 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:37.784 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:22:37.784 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:37.784 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:22:37.784 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:37.784 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:37.784 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:37.785 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:37.785 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:22:37.785 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:37.785 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:37.785 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:22:37.785 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:37.785 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:22:37.786 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:37.786 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:22:37.786 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:37.786 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:37.786 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:37.786 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:22:37.787 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:37.787 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:22:37.787 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:22:37.787 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:22:37.787 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:37.787 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:22:37Z" 2023-05-24T21:22:37.787 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:22:37.788 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684963225.680656_409.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:20:25.680656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:22:37Z" } 2023-05-24T21:22:37.789 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963225.680656_409.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:20:25.680656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:22:37.789 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:22:37.789 DEBUG:rgw_multi.tests:sync_status={0: '1_1684963225.680656_409.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:22:37.790 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684963225.680656_409.1 2023-05-24T21:22:42.790 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:22:42.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:22:42.792 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:22:42.845 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:22:42.845 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:22:42.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:22:42.873+0000 7efcae1be780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:22:42.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:22:42.874+0000 7efcae1be780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:22:42.896 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:22:42.896 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:22:42.896 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:22:42.896 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:22:42.896 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:22:42.897 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:22:42.897 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:42.897 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:42.897 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:22:42.897 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:42.897 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:22:42.897 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:42.898 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:42.898 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963225.410726_402.1", 2023-05-24T21:22:42.898 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:42.898 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:22:42.898 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:42.898 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:20:25.410726Z", 2023-05-24T21:22:42.898 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:42.899 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:42.899 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:42.899 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:42.899 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:22:42.899 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:42.899 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:42.899 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:22:42.900 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:42.900 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:22:42.900 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:42.900 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:22:42.900 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:42.900 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:42.901 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:42.901 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:42.901 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:22:42.901 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:42.901 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:42.901 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:22:42.901 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:42.902 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:22:42.902 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:42.902 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:22:42.902 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:42.902 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:42.902 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:42.902 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:42.903 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:22:42.903 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:42.903 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:42.903 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:22:42.903 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:42.903 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:22:42.903 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:42.904 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:22:42.904 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:42.904 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:42.904 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:42.904 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:22:42.904 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:42.904 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:22:42.905 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:22:42.905 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:22:42.905 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:42.905 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:22:42Z" 2023-05-24T21:22:42.905 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:22:42.906 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684963225.410726_402.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:20:25.410726Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:22:42Z" } 2023-05-24T21:22:42.906 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963225.410726_402.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:20:25.410726Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:22:42.907 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:22:42.907 DEBUG:rgw_multi.tests:sync_status={0: '1_1684963225.410726_402.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:22:42.908 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684963225.410726_402.1 2023-05-24T21:22:47.909 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:22:47.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:22:47.910 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:22:47.961 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:22:47.962 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:22:47.996 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:22:47.989+0000 7f2c1cbbf780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:22:47.996 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:22:47.990+0000 7f2c1cbbf780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:22:48.008 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:22:48.009 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:22:48.009 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:22:48.009 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:22:48.009 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:22:48.009 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:22:48.009 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:48.010 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:48.010 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:22:48.010 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:48.010 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:22:48.010 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:48.010 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:48.010 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963225.487974_408.1", 2023-05-24T21:22:48.011 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:48.011 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:22:48.011 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:48.011 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:20:25.487974Z", 2023-05-24T21:22:48.011 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:48.011 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:48.012 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:48.012 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:48.012 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:22:48.012 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:48.012 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:48.012 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:22:48.012 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:48.013 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:22:48.013 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:48.013 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:22:48.013 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:48.013 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:48.013 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:48.013 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:48.014 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:22:48.014 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:48.014 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:48.014 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:22:48.014 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:48.014 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:22:48.014 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:48.015 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:22:48.015 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:48.015 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:48.015 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:48.015 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:48.015 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:22:48.015 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:48.016 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:48.016 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:22:48.016 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:48.016 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:22:48.016 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:48.016 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:22:48.017 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:48.017 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:48.017 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:48.017 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:22:48.017 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:48.017 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:22:48.017 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:22:48.018 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:22:48.018 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:48.018 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:22:47Z" 2023-05-24T21:22:48.018 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:22:48.019 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684963225.487974_408.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:20:25.487974Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:22:47Z" } 2023-05-24T21:22:48.019 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963225.487974_408.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:20:25.487974Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:22:48.020 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:22:48.020 DEBUG:rgw_multi.tests:sync_status={0: '1_1684963225.487974_408.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:22:48.020 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684963225.487974_408.1 2023-05-24T21:22:53.021 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:22:53.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:22:53.022 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:22:53.073 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:22:53.073 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:22:53.103 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:22:53.096+0000 7f52ac286780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:22:53.103 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:22:53.097+0000 7f52ac286780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:22:53.117 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:22:53.117 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:22:53.117 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:22:53.117 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:22:53.117 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:22:53.118 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:22:53.118 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:53.118 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:53.118 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:22:53.118 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:53.118 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:22:53.118 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:53.119 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:53.119 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963225.704789_414.1", 2023-05-24T21:22:53.119 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:53.119 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:22:53.119 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:53.119 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:20:25.704789Z", 2023-05-24T21:22:53.119 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:53.120 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:53.120 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:53.120 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:53.120 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:22:53.120 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:53.120 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:53.121 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:22:53.121 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:53.121 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:22:53.121 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:53.121 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:22:53.121 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:53.121 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:53.122 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:53.122 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:53.122 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:22:53.122 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:53.122 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:53.122 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:22:53.122 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:53.123 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:22:53.123 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:53.123 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:22:53.123 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:53.123 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:53.123 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:53.123 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:53.124 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:22:53.124 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:53.124 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:53.124 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:22:53.124 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:53.124 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:22:53.124 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:53.125 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:22:53.125 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:53.125 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:53.125 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:53.125 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:22:53.125 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:53.125 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:22:53.126 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:22:53.126 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:22:53.126 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:53.126 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:22:53Z" 2023-05-24T21:22:53.126 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:22:53.127 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684963225.704789_414.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:20:25.704789Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:22:53Z" } 2023-05-24T21:22:53.127 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963225.704789_414.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:20:25.704789Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:22:53.128 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:22:53.128 DEBUG:rgw_multi.tests:sync_status={0: '1_1684963225.704789_414.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:22:53.128 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684963225.704789_414.1 2023-05-24T21:22:58.130 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:22:58.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:22:58.131 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:22:58.188 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:22:58.188 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:22:58.241 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:22:58.235+0000 7f3b462a8780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:22:58.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:22:58.236+0000 7f3b462a8780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:22:58.263 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:22:58.263 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:22:58.263 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:22:58.264 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:22:58.264 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:22:58.264 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:22:58.264 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:58.264 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:58.264 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:22:58.265 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:58.265 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:22:58.265 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:58.265 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:58.265 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963225.680656_409.1", 2023-05-24T21:22:58.265 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:58.265 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:22:58.266 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:58.266 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:20:25.680656Z", 2023-05-24T21:22:58.266 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:58.266 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:58.266 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:58.266 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:58.266 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:22:58.267 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:58.267 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:58.267 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:22:58.267 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:58.267 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:22:58.267 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:58.267 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:22:58.268 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:58.268 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:58.268 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:58.268 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:58.268 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:22:58.268 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:58.269 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:58.269 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:22:58.269 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:58.269 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:22:58.269 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:58.269 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:22:58.269 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:58.269 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:58.270 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:58.270 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:22:58.270 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:22:58.270 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:22:58.270 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:22:58.270 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:22:58.270 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:22:58.271 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:22:58.271 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:22:58.271 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:22:58.271 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:22:58.271 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:58.271 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:22:58.271 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:22:58.272 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:58.272 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:22:58.272 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:22:58.272 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:22:58.272 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:22:58.272 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:22:58Z" 2023-05-24T21:22:58.273 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:22:58.273 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684963225.680656_409.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:20:25.680656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:22:58Z" } 2023-05-24T21:22:58.274 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963225.680656_409.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:20:25.680656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:22:58.274 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:22:58.275 DEBUG:rgw_multi.tests:sync_status={0: '1_1684963225.680656_409.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:22:58.275 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684963225.680656_409.1 2023-05-24T21:23:03.276 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:23:03.277 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:23:03.277 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:23:03.347 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:23:03.347 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:23:03.420 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:23:03.414+0000 7f537ea15780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:23:03.421 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:23:03.414+0000 7f537ea15780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:23:03.438 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:23:03.438 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:23:03.438 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:23:03.438 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:23:03.438 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:23:03.439 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:23:03.439 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:03.439 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:03.439 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:23:03.439 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:03.439 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:23:03.440 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:03.440 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:03.440 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963225.822123_421.1", 2023-05-24T21:23:03.440 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:03.440 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:23:03.441 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:03.441 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:20:25.822123Z", 2023-05-24T21:23:03.441 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:03.441 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:03.441 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:03.441 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:03.441 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:23:03.442 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:03.442 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:03.442 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:23:03.442 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:03.442 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:23:03.442 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:03.442 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:23:03.443 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:03.443 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:03.443 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:03.443 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:03.443 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:23:03.443 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:03.444 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:03.444 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:23:03.444 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:03.444 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:23:03.444 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:03.444 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:23:03.444 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:03.445 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:03.445 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:03.445 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:03.445 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:23:03.445 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:03.445 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:03.445 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:23:03.446 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:03.446 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:23:03.446 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:03.446 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:23:03.446 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:03.446 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:03.446 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:03.447 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:23:03.447 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:03.447 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:23:03.447 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:23:03.447 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:23:03.447 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:03.447 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:23:03Z" 2023-05-24T21:23:03.448 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:23:03.448 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684963225.822123_421.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:20:25.822123Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:23:03Z" } 2023-05-24T21:23:03.449 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963225.822123_421.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:20:25.822123Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:23:03.449 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:23:03.450 DEBUG:rgw_multi.tests:sync_status={0: '1_1684963225.822123_421.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:23:03.450 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684963225.822123_421.1 2023-05-24T21:23:08.451 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:23:08.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:23:08.452 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:23:08.508 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:23:08.509 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:23:08.565 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:23:08.560+0000 7f0eb5ce2780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:23:08.566 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:23:08.561+0000 7f0eb5ce2780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:23:08.587 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:23:08.588 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:23:08.588 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:23:08.588 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:23:08.588 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:23:08.588 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:23:08.589 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:08.589 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:08.589 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:23:08.589 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:08.589 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:23:08.589 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:08.589 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:08.590 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963225.680656_409.1", 2023-05-24T21:23:08.590 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:08.590 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:23:08.590 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:08.590 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:20:25.680656Z", 2023-05-24T21:23:08.590 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:08.590 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:08.591 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:08.591 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:08.591 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:23:08.591 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:08.591 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:08.591 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:23:08.592 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:08.592 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:23:08.592 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:08.592 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:23:08.592 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:08.592 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:08.592 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:08.593 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:08.593 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:23:08.593 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:08.593 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:08.593 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:23:08.593 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:08.593 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:23:08.594 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:08.594 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:23:08.594 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:08.594 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:08.594 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:08.594 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:08.594 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:23:08.595 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:08.595 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:08.595 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:23:08.595 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:08.595 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:23:08.595 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:08.595 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:23:08.596 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:08.596 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:08.596 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:08.596 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:23:08.596 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:08.596 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:23:08.596 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:23:08.597 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:23:08.597 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:08.597 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:23:08Z" 2023-05-24T21:23:08.597 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:23:08.598 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684963225.680656_409.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:20:25.680656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:23:08Z" } 2023-05-24T21:23:08.598 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963225.680656_409.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:20:25.680656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:23:08.599 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:23:08.599 DEBUG:rgw_multi.tests:sync_status={0: '1_1684963225.680656_409.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:23:08.600 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684963225.680656_409.1 2023-05-24T21:23:13.601 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:23:13.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:23:13.602 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:23:13.659 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:23:13.660 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:23:13.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:23:13.688+0000 7f26ecbd3780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:23:13.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:23:13.689+0000 7f26ecbd3780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:23:13.708 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:23:13.708 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:23:13.708 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:23:13.708 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:23:13.709 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:23:13.709 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:23:13.709 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:13.709 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:13.709 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:23:13.709 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:13.710 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:23:13.710 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:13.710 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:13.710 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963225.429748_404.1", 2023-05-24T21:23:13.710 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:13.710 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:23:13.710 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:13.711 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:20:25.429748Z", 2023-05-24T21:23:13.711 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:13.711 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:13.711 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:13.711 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:13.712 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:23:13.712 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:13.712 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:13.712 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:23:13.712 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:13.712 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:23:13.713 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:13.713 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:23:13.713 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:13.713 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:13.713 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:13.713 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:13.713 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:23:13.714 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:13.714 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:13.714 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:23:13.714 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:13.714 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:23:13.714 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:13.714 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:23:13.715 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:13.715 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:13.715 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:13.715 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:13.715 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:23:13.715 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:13.715 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:13.716 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:23:13.716 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:13.716 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:23:13.716 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:13.716 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:23:13.716 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:13.716 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:13.717 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:13.717 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:23:13.717 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:13.717 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:23:13.717 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:23:13.717 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:23:13.718 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:13.718 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:23:13Z" 2023-05-24T21:23:13.718 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:23:13.719 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684963225.429748_404.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:20:25.429748Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:23:13Z" } 2023-05-24T21:23:13.719 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963225.429748_404.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:20:25.429748Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:23:13.720 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:23:13.720 DEBUG:rgw_multi.tests:sync_status={0: '1_1684963225.429748_404.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:23:13.720 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684963225.429748_404.1 2023-05-24T21:23:18.722 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:23:18.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:23:18.723 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:23:18.775 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:23:18.775 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:23:18.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:23:18.806+0000 7fbd844a5780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:23:18.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:23:18.807+0000 7fbd844a5780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:23:18.827 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:23:18.827 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:23:18.827 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:23:18.827 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:23:18.828 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:23:18.828 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:23:18.828 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:18.828 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:18.828 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:23:18.828 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:18.829 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:23:18.829 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:18.829 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:18.829 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963225.680656_409.1", 2023-05-24T21:23:18.829 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:18.829 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:23:18.829 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:18.830 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:20:25.680656Z", 2023-05-24T21:23:18.830 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:18.830 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:18.830 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:18.830 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:18.830 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:23:18.830 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:18.831 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:18.831 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:23:18.831 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:18.831 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:23:18.831 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:18.831 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:23:18.831 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:18.832 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:18.832 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:18.832 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:18.832 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:23:18.832 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:18.832 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:18.832 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:23:18.833 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:18.833 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:23:18.833 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:18.833 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:23:18.833 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:18.833 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:18.833 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:18.834 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:18.834 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:23:18.834 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:18.834 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:18.834 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:23:18.834 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:18.834 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:23:18.835 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:18.835 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:23:18.835 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:18.835 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:18.835 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:18.835 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:23:18.836 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:18.836 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:23:18.836 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:23:18.836 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:23:18.836 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:18.836 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:23:18Z" 2023-05-24T21:23:18.836 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:23:18.837 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684963225.680656_409.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:20:25.680656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:23:18Z" } 2023-05-24T21:23:18.838 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963225.680656_409.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:20:25.680656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:23:18.838 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:23:18.838 DEBUG:rgw_multi.tests:sync_status={0: '1_1684963225.680656_409.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:23:18.839 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684963225.680656_409.1 2023-05-24T21:23:23.840 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:23:23.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:23:23.841 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:23:23.893 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:23:23.893 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:23:23.931 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:23:23.925+0000 7f9568acc780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:23:23.932 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:23:23.926+0000 7f9568acc780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:23:23.945 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:23:23.945 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:23:23.945 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:23:23.945 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:23:23.945 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:23:23.946 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:23:23.946 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:23.946 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:23.946 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:23:23.946 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:23.946 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:23:23.947 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:23.947 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:23.947 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963225.414960_403.1", 2023-05-24T21:23:23.947 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:23.947 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:23:23.947 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:23.947 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:20:25.414960Z", 2023-05-24T21:23:23.948 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:23.948 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:23.948 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:23.948 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:23.948 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:23:23.948 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:23.948 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:23.949 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:23:23.949 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:23.949 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:23:23.949 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:23.949 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:23:23.949 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:23.949 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:23.950 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:23.950 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:23.950 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:23:23.950 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:23.950 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:23.950 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:23:23.950 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:23.951 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:23:23.951 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:23.951 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:23:23.951 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:23.951 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:23.951 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:23.952 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:23.952 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:23:23.952 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:23.952 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:23.952 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:23:23.952 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:23.952 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:23:23.953 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:23.953 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:23:23.953 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:23.953 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:23.953 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:23.953 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:23:23.953 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:23.954 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:23:23.954 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:23:23.954 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:23:23.954 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:23.954 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:23:23Z" 2023-05-24T21:23:23.954 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:23:23.955 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684963225.414960_403.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:20:25.414960Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:23:23Z" } 2023-05-24T21:23:23.956 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963225.414960_403.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:20:25.414960Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:23:23.956 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:23:23.956 DEBUG:rgw_multi.tests:sync_status={0: '1_1684963225.414960_403.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:23:23.957 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684963225.414960_403.1 2023-05-24T21:23:28.957 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:23:28.958 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:23:28.958 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:23:29.012 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:23:29.013 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:23:29.042 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:23:29.036+0000 7fd97282f780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:23:29.043 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:23:29.037+0000 7fd97282f780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:23:29.056 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:23:29.056 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:23:29.056 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:23:29.056 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:23:29.056 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:23:29.057 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:23:29.057 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:29.057 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:29.057 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:23:29.057 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:29.057 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:23:29.057 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:29.058 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:29.058 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963225.680656_409.1", 2023-05-24T21:23:29.058 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:29.058 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:23:29.058 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:29.058 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:20:25.680656Z", 2023-05-24T21:23:29.058 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:29.059 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:29.059 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:29.059 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:29.059 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:23:29.059 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:29.059 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:29.059 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:23:29.060 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:29.060 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:23:29.060 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:29.060 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:23:29.060 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:29.060 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:29.061 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:29.061 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:29.061 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:23:29.061 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:29.061 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:29.061 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:23:29.061 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:29.062 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:23:29.062 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:29.062 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:23:29.062 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:29.062 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:29.062 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:29.062 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:29.063 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:23:29.063 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:29.063 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:29.063 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:23:29.063 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:29.063 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:23:29.063 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:29.064 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:23:29.064 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:29.064 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:29.064 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:29.064 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:23:29.064 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:29.064 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:23:29.065 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:23:29.065 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:23:29.065 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:29.065 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:23:29Z" 2023-05-24T21:23:29.065 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:23:29.066 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684963225.680656_409.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:20:25.680656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:23:29Z" } 2023-05-24T21:23:29.066 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963225.680656_409.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:20:25.680656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:23:29.067 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:23:29.067 DEBUG:rgw_multi.tests:sync_status={0: '1_1684963225.680656_409.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:23:29.067 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684963225.680656_409.1 2023-05-24T21:23:34.068 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:23:34.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:23:34.070 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:23:34.126 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:23:34.127 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:23:34.160 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:23:34.155+0000 7f65075d6780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:23:34.161 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:23:34.156+0000 7f65075d6780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:23:34.175 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:23:34.175 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:23:34.175 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:23:34.175 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:23:34.175 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:23:34.176 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:23:34.176 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:34.176 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:34.176 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:23:34.176 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:34.177 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:23:34.177 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:34.177 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:34.177 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963225.777380_415.1", 2023-05-24T21:23:34.177 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:34.177 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:23:34.178 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:34.178 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:20:25.777380Z", 2023-05-24T21:23:34.178 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:34.178 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:34.178 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:34.178 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:34.179 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:23:34.179 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:34.179 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:34.179 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:23:34.179 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:34.179 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:23:34.180 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:34.180 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:23:34.180 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:34.180 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:34.180 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:34.180 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:34.181 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:23:34.181 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:34.181 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:34.181 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:23:34.181 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:34.181 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:23:34.182 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:34.182 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:23:34.182 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:34.182 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:34.182 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:34.182 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:34.183 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:23:34.183 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:34.183 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:34.183 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:23:34.183 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:34.183 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:23:34.184 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:34.184 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:23:34.184 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:34.184 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:34.184 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:34.184 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:23:34.185 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:34.185 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:23:34.185 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:23:34.185 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:23:34.185 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:34.185 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:23:34Z" 2023-05-24T21:23:34.185 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:23:34.186 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684963225.777380_415.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:20:25.777380Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:23:34Z" } 2023-05-24T21:23:34.187 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963225.777380_415.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:20:25.777380Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:23:34.187 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:23:34.188 DEBUG:rgw_multi.tests:sync_status={0: '1_1684963225.777380_415.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:23:34.188 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684963225.777380_415.1 2023-05-24T21:23:39.189 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:23:39.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:23:39.190 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:23:39.240 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:23:39.240 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:23:39.282 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:23:39.276+0000 7f311e8f8780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:23:39.283 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:23:39.277+0000 7f311e8f8780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:23:39.304 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:23:39.304 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:23:39.304 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:23:39.305 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:23:39.305 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:23:39.305 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:23:39.306 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:39.306 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:39.306 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:23:39.307 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:39.307 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:23:39.307 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:39.307 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:39.307 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963225.680656_409.1", 2023-05-24T21:23:39.307 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:39.308 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:23:39.308 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:39.308 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:20:25.680656Z", 2023-05-24T21:23:39.308 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:39.308 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:39.308 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:39.308 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:39.309 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:23:39.309 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:39.309 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:39.309 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:23:39.309 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:39.309 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:23:39.309 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:39.310 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:23:39.310 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:39.310 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:39.310 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:39.310 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:39.310 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:23:39.311 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:39.311 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:39.311 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:23:39.311 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:39.311 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:23:39.311 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:39.311 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:23:39.312 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:39.312 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:39.312 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:39.312 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:39.312 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:23:39.312 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:39.313 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:39.313 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:23:39.313 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:39.313 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:23:39.313 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:39.313 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:23:39.313 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:39.314 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:39.314 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:39.314 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:23:39.314 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:39.314 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:23:39.314 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:23:39.314 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:23:39.315 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:39.315 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:23:39Z" 2023-05-24T21:23:39.315 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:23:39.316 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684963225.680656_409.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:20:25.680656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:23:39Z" } 2023-05-24T21:23:39.316 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963225.680656_409.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:20:25.680656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:23:39.317 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:23:39.317 DEBUG:rgw_multi.tests:sync_status={0: '1_1684963225.680656_409.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:23:39.317 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684963225.680656_409.1 2023-05-24T21:23:44.319 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:23:44.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:23:44.320 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:23:44.373 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:23:44.373 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:23:44.423 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:23:44.417+0000 7fe99a7c3780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:23:44.424 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:23:44.418+0000 7fe99a7c3780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:23:44.437 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:23:44.437 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:23:44.437 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:23:44.437 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:23:44.438 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:23:44.438 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:23:44.438 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:44.438 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:44.438 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:23:44.438 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:44.439 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:23:44.439 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:44.439 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:44.439 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963225.429748_404.1", 2023-05-24T21:23:44.439 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:44.439 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:23:44.439 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:44.440 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:20:25.429748Z", 2023-05-24T21:23:44.440 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:44.440 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:44.440 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:44.440 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:44.440 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:23:44.440 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:44.441 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:44.441 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:23:44.441 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:44.441 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:23:44.441 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:44.441 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:23:44.441 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:44.442 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:44.442 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:44.442 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:44.442 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:23:44.442 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:44.442 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:44.442 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:23:44.443 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:44.443 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:23:44.443 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:44.443 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:23:44.443 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:44.443 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:44.443 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:44.444 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:44.444 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:23:44.444 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:44.444 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:44.444 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:23:44.444 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:44.444 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:23:44.445 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:44.445 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:23:44.445 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:44.445 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:44.445 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:44.445 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:23:44.445 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:44.446 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:23:44.446 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:23:44.446 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:23:44.446 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:44.446 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:23:44Z" 2023-05-24T21:23:44.446 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:23:44.447 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684963225.429748_404.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:20:25.429748Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:23:44Z" } 2023-05-24T21:23:44.448 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963225.429748_404.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:20:25.429748Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:23:44.448 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:23:44.448 DEBUG:rgw_multi.tests:sync_status={0: '1_1684963225.429748_404.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:23:44.449 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684963225.429748_404.1 2023-05-24T21:23:49.449 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:23:49.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:23:49.451 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:23:49.509 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:23:49.509 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:23:49.550 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:23:49.543+0000 7f522f084780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:23:49.551 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:23:49.544+0000 7f522f084780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:23:49.570 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:23:49.570 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:23:49.570 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:23:49.570 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:23:49.571 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:23:49.571 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:23:49.571 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:49.571 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:49.571 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:23:49.571 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:49.571 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:23:49.572 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:49.572 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:49.572 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963225.410726_402.1", 2023-05-24T21:23:49.572 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:49.572 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:23:49.572 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:49.572 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:20:25.410726Z", 2023-05-24T21:23:49.573 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:49.573 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:49.573 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:49.573 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:49.573 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:23:49.573 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:49.573 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:49.574 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:23:49.574 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:49.574 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:23:49.574 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:49.574 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:23:49.574 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:49.575 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:49.575 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:49.575 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:49.575 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:23:49.575 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:49.575 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:49.575 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:23:49.576 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:49.576 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:23:49.576 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:49.576 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:23:49.576 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:49.576 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:49.576 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:49.577 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:49.577 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:23:49.577 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:49.577 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:49.577 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:23:49.577 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:49.577 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:23:49.578 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:49.578 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:23:49.578 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:49.578 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:49.578 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:49.578 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:23:49.578 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:49.579 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:23:49.579 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:23:49.579 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:23:49.579 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:49.579 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:23:49Z" 2023-05-24T21:23:49.579 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:23:49.580 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684963225.410726_402.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:20:25.410726Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:23:49Z" } 2023-05-24T21:23:49.580 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963225.410726_402.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:20:25.410726Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:23:49.581 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:23:49.581 DEBUG:rgw_multi.tests:sync_status={0: '1_1684963225.410726_402.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:23:49.582 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684963225.410726_402.1 2023-05-24T21:23:54.582 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:23:54.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:23:54.584 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:23:54.640 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:23:54.641 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:23:54.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:23:54.710+0000 7f49ace78780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:23:54.722 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:23:54.715+0000 7f49ace78780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:23:54.770 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:23:54.770 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:23:54.770 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:23:54.770 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:23:54.770 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:23:54.771 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:23:54.771 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:54.771 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:54.771 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:23:54.771 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:54.771 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:23:54.772 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:54.772 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:54.772 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963225.680656_409.1", 2023-05-24T21:23:54.772 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:54.772 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:23:54.772 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:54.772 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:20:25.680656Z", 2023-05-24T21:23:54.773 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:54.773 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:54.773 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:54.773 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:54.773 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:23:54.773 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:54.773 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:54.774 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:23:54.774 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:54.774 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:23:54.774 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:54.774 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:23:54.774 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:54.774 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:54.775 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:54.775 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:54.775 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:23:54.775 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:54.775 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:54.775 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:23:54.776 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:54.776 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:23:54.776 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:54.776 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:23:54.776 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:54.776 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:54.776 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:54.777 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:54.777 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:23:54.777 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:54.777 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:54.777 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:23:54.777 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:54.777 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:23:54.778 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:54.778 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:23:54.778 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:54.778 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:54.778 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:54.778 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:23:54.778 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:54.779 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:23:54.779 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:23:54.779 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:23:54.779 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:54.779 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:23:54Z" 2023-05-24T21:23:54.779 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:23:54.780 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684963225.680656_409.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:20:25.680656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:23:54Z" } 2023-05-24T21:23:54.781 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963225.680656_409.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:20:25.680656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:23:54.781 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:23:54.781 DEBUG:rgw_multi.tests:sync_status={0: '1_1684963225.680656_409.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:23:54.782 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684963225.680656_409.1 2023-05-24T21:23:59.783 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:23:59.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:23:59.784 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:23:59.836 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:23:59.837 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:23:59.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:23:59.877+0000 7f75af756780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:23:59.884 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:23:59.877+0000 7f75af756780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:23:59.905 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:23:59.906 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:23:59.906 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:23:59.906 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:23:59.906 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:23:59.906 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:23:59.906 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:59.907 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:59.907 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:23:59.907 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:59.907 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:23:59.907 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:59.907 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:59.908 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963225.680656_409.1", 2023-05-24T21:23:59.908 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:59.908 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:23:59.908 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:59.908 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:20:25.680656Z", 2023-05-24T21:23:59.908 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:59.908 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:59.909 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:59.909 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:59.909 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:23:59.909 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:59.909 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:59.909 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:23:59.909 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:59.910 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:23:59.910 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:59.910 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:23:59.910 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:59.910 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:59.910 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:59.910 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:59.911 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:23:59.911 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:59.911 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:59.911 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:23:59.911 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:59.911 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:23:59.911 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:59.912 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:23:59.912 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:59.912 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:59.912 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:59.912 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:23:59.912 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:23:59.913 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:23:59.913 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:23:59.913 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:23:59.913 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:23:59.913 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:23:59.913 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:23:59.913 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:23:59.914 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:23:59.914 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:59.914 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:23:59.914 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:23:59.914 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:59.914 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:23:59.914 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:23:59.915 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:23:59.915 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:23:59.915 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:23:59Z" 2023-05-24T21:23:59.915 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:23:59.916 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684963225.680656_409.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:20:25.680656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:23:59Z" } 2023-05-24T21:23:59.916 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963225.680656_409.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:20:25.680656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:23:59.917 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:23:59.917 DEBUG:rgw_multi.tests:sync_status={0: '1_1684963225.680656_409.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:23:59.917 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684963225.680656_409.1 2023-05-24T21:24:04.918 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:24:04.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:24:04.920 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:24:04.974 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:24:04.975 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:24:05.014 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:24:05.008+0000 7fdad658e780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:24:05.014 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:24:05.009+0000 7fdad658e780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:24:05.035 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:24:05.036 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:24:05.036 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:24:05.036 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:24:05.036 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:24:05.036 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:24:05.036 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:05.037 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:05.037 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:24:05.037 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:24:05.037 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:24:05.037 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:24:05.037 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:24:05.037 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963225.872440_427.1", 2023-05-24T21:24:05.038 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:24:05.038 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:24:05.038 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:24:05.038 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:20:25.872440Z", 2023-05-24T21:24:05.038 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:05.038 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:05.038 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:05.039 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:24:05.039 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:24:05.039 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:24:05.039 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:24:05.039 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:24:05.039 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:24:05.040 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:24:05.040 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:24:05.040 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:24:05.040 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:05.040 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:05.040 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:05.040 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:24:05.041 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:24:05.041 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:24:05.041 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:24:05.041 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:24:05.041 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:24:05.041 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:24:05.041 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:24:05.042 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:24:05.042 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:05.042 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:05.042 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:05.042 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:24:05.042 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:24:05.042 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:24:05.043 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:24:05.043 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:24:05.043 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:24:05.043 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:24:05.043 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:24:05.043 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:24:05.043 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:05.044 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:05.044 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:05.044 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:24:05.044 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:05.044 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:24:05.044 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:24:05.044 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:24:05.045 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:05.045 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:24:05Z" 2023-05-24T21:24:05.045 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:24:05.046 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684963225.872440_427.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:20:25.872440Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:24:05Z" } 2023-05-24T21:24:05.046 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963225.872440_427.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:20:25.872440Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:24:05.047 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:24:05.047 DEBUG:rgw_multi.tests:sync_status={0: '1_1684963225.872440_427.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:24:05.047 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684963225.872440_427.1 2023-05-24T21:24:10.049 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:24:10.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:24:10.050 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:24:10.107 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:24:10.107 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:24:10.148 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:24:10.142+0000 7fc0e7d97780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:24:10.149 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:24:10.143+0000 7fc0e7d97780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:24:10.170 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:24:10.170 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:24:10.170 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:24:10.170 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:24:10.170 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:24:10.171 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:24:10.171 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:10.171 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:10.171 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:24:10.171 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:24:10.171 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:24:10.172 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:24:10.172 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:24:10.172 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963225.414960_403.1", 2023-05-24T21:24:10.172 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:24:10.172 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:24:10.172 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:24:10.172 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:20:25.414960Z", 2023-05-24T21:24:10.173 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:10.173 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:10.173 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:10.173 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:24:10.173 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:24:10.173 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:24:10.173 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:24:10.174 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:24:10.174 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:24:10.174 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:24:10.174 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:24:10.174 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:24:10.174 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:10.174 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:10.175 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:10.175 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:24:10.175 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:24:10.175 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:24:10.175 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:24:10.175 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:24:10.176 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:24:10.176 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:24:10.176 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:24:10.176 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:24:10.176 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:10.176 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:10.176 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:10.177 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:24:10.177 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:24:10.177 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:24:10.177 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:24:10.177 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:24:10.177 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:24:10.177 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:24:10.178 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:24:10.178 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:24:10.178 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:10.178 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:10.178 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:10.178 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:24:10.178 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:10.179 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:24:10.179 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:24:10.179 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:24:10.179 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:10.179 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:24:10Z" 2023-05-24T21:24:10.179 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:24:10.180 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684963225.414960_403.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:20:25.414960Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:24:10Z" } 2023-05-24T21:24:10.181 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963225.414960_403.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:20:25.414960Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:24:10.181 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:24:10.181 DEBUG:rgw_multi.tests:sync_status={0: '1_1684963225.414960_403.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:24:10.182 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684963225.414960_403.1 2023-05-24T21:24:15.183 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:24:15.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:24:15.184 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:24:15.234 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:24:15.235 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:24:15.281 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:24:15.276+0000 7f23af4de780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:24:15.282 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:24:15.277+0000 7f23af4de780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:24:15.302 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:24:15.303 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:24:15.303 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:24:15.303 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:24:15.303 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:24:15.303 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:24:15.304 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:15.304 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:15.304 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:24:15.304 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:24:15.304 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:24:15.304 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:24:15.305 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:24:15.305 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963225.410726_402.1", 2023-05-24T21:24:15.305 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:24:15.305 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:24:15.305 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:24:15.305 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:20:25.410726Z", 2023-05-24T21:24:15.305 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:15.306 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:15.306 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:15.306 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:24:15.306 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:24:15.306 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:24:15.306 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:24:15.306 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:24:15.307 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:24:15.307 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:24:15.307 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:24:15.307 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:24:15.307 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:15.307 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:15.307 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:15.308 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:24:15.308 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:24:15.308 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:24:15.308 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:24:15.308 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:24:15.308 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:24:15.308 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:24:15.309 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:24:15.309 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:24:15.309 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:15.309 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:15.309 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:15.309 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:24:15.309 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:24:15.310 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:24:15.310 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:24:15.310 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:24:15.310 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:24:15.310 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:24:15.310 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:24:15.310 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:24:15.311 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:15.311 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:15.311 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:15.311 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:24:15.311 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:15.311 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:24:15.311 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:24:15.312 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:24:15.312 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:15.312 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:24:15Z" 2023-05-24T21:24:15.312 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:24:15.313 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684963225.410726_402.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:20:25.410726Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:24:15Z" } 2023-05-24T21:24:15.313 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963225.410726_402.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:20:25.410726Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:24:15.314 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:24:15.314 DEBUG:rgw_multi.tests:sync_status={0: '1_1684963225.410726_402.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:24:15.314 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684963225.410726_402.1 2023-05-24T21:24:20.316 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:24:20.316 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:24:20.317 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:24:20.371 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:24:20.372 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:24:20.417 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:24:20.411+0000 7f8a31c07780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:24:20.418 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:24:20.412+0000 7f8a31c07780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:24:20.437 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:24:20.437 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:24:20.437 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:24:20.438 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:24:20.438 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:24:20.438 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:24:20.438 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:20.438 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:20.438 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:24:20.438 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:24:20.439 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:24:20.439 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:24:20.439 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:24:20.439 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963225.680656_409.1", 2023-05-24T21:24:20.439 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:24:20.439 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:24:20.439 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:24:20.440 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:20:25.680656Z", 2023-05-24T21:24:20.440 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:20.440 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:20.440 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:20.440 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:24:20.440 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:24:20.441 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:24:20.441 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:24:20.441 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:24:20.441 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:24:20.441 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:24:20.441 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:24:20.442 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:24:20.442 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:20.442 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:20.442 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:20.442 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:24:20.442 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:24:20.442 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:24:20.443 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:24:20.443 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:24:20.443 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:24:20.443 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:24:20.443 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:24:20.443 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:24:20.443 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:20.444 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:20.444 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:20.444 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:24:20.444 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:24:20.444 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:24:20.444 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:24:20.444 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:24:20.445 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:24:20.445 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:24:20.445 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:24:20.445 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:24:20.445 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:20.445 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:20.445 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:20.446 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:24:20.446 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:20.446 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:24:20.446 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:24:20.446 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:24:20.446 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:20.446 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:24:20Z" 2023-05-24T21:24:20.447 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:24:20.447 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684963225.680656_409.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:20:25.680656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:24:20Z" } 2023-05-24T21:24:20.447 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963225.680656_409.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:20:25.680656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:24:20.448 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:24:20.448 DEBUG:rgw_multi.tests:sync_status={0: '1_1684963225.680656_409.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:24:20.448 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684963225.680656_409.1 2023-05-24T21:24:25.449 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:24:25.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:24:25.450 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:24:25.507 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:24:25.508 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:24:25.547 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:24:25.541+0000 7fb731954780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:24:25.548 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:24:25.542+0000 7fb731954780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:24:25.567 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:24:25.567 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:24:25.567 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:24:25.568 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:24:25.568 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:24:25.568 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:24:25.568 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:25.568 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:25.568 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:24:25.569 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:24:25.569 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:24:25.569 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:24:25.569 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:24:25.569 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963225.471672_406.1", 2023-05-24T21:24:25.569 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:24:25.569 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:24:25.570 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:24:25.570 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:20:25.471672Z", 2023-05-24T21:24:25.570 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:25.570 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:25.570 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:25.570 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:24:25.570 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:24:25.571 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:24:25.571 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:24:25.571 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:24:25.571 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:24:25.571 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:24:25.571 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:24:25.571 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:24:25.572 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:25.572 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:25.572 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:25.572 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:24:25.572 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:24:25.572 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:24:25.572 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:24:25.573 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:24:25.573 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:24:25.573 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:24:25.573 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:24:25.573 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:24:25.573 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:25.573 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:25.574 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:25.574 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:24:25.574 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:24:25.574 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:24:25.574 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:24:25.574 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:24:25.574 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:24:25.575 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:24:25.575 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:24:25.575 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:24:25.575 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:25.575 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:25.575 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:25.576 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:24:25.576 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:25.576 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:24:25.576 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:24:25.576 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:24:25.576 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:25.576 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:24:25Z" 2023-05-24T21:24:25.577 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:24:25.577 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684963225.471672_406.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:20:25.471672Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:24:25Z" } 2023-05-24T21:24:25.578 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963225.471672_406.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:20:25.471672Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:24:25.578 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:24:25.579 DEBUG:rgw_multi.tests:sync_status={0: '1_1684963225.471672_406.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:24:25.579 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684963225.471672_406.1 2023-05-24T21:24:30.580 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:24:30.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:24:30.581 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:24:30.633 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:24:30.634 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:24:30.666 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:24:30.660+0000 7f90b53e8780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:24:30.667 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:24:30.661+0000 7f90b53e8780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:24:30.679 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:24:30.680 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:24:30.680 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:24:30.680 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:24:30.680 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:24:30.680 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:24:30.681 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:30.681 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:30.681 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:24:30.681 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:24:30.681 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:24:30.681 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:24:30.681 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:24:30.682 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963225.429748_404.1", 2023-05-24T21:24:30.682 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:24:30.682 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:24:30.682 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:24:30.682 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:20:25.429748Z", 2023-05-24T21:24:30.682 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:30.682 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:30.683 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:30.683 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:24:30.683 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:24:30.683 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:24:30.683 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:24:30.683 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:24:30.684 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:24:30.684 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:24:30.684 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:24:30.684 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:24:30.684 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:30.684 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:30.684 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:30.685 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:24:30.685 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:24:30.685 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:24:30.685 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:24:30.685 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:24:30.685 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:24:30.685 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:24:30.686 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:24:30.686 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:24:30.686 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:30.686 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:30.686 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:30.686 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:24:30.686 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:24:30.687 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:24:30.687 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:24:30.687 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:24:30.687 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:24:30.687 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:24:30.687 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:24:30.687 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:24:30.688 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:30.688 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:30.688 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:30.688 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:24:30.688 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:30.688 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:24:30.689 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:24:30.689 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:24:30.689 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:30.689 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:24:30Z" 2023-05-24T21:24:30.689 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:24:30.690 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684963225.429748_404.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:20:25.429748Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:24:30Z" } 2023-05-24T21:24:30.690 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963225.429748_404.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:20:25.429748Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:24:30.691 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:24:30.691 DEBUG:rgw_multi.tests:sync_status={0: '1_1684963225.429748_404.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:24:30.691 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684963225.429748_404.1 2023-05-24T21:24:35.692 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:24:35.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:24:35.693 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:24:35.750 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:24:35.751 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:24:35.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:24:35.781+0000 7f920828c780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:24:35.788 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:24:35.782+0000 7f920828c780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:24:35.803 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:24:35.803 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:24:35.804 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:24:35.804 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:24:35.804 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:24:35.804 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:24:35.804 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:35.804 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:35.804 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:24:35.805 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:24:35.805 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:24:35.805 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:24:35.805 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:24:35.805 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963225.680656_409.1", 2023-05-24T21:24:35.805 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:24:35.806 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:24:35.806 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:24:35.806 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:20:25.680656Z", 2023-05-24T21:24:35.806 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:35.806 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:35.806 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:35.806 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:24:35.807 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:24:35.807 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:24:35.807 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:24:35.807 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:24:35.807 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:24:35.807 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:24:35.807 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:24:35.808 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:24:35.808 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:35.808 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:35.808 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:35.808 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:24:35.808 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:24:35.808 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:24:35.809 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:24:35.809 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:24:35.809 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:24:35.809 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:24:35.809 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:24:35.809 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:24:35.809 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:35.810 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:35.810 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:35.810 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:24:35.810 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:24:35.810 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:24:35.810 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:24:35.810 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:24:35.811 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:24:35.811 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:24:35.811 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:24:35.811 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:24:35.811 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:35.811 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:35.811 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:35.812 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:24:35.812 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:35.812 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:24:35.812 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:24:35.812 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:24:35.812 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:35.813 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:24:35Z" 2023-05-24T21:24:35.813 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:24:35.813 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684963225.680656_409.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:20:25.680656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:24:35Z" } 2023-05-24T21:24:35.814 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963225.680656_409.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:20:25.680656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:24:35.814 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:24:35.815 DEBUG:rgw_multi.tests:sync_status={0: '1_1684963225.680656_409.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:24:35.815 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684963225.680656_409.1 2023-05-24T21:24:40.816 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:24:40.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:24:40.817 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:24:40.876 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:24:40.876 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:24:40.950 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:24:40.943+0000 7f33b4d4a780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:24:40.952 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:24:40.946+0000 7f33b4d4a780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:24:41.006 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:24:41.007 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:24:41.007 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:24:41.007 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:24:41.007 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:24:41.007 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:24:41.007 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:41.008 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:41.008 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:24:41.008 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:24:41.008 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:24:41.008 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:24:41.008 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:24:41.008 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963225.680656_409.1", 2023-05-24T21:24:41.009 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:24:41.009 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:24:41.009 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:24:41.009 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:20:25.680656Z", 2023-05-24T21:24:41.009 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:41.009 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:41.009 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:41.010 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:24:41.010 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:24:41.010 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:24:41.010 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:24:41.010 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:24:41.010 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:24:41.010 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:24:41.011 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:24:41.011 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:24:41.011 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:41.011 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:41.011 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:41.011 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:24:41.012 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:24:41.012 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:24:41.012 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:24:41.012 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:24:41.012 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:24:41.012 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:24:41.012 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:24:41.013 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:24:41.013 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:41.013 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:41.013 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:41.013 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:24:41.013 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:24:41.013 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:24:41.014 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:24:41.014 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:24:41.014 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:24:41.014 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:24:41.014 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:24:41.014 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:24:41.014 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:41.015 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:41.015 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:41.015 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:24:41.015 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:41.015 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:24:41.015 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:24:41.015 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:24:41.016 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:41.016 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:24:40Z" 2023-05-24T21:24:41.016 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:24:41.017 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684963225.680656_409.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:20:25.680656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:24:40Z" } 2023-05-24T21:24:41.017 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963225.680656_409.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:20:25.680656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:24:41.017 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:24:41.018 DEBUG:rgw_multi.tests:sync_status={0: '1_1684963225.680656_409.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:24:41.018 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684963225.680656_409.1 2023-05-24T21:24:46.020 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:24:46.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:24:46.021 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:24:46.073 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:24:46.073 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:24:46.119 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:24:46.112+0000 7f54dd13b780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:24:46.120 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:24:46.113+0000 7f54dd13b780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:24:46.140 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:24:46.140 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:24:46.140 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:24:46.141 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:24:46.141 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:24:46.141 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:24:46.141 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:46.141 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:46.141 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:24:46.141 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:24:46.142 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:24:46.142 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:24:46.142 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:24:46.142 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963225.410726_402.1", 2023-05-24T21:24:46.142 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:24:46.142 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:24:46.143 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:24:46.143 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:20:25.410726Z", 2023-05-24T21:24:46.143 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:46.143 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:46.143 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:46.143 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:24:46.143 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:24:46.144 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:24:46.144 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:24:46.144 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:24:46.144 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:24:46.144 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:24:46.144 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:24:46.144 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:24:46.145 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:46.145 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:46.145 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:46.145 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:24:46.145 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:24:46.145 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:24:46.145 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:24:46.146 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:24:46.146 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:24:46.146 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:24:46.146 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:24:46.146 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:24:46.146 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:46.146 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:46.147 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:46.147 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:24:46.147 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:24:46.147 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:24:46.147 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:24:46.147 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:24:46.147 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:24:46.148 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:24:46.148 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:24:46.148 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:24:46.148 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:46.148 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:46.148 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:46.148 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:24:46.149 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:46.149 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:24:46.149 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:24:46.149 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:24:46.149 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:46.149 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:24:46Z" 2023-05-24T21:24:46.149 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:24:46.150 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684963225.410726_402.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:20:25.410726Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:24:46Z" } 2023-05-24T21:24:46.151 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963225.410726_402.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:20:25.410726Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:24:46.151 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:24:46.151 DEBUG:rgw_multi.tests:sync_status={0: '1_1684963225.410726_402.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:24:46.152 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684963225.410726_402.1 2023-05-24T21:24:51.153 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:24:51.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:24:51.154 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:24:51.208 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:24:51.209 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:24:51.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:24:51.252+0000 7fabf4934780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:24:51.259 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:24:51.253+0000 7fabf4934780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:24:51.278 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:24:51.278 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:24:51.278 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:24:51.279 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:24:51.279 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:24:51.279 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:24:51.279 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:51.279 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:51.279 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:24:51.280 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:24:51.280 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:24:51.280 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:24:51.280 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:24:51.280 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963225.704789_414.1", 2023-05-24T21:24:51.280 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:24:51.280 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:24:51.281 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:24:51.281 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:20:25.704789Z", 2023-05-24T21:24:51.281 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:51.281 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:51.281 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:51.281 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:24:51.281 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:24:51.282 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:24:51.282 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:24:51.282 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:24:51.282 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:24:51.282 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:24:51.282 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:24:51.282 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:24:51.283 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:51.283 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:51.283 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:51.283 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:24:51.283 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:24:51.283 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:24:51.284 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:24:51.284 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:24:51.284 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:24:51.284 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:24:51.284 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:24:51.284 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:24:51.284 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:51.285 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:51.285 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:51.285 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:24:51.285 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:24:51.285 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:24:51.285 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:24:51.285 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:24:51.286 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:24:51.286 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:24:51.286 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:24:51.286 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:24:51.286 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:51.286 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:51.286 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:51.287 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:24:51.287 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:51.287 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:24:51.287 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:24:51.287 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:24:51.287 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:51.287 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:24:51Z" 2023-05-24T21:24:51.288 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:24:51.288 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684963225.704789_414.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:20:25.704789Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:24:51Z" } 2023-05-24T21:24:51.289 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963225.704789_414.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:20:25.704789Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:24:51.289 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:24:51.290 DEBUG:rgw_multi.tests:sync_status={0: '1_1684963225.704789_414.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:24:51.290 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684963225.704789_414.1 2023-05-24T21:24:56.291 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:24:56.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:24:56.292 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:24:56.349 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:24:56.350 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:24:56.386 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:24:56.380+0000 7f22b7658780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:24:56.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:24:56.381+0000 7f22b7658780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:24:56.401 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:24:56.401 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:24:56.401 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:24:56.401 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:24:56.402 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:24:56.402 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:24:56.402 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:56.402 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:56.402 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:24:56.402 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:24:56.402 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:24:56.403 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:24:56.403 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:24:56.403 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963225.487974_408.1", 2023-05-24T21:24:56.403 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:24:56.403 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:24:56.403 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:24:56.403 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:20:25.487974Z", 2023-05-24T21:24:56.404 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:56.404 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:56.404 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:56.404 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:24:56.404 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:24:56.404 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:24:56.405 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:24:56.405 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:24:56.405 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:24:56.405 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:24:56.405 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:24:56.405 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:24:56.405 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:56.406 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:56.406 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:56.406 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:24:56.406 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:24:56.406 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:24:56.406 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:24:56.406 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:24:56.407 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:24:56.407 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:24:56.407 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:24:56.407 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:24:56.407 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:56.407 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:56.407 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:56.408 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:24:56.408 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:24:56.408 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:24:56.408 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:24:56.408 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:24:56.408 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:24:56.408 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:24:56.409 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:24:56.409 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:24:56.409 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:24:56.409 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:56.409 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:24:56.409 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:24:56.409 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:56.410 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:24:56.410 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:24:56.410 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:24:56.410 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:24:56.410 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:24:56Z" 2023-05-24T21:24:56.410 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:24:56.411 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684963225.487974_408.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:20:25.487974Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:24:56Z" } 2023-05-24T21:24:56.411 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963225.487974_408.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:20:25.487974Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:24:56.412 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:24:56.412 DEBUG:rgw_multi.tests:sync_status={0: '1_1684963225.487974_408.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:24:56.413 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684963225.487974_408.1 2023-05-24T21:25:01.414 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:25:01.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:25:01.415 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:25:01.491 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:25:01.491 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:25:01.554 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:25:01.549+0000 7fc050ffb780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:25:01.555 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:25:01.550+0000 7fc050ffb780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:25:01.574 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:25:01.574 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:25:01.574 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:25:01.575 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:25:01.575 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:25:01.575 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:25:01.575 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:25:01.575 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:01.575 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:25:01.576 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:01.576 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:25:01.576 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:01.576 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:25:01.576 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963225.680656_409.1", 2023-05-24T21:25:01.576 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:01.576 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:25:01.577 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:01.577 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:20:25.680656Z", 2023-05-24T21:25:01.577 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:25:01.577 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:01.577 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:01.577 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:01.578 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:25:01.578 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:01.578 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:25:01.578 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:25:01.578 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:01.578 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:01.578 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:01.579 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:25:01.579 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:25:01.579 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:01.579 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:01.579 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:01.579 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:25:01.579 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:01.580 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:25:01.580 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:25:01.580 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:01.580 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:01.580 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:01.580 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:25:01.580 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:25:01.581 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:01.581 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:01.581 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:01.581 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:25:01.581 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:01.581 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:25:01.581 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:25:01.582 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:01.582 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:01.582 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:01.582 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:25:01.582 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:25:01.582 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:01.583 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:01.583 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:25:01.583 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:01.583 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:25:01.583 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:25:01.583 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:25:01.583 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:01.584 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:25:01Z" 2023-05-24T21:25:01.584 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:25:01.585 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684963225.680656_409.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:20:25.680656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:25:01Z" } 2023-05-24T21:25:01.585 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963225.680656_409.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:20:25.680656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:25:01.586 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:25:01.586 DEBUG:rgw_multi.tests:sync_status={0: '1_1684963225.680656_409.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:25:01.586 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684963225.680656_409.1 2023-05-24T21:25:06.588 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:25:06.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:25:06.589 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:25:06.639 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:25:06.639 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:25:06.681 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:25:06.675+0000 7f9cbd9df780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:25:06.682 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:25:06.676+0000 7f9cbd9df780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:25:06.702 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:25:06.703 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:25:06.703 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:25:06.703 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:25:06.703 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:25:06.703 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:25:06.704 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:25:06.704 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:06.704 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:25:06.704 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:06.704 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:25:06.704 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:06.705 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:25:06.705 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963225.680656_409.1", 2023-05-24T21:25:06.705 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:06.705 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:25:06.705 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:06.705 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:20:25.680656Z", 2023-05-24T21:25:06.705 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:25:06.706 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:06.706 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:06.706 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:06.706 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:25:06.706 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:06.706 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:25:06.706 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:25:06.707 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:06.707 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:06.707 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:06.707 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:25:06.707 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:25:06.707 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:06.707 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:06.708 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:06.708 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:25:06.708 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:06.708 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:25:06.708 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:25:06.708 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:06.708 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:06.709 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:06.709 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:25:06.709 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:25:06.709 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:06.709 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:06.709 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:06.709 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:25:06.710 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:06.710 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:25:06.710 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:25:06.710 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:06.710 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:06.710 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:06.711 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:25:06.711 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:25:06.711 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:06.711 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:06.711 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:25:06.711 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:06.711 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:25:06.712 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:25:06.712 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:25:06.712 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:06.712 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:25:06Z" 2023-05-24T21:25:06.712 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:25:06.713 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684963225.680656_409.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:20:25.680656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:25:06Z" } 2023-05-24T21:25:06.713 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963225.680656_409.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:20:25.680656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:25:06.714 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:25:06.714 DEBUG:rgw_multi.tests:sync_status={0: '1_1684963225.680656_409.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:25:06.714 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684963225.680656_409.1 2023-05-24T21:25:11.716 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:25:11.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:25:11.717 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:25:11.770 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:25:11.771 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:25:11.805 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:25:11.800+0000 7f3770e2e780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:25:11.806 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:25:11.801+0000 7f3770e2e780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:25:11.819 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:25:11.820 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:25:11.820 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:25:11.820 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:25:11.820 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:25:11.820 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:25:11.820 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:25:11.821 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:11.821 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:25:11.821 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:11.821 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:25:11.821 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:11.821 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:25:11.821 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963225.680656_409.1", 2023-05-24T21:25:11.822 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:11.822 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:25:11.822 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:11.822 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:20:25.680656Z", 2023-05-24T21:25:11.822 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:25:11.822 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:11.822 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:11.823 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:11.823 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:25:11.823 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:11.823 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:25:11.823 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:25:11.823 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:11.824 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:11.824 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:11.824 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:25:11.824 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:25:11.824 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:11.824 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:11.824 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:11.825 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:25:11.825 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:11.825 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:25:11.825 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:25:11.825 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:11.825 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:11.825 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:11.826 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:25:11.826 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:25:11.826 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:11.826 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:11.826 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:11.826 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:25:11.826 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:11.827 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:25:11.827 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:25:11.827 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:11.827 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:11.827 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:11.827 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:25:11.827 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:25:11.828 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:11.828 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:11.828 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:25:11.828 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:11.828 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:25:11.828 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:25:11.828 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:25:11.829 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:11.829 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:25:11Z" 2023-05-24T21:25:11.829 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:25:11.830 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684963225.680656_409.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:20:25.680656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:25:11Z" } 2023-05-24T21:25:11.830 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963225.680656_409.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:20:25.680656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:25:11.831 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:25:11.831 DEBUG:rgw_multi.tests:sync_status={0: '1_1684963225.680656_409.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:25:11.831 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684963225.680656_409.1 2023-05-24T21:25:16.832 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:25:16.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:25:16.833 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:25:16.891 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:25:16.891 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:25:16.926 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:25:16.920+0000 7fd4802ab780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:25:16.927 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:25:16.921+0000 7fd4802ab780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:25:16.944 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:25:16.945 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:25:16.945 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:25:16.945 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:25:16.945 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:25:16.945 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:25:16.945 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:25:16.946 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:16.946 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:25:16.946 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:16.946 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:25:16.946 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:16.946 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:25:16.947 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963225.429748_404.1", 2023-05-24T21:25:16.947 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:16.947 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:25:16.947 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:16.947 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:20:25.429748Z", 2023-05-24T21:25:16.947 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:25:16.947 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:16.948 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:16.948 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:16.948 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:25:16.948 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:16.948 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:25:16.948 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:25:16.948 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:16.949 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:16.949 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:16.949 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:25:16.949 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:25:16.949 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:16.949 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:16.949 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:16.950 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:25:16.950 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:16.950 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:25:16.950 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:25:16.950 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:16.950 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:16.950 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:16.951 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:25:16.951 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:25:16.951 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:16.951 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:16.951 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:16.951 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:25:16.951 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:16.952 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:25:16.952 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:25:16.952 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:16.952 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:16.952 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:16.952 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:25:16.952 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:25:16.953 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:16.953 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:16.953 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:25:16.953 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:16.953 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:25:16.953 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:25:16.953 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:25:16.954 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:16.954 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:25:16Z" 2023-05-24T21:25:16.954 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:25:16.955 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684963225.429748_404.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:20:25.429748Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:25:16Z" } 2023-05-24T21:25:16.955 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963225.429748_404.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:20:25.429748Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:25:16.956 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:25:16.956 DEBUG:rgw_multi.tests:sync_status={0: '1_1684963225.429748_404.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:25:16.956 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684963225.429748_404.1 2023-05-24T21:25:21.957 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:25:21.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:25:21.958 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:25:22.010 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:25:22.010 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:25:22.061 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:25:22.055+0000 7fa136d4a780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:25:22.062 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:25:22.056+0000 7fa136d4a780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:25:22.074 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:25:22.075 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:25:22.075 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:25:22.075 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:25:22.075 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:25:22.075 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:25:22.075 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:25:22.076 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:22.076 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:25:22.076 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:22.076 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:25:22.076 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:22.076 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:25:22.077 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963225.680656_409.1", 2023-05-24T21:25:22.077 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:22.077 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:25:22.077 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:22.077 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:20:25.680656Z", 2023-05-24T21:25:22.077 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:25:22.078 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:22.078 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:22.078 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:22.078 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:25:22.078 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:22.078 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:25:22.078 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:25:22.079 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:22.079 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:22.079 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:22.079 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:25:22.079 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:25:22.079 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:22.079 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:22.080 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:22.080 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:25:22.080 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:22.080 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:25:22.080 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:25:22.080 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:22.081 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:22.081 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:22.081 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:25:22.081 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:25:22.081 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:22.081 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:22.081 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:22.082 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:25:22.082 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:22.082 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:25:22.082 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:25:22.082 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:22.082 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:22.083 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:22.083 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:25:22.083 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:25:22.083 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:22.083 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:22.083 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:25:22.083 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:22.084 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:25:22.084 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:25:22.084 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:25:22.084 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:22.084 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:25:22Z" 2023-05-24T21:25:22.084 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:25:22.085 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684963225.680656_409.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:20:25.680656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:25:22Z" } 2023-05-24T21:25:22.086 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963225.680656_409.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:20:25.680656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:25:22.086 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:25:22.087 DEBUG:rgw_multi.tests:sync_status={0: '1_1684963225.680656_409.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:25:22.087 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684963225.680656_409.1 2023-05-24T21:25:27.088 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:25:27.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:25:27.090 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:25:27.141 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:25:27.142 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:25:27.194 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:25:27.188+0000 7fb957980780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:25:27.195 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:25:27.189+0000 7fb957980780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:25:27.214 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:25:27.214 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:25:27.214 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:25:27.214 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:25:27.214 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:25:27.215 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:25:27.215 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:25:27.215 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:27.215 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:25:27.215 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:27.215 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:25:27.216 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:27.216 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:25:27.216 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963225.429748_404.1", 2023-05-24T21:25:27.216 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:27.216 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:25:27.216 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:27.217 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:20:25.429748Z", 2023-05-24T21:25:27.217 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:25:27.217 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:27.217 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:27.217 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:27.217 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:25:27.217 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:27.218 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:25:27.218 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:25:27.218 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:27.218 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:27.218 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:27.218 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:25:27.218 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:25:27.219 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:27.219 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:27.219 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:27.219 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:25:27.219 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:27.219 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:25:27.219 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:25:27.220 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:27.220 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:27.220 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:27.220 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:25:27.220 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:25:27.220 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:27.221 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:27.221 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:27.221 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:25:27.221 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:27.221 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:25:27.221 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:25:27.222 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:27.222 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:27.222 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:27.222 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:25:27.222 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:25:27.222 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:27.222 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:27.223 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:25:27.223 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:27.223 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:25:27.223 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:25:27.223 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:25:27.223 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:27.224 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:25:27Z" 2023-05-24T21:25:27.224 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:25:27.224 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684963225.429748_404.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:20:25.429748Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:25:27Z" } 2023-05-24T21:25:27.225 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963225.429748_404.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:20:25.429748Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:25:27.225 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:25:27.226 DEBUG:rgw_multi.tests:sync_status={0: '1_1684963225.429748_404.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:25:27.226 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963229.148253_462.1 target=1_1684963225.429748_404.1 2023-05-24T21:25:32.227 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:25:32.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:25:32.229 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:25:32.282 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:25:32.283 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:25:32.322 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:25:32.315+0000 7efc3de6b780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:25:32.323 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:25:32.316+0000 7efc3de6b780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:25:32.336 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:25:32.337 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:25:32.337 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:25:32.337 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:25:32.337 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:25:32.337 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:25:32.337 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:25:32.338 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:32.338 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:25:32.338 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:32.338 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:25:32.338 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:32.338 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:25:32.339 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963229.148253_462.1", 2023-05-24T21:25:32.339 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:32.339 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:25:32.339 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:32.339 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:20:29.148253Z", 2023-05-24T21:25:32.339 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:25:32.339 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:32.340 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:32.340 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:32.340 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:25:32.340 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:32.340 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:25:32.340 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:25:32.340 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:32.341 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:32.341 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:32.341 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:25:32.341 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:25:32.341 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:32.341 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:32.341 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:32.342 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:25:32.342 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:32.342 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:25:32.342 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:25:32.342 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:32.342 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:32.342 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:32.343 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:25:32.343 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:25:32.343 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:32.343 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:32.343 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:32.343 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:25:32.344 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:32.344 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:25:32.344 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:25:32.344 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:32.344 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:32.344 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:32.344 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:25:32.345 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:25:32.345 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:32.345 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:32.345 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:25:32.345 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:32.345 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:25:32.345 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:25:32.346 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:25:32.346 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:32.346 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:25:32Z" 2023-05-24T21:25:32.346 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:25:32.347 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684963229.148253_462.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:20:29.148253Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:25:32Z" } 2023-05-24T21:25:32.348 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963229.148253_462.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:20:29.148253Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:25:32.348 DEBUG:rgw_multi.tests:log_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:25:32.348 DEBUG:rgw_multi.tests:sync_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:25:32.349 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-24T21:25:32.349 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'qskrps-44', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:25:32.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'qskrps-44', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:25:32.350 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket qskrps-44 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-24T21:25:32.400 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:25:32.401 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:25:32.473 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'qskrps-45', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:25:32.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'qskrps-45', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:25:32.475 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket qskrps-45 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-24T21:25:32.524 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:25:32.525 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:25:32.624 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:25:32.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:25:32.625 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:25:32.671 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:25:32.672 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:25:32.716 INFO:teuthology.orchestra.run.smithi071.stdout:[ 2023-05-24T21:25:32.716 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-14:299a3349-fae0-459e-8632-431fc36beb1b.4368.14:10", 2023-05-24T21:25:32.716 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-32:299a3349-fae0-459e-8632-431fc36beb1b.4368.32:9", 2023-05-24T21:25:32.716 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000078", 2023-05-24T21:25:32.717 INFO:teuthology.orchestra.run.smithi071.stdout: "meta.history", 2023-05-24T21:25:32.717 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-16:299a3349-fae0-459e-8632-431fc36beb1b.4368.16:1", 2023-05-24T21:25:32.717 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000070", 2023-05-24T21:25:32.717 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000104", 2023-05-24T21:25:32.717 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-32:299a3349-fae0-459e-8632-431fc36beb1b.4368.32:2", 2023-05-24T21:25:32.718 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000026", 2023-05-24T21:25:32.718 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-35:299a3349-fae0-459e-8632-431fc36beb1b.5347.1:5", 2023-05-24T21:25:32.718 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-33:299a3349-fae0-459e-8632-431fc36beb1b.4368.33:5", 2023-05-24T21:25:32.718 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-14:299a3349-fae0-459e-8632-431fc36beb1b.4368.14:0", 2023-05-24T21:25:32.718 INFO:teuthology.orchestra.run.smithi071.stdout: "data_log.3.obNkxGm+maN.0", 2023-05-24T21:25:32.718 INFO:teuthology.orchestra.run.smithi071.stdout: "sync.error-log.8", 2023-05-24T21:25:32.719 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-18:299a3349-fae0-459e-8632-431fc36beb1b.4368.18:8", 2023-05-24T21:25:32.719 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-8:299a3349-fae0-459e-8632-431fc36beb1b.4368.8:3", 2023-05-24T21:25:32.719 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000028", 2023-05-24T21:25:32.719 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-33:299a3349-fae0-459e-8632-431fc36beb1b.4368.33:3", 2023-05-24T21:25:32.719 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-8:299a3349-fae0-459e-8632-431fc36beb1b.4368.8:4", 2023-05-24T21:25:32.719 INFO:teuthology.orchestra.run.smithi071.stdout: "datalog.sync-status.shard.5af1588c-f9b9-4687-a040-ce6e0cc808d9.1.retry", 2023-05-24T21:25:32.720 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-33:299a3349-fae0-459e-8632-431fc36beb1b.4368.33:10", 2023-05-24T21:25:32.720 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000040", 2023-05-24T21:25:32.720 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-31:299a3349-fae0-459e-8632-431fc36beb1b.4368.31:2", 2023-05-24T21:25:32.720 INFO:teuthology.orchestra.run.smithi071.stdout: "sync.error-log.6", 2023-05-24T21:25:32.720 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-7:299a3349-fae0-459e-8632-431fc36beb1b.4368.7:8", 2023-05-24T21:25:32.721 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-7:299a3349-fae0-459e-8632-431fc36beb1b.4368.7:2", 2023-05-24T21:25:32.721 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000015", 2023-05-24T21:25:32.721 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.full-sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-35:299a3349-fae0-459e-8632-431fc36beb1b.5347.1", 2023-05-24T21:25:32.721 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000069", 2023-05-24T21:25:32.721 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-12:299a3349-fae0-459e-8632-431fc36beb1b.4368.12:2", 2023-05-24T21:25:32.722 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-32:299a3349-fae0-459e-8632-431fc36beb1b.4368.32:1", 2023-05-24T21:25:32.722 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000095", 2023-05-24T21:25:32.722 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-34:299a3349-fae0-459e-8632-431fc36beb1b.4368.34:2", 2023-05-24T21:25:32.722 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-12:299a3349-fae0-459e-8632-431fc36beb1b.4368.12:6", 2023-05-24T21:25:32.722 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000003", 2023-05-24T21:25:32.723 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000047", 2023-05-24T21:25:32.723 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-31:299a3349-fae0-459e-8632-431fc36beb1b.4368.31:3", 2023-05-24T21:25:32.723 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000035", 2023-05-24T21:25:32.723 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000037", 2023-05-24T21:25:32.723 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-31:299a3349-fae0-459e-8632-431fc36beb1b.4368.31:9", 2023-05-24T21:25:32.724 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-34:299a3349-fae0-459e-8632-431fc36beb1b.4368.34:5", 2023-05-24T21:25:32.724 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-14:299a3349-fae0-459e-8632-431fc36beb1b.4368.14:4", 2023-05-24T21:25:32.724 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-31:299a3349-fae0-459e-8632-431fc36beb1b.4368.31:1", 2023-05-24T21:25:32.724 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-35:299a3349-fae0-459e-8632-431fc36beb1b.5347.1:6", 2023-05-24T21:25:32.724 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-7:299a3349-fae0-459e-8632-431fc36beb1b.4368.7:6", 2023-05-24T21:25:32.725 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-8:299a3349-fae0-459e-8632-431fc36beb1b.4368.8:8", 2023-05-24T21:25:32.725 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-15:299a3349-fae0-459e-8632-431fc36beb1b.4368.15:8", 2023-05-24T21:25:32.725 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000024", 2023-05-24T21:25:32.725 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000000", 2023-05-24T21:25:32.725 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000031", 2023-05-24T21:25:32.726 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-16:299a3349-fae0-459e-8632-431fc36beb1b.4368.16:3", 2023-05-24T21:25:32.726 INFO:teuthology.orchestra.run.smithi071.stdout: "data_log.3", 2023-05-24T21:25:32.726 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.full-sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-15:299a3349-fae0-459e-8632-431fc36beb1b.4368.15", 2023-05-24T21:25:32.726 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-14:299a3349-fae0-459e-8632-431fc36beb1b.4368.14:6", 2023-05-24T21:25:32.726 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-8:299a3349-fae0-459e-8632-431fc36beb1b.4368.8:0", 2023-05-24T21:25:32.726 INFO:teuthology.orchestra.run.smithi071.stdout: "datalog.sync-status.shard.5af1588c-f9b9-4687-a040-ce6e0cc808d9.3", 2023-05-24T21:25:32.727 INFO:teuthology.orchestra.run.smithi071.stdout: "data_loggenerations_metadata", 2023-05-24T21:25:32.727 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000076", 2023-05-24T21:25:32.727 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-13:299a3349-fae0-459e-8632-431fc36beb1b.4368.13:5", 2023-05-24T21:25:32.727 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-11:299a3349-fae0-459e-8632-431fc36beb1b.4368.11:0", 2023-05-24T21:25:32.727 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000113", 2023-05-24T21:25:32.727 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000005", 2023-05-24T21:25:32.727 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000011", 2023-05-24T21:25:32.728 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-18:299a3349-fae0-459e-8632-431fc36beb1b.4368.18:10", 2023-05-24T21:25:32.728 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000058", 2023-05-24T21:25:32.728 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-33:299a3349-fae0-459e-8632-431fc36beb1b.4368.33:0", 2023-05-24T21:25:32.728 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000052", 2023-05-24T21:25:32.728 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-33:299a3349-fae0-459e-8632-431fc36beb1b.4368.33:9", 2023-05-24T21:25:32.728 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-18:299a3349-fae0-459e-8632-431fc36beb1b.4368.18:9", 2023-05-24T21:25:32.728 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-13:299a3349-fae0-459e-8632-431fc36beb1b.4368.13:3", 2023-05-24T21:25:32.729 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-11:299a3349-fae0-459e-8632-431fc36beb1b.4368.11:3", 2023-05-24T21:25:32.729 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-13:299a3349-fae0-459e-8632-431fc36beb1b.4368.13:6", 2023-05-24T21:25:32.729 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-18:299a3349-fae0-459e-8632-431fc36beb1b.4368.18:5", 2023-05-24T21:25:32.729 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000088", 2023-05-24T21:25:32.729 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000080", 2023-05-24T21:25:32.729 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000090", 2023-05-24T21:25:32.729 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000110", 2023-05-24T21:25:32.730 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-11:299a3349-fae0-459e-8632-431fc36beb1b.4368.11:7", 2023-05-24T21:25:32.730 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000096", 2023-05-24T21:25:32.730 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000087", 2023-05-24T21:25:32.730 INFO:teuthology.orchestra.run.smithi071.stdout: "datalog.sync-status.shard.5af1588c-f9b9-4687-a040-ce6e0cc808d9.2.retry", 2023-05-24T21:25:32.730 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000008", 2023-05-24T21:25:32.730 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000006", 2023-05-24T21:25:32.731 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000029", 2023-05-24T21:25:32.731 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-32:299a3349-fae0-459e-8632-431fc36beb1b.4368.32:0", 2023-05-24T21:25:32.731 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000089", 2023-05-24T21:25:32.731 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000022", 2023-05-24T21:25:32.731 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-7:299a3349-fae0-459e-8632-431fc36beb1b.4368.7:9", 2023-05-24T21:25:32.731 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-16:299a3349-fae0-459e-8632-431fc36beb1b.4368.16:8", 2023-05-24T21:25:32.731 INFO:teuthology.orchestra.run.smithi071.stdout: "bilog.trim", 2023-05-24T21:25:32.732 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000016", 2023-05-24T21:25:32.732 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-7:299a3349-fae0-459e-8632-431fc36beb1b.4368.7:7", 2023-05-24T21:25:32.732 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000041", 2023-05-24T21:25:32.732 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-34:299a3349-fae0-459e-8632-431fc36beb1b.4368.34:10", 2023-05-24T21:25:32.732 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000018", 2023-05-24T21:25:32.732 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000092", 2023-05-24T21:25:32.732 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-9:299a3349-fae0-459e-8632-431fc36beb1b.4368.9:6", 2023-05-24T21:25:32.733 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-9:299a3349-fae0-459e-8632-431fc36beb1b.4368.9:4", 2023-05-24T21:25:32.733 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-7:299a3349-fae0-459e-8632-431fc36beb1b.4368.7:10", 2023-05-24T21:25:32.733 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-13:299a3349-fae0-459e-8632-431fc36beb1b.4368.13:4", 2023-05-24T21:25:32.733 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-17:299a3349-fae0-459e-8632-431fc36beb1b.4368.17:1", 2023-05-24T21:25:32.733 INFO:teuthology.orchestra.run.smithi071.stdout: "sync.error-log.4", 2023-05-24T21:25:32.733 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-15:299a3349-fae0-459e-8632-431fc36beb1b.4368.15:6", 2023-05-24T21:25:32.733 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000014", 2023-05-24T21:25:32.734 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-7:299a3349-fae0-459e-8632-431fc36beb1b.4368.7:0", 2023-05-24T21:25:32.734 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-7:299a3349-fae0-459e-8632-431fc36beb1b.4368.7:5", 2023-05-24T21:25:32.734 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000112", 2023-05-24T21:25:32.734 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-13:299a3349-fae0-459e-8632-431fc36beb1b.4368.13:2", 2023-05-24T21:25:32.734 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000007", 2023-05-24T21:25:32.734 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000021", 2023-05-24T21:25:32.734 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-13:299a3349-fae0-459e-8632-431fc36beb1b.4368.13:7", 2023-05-24T21:25:32.735 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.full-sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-17:299a3349-fae0-459e-8632-431fc36beb1b.4368.17", 2023-05-24T21:25:32.735 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000064", 2023-05-24T21:25:32.735 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-17:299a3349-fae0-459e-8632-431fc36beb1b.4368.17:3", 2023-05-24T21:25:32.735 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000071", 2023-05-24T21:25:32.735 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-10:299a3349-fae0-459e-8632-431fc36beb1b.4368.10:2", 2023-05-24T21:25:32.735 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000074", 2023-05-24T21:25:32.735 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-12:299a3349-fae0-459e-8632-431fc36beb1b.4368.12:5", 2023-05-24T21:25:32.736 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-15:299a3349-fae0-459e-8632-431fc36beb1b.4368.15:1", 2023-05-24T21:25:32.736 INFO:teuthology.orchestra.run.smithi071.stdout: "datalog.sync-status.shard.5af1588c-f9b9-4687-a040-ce6e0cc808d9.3.retry", 2023-05-24T21:25:32.736 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-16:299a3349-fae0-459e-8632-431fc36beb1b.4368.16:9", 2023-05-24T21:25:32.736 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-17:299a3349-fae0-459e-8632-431fc36beb1b.4368.17:5", 2023-05-24T21:25:32.736 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000081", 2023-05-24T21:25:32.736 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-35:299a3349-fae0-459e-8632-431fc36beb1b.5347.1:8", 2023-05-24T21:25:32.736 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000009", 2023-05-24T21:25:32.737 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-11:299a3349-fae0-459e-8632-431fc36beb1b.4368.11:5", 2023-05-24T21:25:32.737 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000121", 2023-05-24T21:25:32.737 INFO:teuthology.orchestra.run.smithi071.stdout: "data_log.1", 2023-05-24T21:25:32.737 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-32:299a3349-fae0-459e-8632-431fc36beb1b.4368.32:10", 2023-05-24T21:25:32.737 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-34:299a3349-fae0-459e-8632-431fc36beb1b.4368.34:4", 2023-05-24T21:25:32.737 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-12:299a3349-fae0-459e-8632-431fc36beb1b.4368.12:0", 2023-05-24T21:25:32.737 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000125", 2023-05-24T21:25:32.738 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-18:299a3349-fae0-459e-8632-431fc36beb1b.4368.18:2", 2023-05-24T21:25:32.738 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000082", 2023-05-24T21:25:32.738 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000105", 2023-05-24T21:25:32.738 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000059", 2023-05-24T21:25:32.738 INFO:teuthology.orchestra.run.smithi071.stdout: "datalog.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9", 2023-05-24T21:25:32.738 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000077", 2023-05-24T21:25:32.738 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000032", 2023-05-24T21:25:32.739 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-14:299a3349-fae0-459e-8632-431fc36beb1b.4368.14:8", 2023-05-24T21:25:32.739 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.full-sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-11:299a3349-fae0-459e-8632-431fc36beb1b.4368.11", 2023-05-24T21:25:32.739 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-14:299a3349-fae0-459e-8632-431fc36beb1b.4368.14:3", 2023-05-24T21:25:32.739 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-9:299a3349-fae0-459e-8632-431fc36beb1b.4368.9:5", 2023-05-24T21:25:32.739 INFO:teuthology.orchestra.run.smithi071.stdout: "datalog.sync-status.shard.5af1588c-f9b9-4687-a040-ce6e0cc808d9.1", 2023-05-24T21:25:32.739 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-34:299a3349-fae0-459e-8632-431fc36beb1b.4368.34:8", 2023-05-24T21:25:32.739 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000053", 2023-05-24T21:25:32.740 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000091", 2023-05-24T21:25:32.740 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-15:299a3349-fae0-459e-8632-431fc36beb1b.4368.15:10", 2023-05-24T21:25:32.740 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000065", 2023-05-24T21:25:32.740 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-8:299a3349-fae0-459e-8632-431fc36beb1b.4368.8:9", 2023-05-24T21:25:32.740 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000083", 2023-05-24T21:25:32.740 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000010", 2023-05-24T21:25:32.740 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-35:299a3349-fae0-459e-8632-431fc36beb1b.5347.1:7", 2023-05-24T21:25:32.741 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-31:299a3349-fae0-459e-8632-431fc36beb1b.4368.31:6", 2023-05-24T21:25:32.741 INFO:teuthology.orchestra.run.smithi071.stdout: "sync.error-log.7", 2023-05-24T21:25:32.741 INFO:teuthology.orchestra.run.smithi071.stdout: "sync.error-log.3", 2023-05-24T21:25:32.741 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-9:299a3349-fae0-459e-8632-431fc36beb1b.4368.9:0", 2023-05-24T21:25:32.741 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000045", 2023-05-24T21:25:32.741 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-31:299a3349-fae0-459e-8632-431fc36beb1b.4368.31:0", 2023-05-24T21:25:32.742 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000002", 2023-05-24T21:25:32.742 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-17:299a3349-fae0-459e-8632-431fc36beb1b.4368.17:4", 2023-05-24T21:25:32.742 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000116", 2023-05-24T21:25:32.742 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-11:299a3349-fae0-459e-8632-431fc36beb1b.4368.11:2", 2023-05-24T21:25:32.742 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.full-sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-8:299a3349-fae0-459e-8632-431fc36beb1b.4368.8", 2023-05-24T21:25:32.742 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.full-sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-31:299a3349-fae0-459e-8632-431fc36beb1b.4368.31", 2023-05-24T21:25:32.743 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000034", 2023-05-24T21:25:32.743 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-17:299a3349-fae0-459e-8632-431fc36beb1b.4368.17:10", 2023-05-24T21:25:32.743 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-9:299a3349-fae0-459e-8632-431fc36beb1b.4368.9:2", 2023-05-24T21:25:32.743 INFO:teuthology.orchestra.run.smithi071.stdout: "data_log.0.GFuRh9Iht5A.0", 2023-05-24T21:25:32.743 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000101", 2023-05-24T21:25:32.744 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-35:299a3349-fae0-459e-8632-431fc36beb1b.5347.1:0", 2023-05-24T21:25:32.744 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-11:299a3349-fae0-459e-8632-431fc36beb1b.4368.11:6", 2023-05-24T21:25:32.744 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-17:299a3349-fae0-459e-8632-431fc36beb1b.4368.17:8", 2023-05-24T21:25:32.744 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000079", 2023-05-24T21:25:32.744 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000049", 2023-05-24T21:25:32.744 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-14:299a3349-fae0-459e-8632-431fc36beb1b.4368.14:7", 2023-05-24T21:25:32.744 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000117", 2023-05-24T21:25:32.745 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000044", 2023-05-24T21:25:32.745 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-18:299a3349-fae0-459e-8632-431fc36beb1b.4368.18:1", 2023-05-24T21:25:32.745 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000066", 2023-05-24T21:25:32.745 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-7:299a3349-fae0-459e-8632-431fc36beb1b.4368.7:4", 2023-05-24T21:25:32.745 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000068", 2023-05-24T21:25:32.745 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-12:299a3349-fae0-459e-8632-431fc36beb1b.4368.12:7", 2023-05-24T21:25:32.745 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000085", 2023-05-24T21:25:32.746 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-35:299a3349-fae0-459e-8632-431fc36beb1b.5347.1:10", 2023-05-24T21:25:32.746 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-10:299a3349-fae0-459e-8632-431fc36beb1b.4368.10:9", 2023-05-24T21:25:32.746 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-11:299a3349-fae0-459e-8632-431fc36beb1b.4368.11:4", 2023-05-24T21:25:32.746 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-11:299a3349-fae0-459e-8632-431fc36beb1b.4368.11:1", 2023-05-24T21:25:32.746 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000073", 2023-05-24T21:25:32.746 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-17:299a3349-fae0-459e-8632-431fc36beb1b.4368.17:2", 2023-05-24T21:25:32.746 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-11:299a3349-fae0-459e-8632-431fc36beb1b.4368.11:10", 2023-05-24T21:25:32.747 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000038", 2023-05-24T21:25:32.747 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-15:299a3349-fae0-459e-8632-431fc36beb1b.4368.15:0", 2023-05-24T21:25:32.747 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000118", 2023-05-24T21:25:32.747 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-15:299a3349-fae0-459e-8632-431fc36beb1b.4368.15:2", 2023-05-24T21:25:32.747 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000036", 2023-05-24T21:25:32.747 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.full-sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-34:299a3349-fae0-459e-8632-431fc36beb1b.4368.34", 2023-05-24T21:25:32.747 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000103", 2023-05-24T21:25:32.748 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-12:299a3349-fae0-459e-8632-431fc36beb1b.4368.12:9", 2023-05-24T21:25:32.748 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000119", 2023-05-24T21:25:32.748 INFO:teuthology.orchestra.run.smithi071.stdout: "datalog.sync-status.shard.5af1588c-f9b9-4687-a040-ce6e0cc808d9.2", 2023-05-24T21:25:32.748 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000098", 2023-05-24T21:25:32.748 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-13:299a3349-fae0-459e-8632-431fc36beb1b.4368.13:9", 2023-05-24T21:25:32.748 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-8:299a3349-fae0-459e-8632-431fc36beb1b.4368.8:2", 2023-05-24T21:25:32.749 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-33:299a3349-fae0-459e-8632-431fc36beb1b.4368.33:4", 2023-05-24T21:25:32.749 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.full-sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-32:299a3349-fae0-459e-8632-431fc36beb1b.4368.32", 2023-05-24T21:25:32.749 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000027", 2023-05-24T21:25:32.749 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000019", 2023-05-24T21:25:32.749 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-10:299a3349-fae0-459e-8632-431fc36beb1b.4368.10:7", 2023-05-24T21:25:32.749 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-16:299a3349-fae0-459e-8632-431fc36beb1b.4368.16:6", 2023-05-24T21:25:32.749 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000039", 2023-05-24T21:25:32.750 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-9:299a3349-fae0-459e-8632-431fc36beb1b.4368.9:1", 2023-05-24T21:25:32.750 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-13:299a3349-fae0-459e-8632-431fc36beb1b.4368.13:10", 2023-05-24T21:25:32.750 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-12:299a3349-fae0-459e-8632-431fc36beb1b.4368.12:10", 2023-05-24T21:25:32.750 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000100", 2023-05-24T21:25:32.750 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-15:299a3349-fae0-459e-8632-431fc36beb1b.4368.15:7", 2023-05-24T21:25:32.750 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-17:299a3349-fae0-459e-8632-431fc36beb1b.4368.17:6", 2023-05-24T21:25:32.750 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-16:299a3349-fae0-459e-8632-431fc36beb1b.4368.16:10", 2023-05-24T21:25:32.751 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000093", 2023-05-24T21:25:32.751 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000004", 2023-05-24T21:25:32.751 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000063", 2023-05-24T21:25:32.751 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-15:299a3349-fae0-459e-8632-431fc36beb1b.4368.15:5", 2023-05-24T21:25:32.751 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000122", 2023-05-24T21:25:32.751 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000057", 2023-05-24T21:25:32.751 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000054", 2023-05-24T21:25:32.752 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000114", 2023-05-24T21:25:32.752 INFO:teuthology.orchestra.run.smithi071.stdout: "sync.error-log.2", 2023-05-24T21:25:32.752 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-33:299a3349-fae0-459e-8632-431fc36beb1b.4368.33:8", 2023-05-24T21:25:32.752 INFO:teuthology.orchestra.run.smithi071.stdout: "data_log.0", 2023-05-24T21:25:32.752 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-35:299a3349-fae0-459e-8632-431fc36beb1b.5347.1:1", 2023-05-24T21:25:32.752 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000012", 2023-05-24T21:25:32.752 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-35:299a3349-fae0-459e-8632-431fc36beb1b.5347.1:2", 2023-05-24T21:25:32.753 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000084", 2023-05-24T21:25:32.753 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000043", 2023-05-24T21:25:32.753 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-14:299a3349-fae0-459e-8632-431fc36beb1b.4368.14:9", 2023-05-24T21:25:32.753 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-11:299a3349-fae0-459e-8632-431fc36beb1b.4368.11:8", 2023-05-24T21:25:32.753 INFO:teuthology.orchestra.run.smithi071.stdout: "meta.log.5c04936b-70ea-4415-b4eb-3aad3dc612e5.1", 2023-05-24T21:25:32.753 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-13:299a3349-fae0-459e-8632-431fc36beb1b.4368.13:8", 2023-05-24T21:25:32.753 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-14:299a3349-fae0-459e-8632-431fc36beb1b.4368.14:2", 2023-05-24T21:25:32.754 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-13:299a3349-fae0-459e-8632-431fc36beb1b.4368.13:0", 2023-05-24T21:25:32.754 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-12:299a3349-fae0-459e-8632-431fc36beb1b.4368.12:4", 2023-05-24T21:25:32.754 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-8:299a3349-fae0-459e-8632-431fc36beb1b.4368.8:1", 2023-05-24T21:25:32.754 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-8:299a3349-fae0-459e-8632-431fc36beb1b.4368.8:6", 2023-05-24T21:25:32.754 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-34:299a3349-fae0-459e-8632-431fc36beb1b.4368.34:0", 2023-05-24T21:25:32.754 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000111", 2023-05-24T21:25:32.754 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000048", 2023-05-24T21:25:32.755 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-12:299a3349-fae0-459e-8632-431fc36beb1b.4368.12:8", 2023-05-24T21:25:32.755 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000020", 2023-05-24T21:25:32.755 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.full-sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-14:299a3349-fae0-459e-8632-431fc36beb1b.4368.14", 2023-05-24T21:25:32.755 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000099", 2023-05-24T21:25:32.755 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000056", 2023-05-24T21:25:32.755 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-34:299a3349-fae0-459e-8632-431fc36beb1b.4368.34:3", 2023-05-24T21:25:32.755 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-10:299a3349-fae0-459e-8632-431fc36beb1b.4368.10:1", 2023-05-24T21:25:32.756 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-10:299a3349-fae0-459e-8632-431fc36beb1b.4368.10:4", 2023-05-24T21:25:32.756 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-14:299a3349-fae0-459e-8632-431fc36beb1b.4368.14:5", 2023-05-24T21:25:32.756 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-16:299a3349-fae0-459e-8632-431fc36beb1b.4368.16:4", 2023-05-24T21:25:32.756 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-17:299a3349-fae0-459e-8632-431fc36beb1b.4368.17:0", 2023-05-24T21:25:32.756 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-18:299a3349-fae0-459e-8632-431fc36beb1b.4368.18:6", 2023-05-24T21:25:32.756 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-32:299a3349-fae0-459e-8632-431fc36beb1b.4368.32:5", 2023-05-24T21:25:32.756 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.full-sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-16:299a3349-fae0-459e-8632-431fc36beb1b.4368.16", 2023-05-24T21:25:32.757 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-16:299a3349-fae0-459e-8632-431fc36beb1b.4368.16:0", 2023-05-24T21:25:32.757 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000072", 2023-05-24T21:25:32.757 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000062", 2023-05-24T21:25:32.757 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.full-sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-33:299a3349-fae0-459e-8632-431fc36beb1b.4368.33", 2023-05-24T21:25:32.757 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-7:299a3349-fae0-459e-8632-431fc36beb1b.4368.7:1", 2023-05-24T21:25:32.757 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000115", 2023-05-24T21:25:32.757 INFO:teuthology.orchestra.run.smithi071.stdout: "sync.error-log.1", 2023-05-24T21:25:32.758 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000033", 2023-05-24T21:25:32.758 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-32:299a3349-fae0-459e-8632-431fc36beb1b.4368.32:4", 2023-05-24T21:25:32.758 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-31:299a3349-fae0-459e-8632-431fc36beb1b.4368.31:4", 2023-05-24T21:25:32.758 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-31:299a3349-fae0-459e-8632-431fc36beb1b.4368.31:5", 2023-05-24T21:25:32.758 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-10:299a3349-fae0-459e-8632-431fc36beb1b.4368.10:5", 2023-05-24T21:25:32.758 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000025", 2023-05-24T21:25:32.758 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000123", 2023-05-24T21:25:32.759 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-35:299a3349-fae0-459e-8632-431fc36beb1b.5347.1:4", 2023-05-24T21:25:32.759 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000108", 2023-05-24T21:25:32.759 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000094", 2023-05-24T21:25:32.759 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-34:299a3349-fae0-459e-8632-431fc36beb1b.4368.34:1", 2023-05-24T21:25:32.759 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-18:299a3349-fae0-459e-8632-431fc36beb1b.4368.18:0", 2023-05-24T21:25:32.759 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000060", 2023-05-24T21:25:32.759 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-9:299a3349-fae0-459e-8632-431fc36beb1b.4368.9:7", 2023-05-24T21:25:32.760 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-9:299a3349-fae0-459e-8632-431fc36beb1b.4368.9:8", 2023-05-24T21:25:32.760 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000109", 2023-05-24T21:25:32.760 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-10:299a3349-fae0-459e-8632-431fc36beb1b.4368.10:8", 2023-05-24T21:25:32.760 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.full-sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-13:299a3349-fae0-459e-8632-431fc36beb1b.4368.13", 2023-05-24T21:25:32.760 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000013", 2023-05-24T21:25:32.761 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-33:299a3349-fae0-459e-8632-431fc36beb1b.4368.33:6", 2023-05-24T21:25:32.761 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-31:299a3349-fae0-459e-8632-431fc36beb1b.4368.31:10", 2023-05-24T21:25:32.761 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-11:299a3349-fae0-459e-8632-431fc36beb1b.4368.11:9", 2023-05-24T21:25:32.761 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-8:299a3349-fae0-459e-8632-431fc36beb1b.4368.8:5", 2023-05-24T21:25:32.761 INFO:teuthology.orchestra.run.smithi071.stdout: "data_log.1.HYR6Mj0wKne.0", 2023-05-24T21:25:32.761 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-17:299a3349-fae0-459e-8632-431fc36beb1b.4368.17:9", 2023-05-24T21:25:32.762 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000051", 2023-05-24T21:25:32.762 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-8:299a3349-fae0-459e-8632-431fc36beb1b.4368.8:10", 2023-05-24T21:25:32.762 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000106", 2023-05-24T21:25:32.762 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000017", 2023-05-24T21:25:32.762 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-32:299a3349-fae0-459e-8632-431fc36beb1b.4368.32:8", 2023-05-24T21:25:32.762 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-17:299a3349-fae0-459e-8632-431fc36beb1b.4368.17:7", 2023-05-24T21:25:32.762 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-18:299a3349-fae0-459e-8632-431fc36beb1b.4368.18:3", 2023-05-24T21:25:32.763 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-33:299a3349-fae0-459e-8632-431fc36beb1b.4368.33:1", 2023-05-24T21:25:32.763 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-10:299a3349-fae0-459e-8632-431fc36beb1b.4368.10:3", 2023-05-24T21:25:32.763 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000023", 2023-05-24T21:25:32.763 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-9:299a3349-fae0-459e-8632-431fc36beb1b.4368.9:9", 2023-05-24T21:25:32.763 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-32:299a3349-fae0-459e-8632-431fc36beb1b.4368.32:6", 2023-05-24T21:25:32.763 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-32:299a3349-fae0-459e-8632-431fc36beb1b.4368.32:3", 2023-05-24T21:25:32.763 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000107", 2023-05-24T21:25:32.764 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-34:299a3349-fae0-459e-8632-431fc36beb1b.4368.34:9", 2023-05-24T21:25:32.764 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000046", 2023-05-24T21:25:32.764 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-15:299a3349-fae0-459e-8632-431fc36beb1b.4368.15:4", 2023-05-24T21:25:32.764 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-35:299a3349-fae0-459e-8632-431fc36beb1b.5347.1:3", 2023-05-24T21:25:32.764 INFO:teuthology.orchestra.run.smithi071.stdout: "data_log.2", 2023-05-24T21:25:32.764 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-12:299a3349-fae0-459e-8632-431fc36beb1b.4368.12:3", 2023-05-24T21:25:32.764 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.full-sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-10:299a3349-fae0-459e-8632-431fc36beb1b.4368.10", 2023-05-24T21:25:32.765 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000067", 2023-05-24T21:25:32.765 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-12:299a3349-fae0-459e-8632-431fc36beb1b.4368.12:1", 2023-05-24T21:25:32.765 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-18:299a3349-fae0-459e-8632-431fc36beb1b.4368.18:4", 2023-05-24T21:25:32.765 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-35:299a3349-fae0-459e-8632-431fc36beb1b.5347.1:9", 2023-05-24T21:25:32.765 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-10:299a3349-fae0-459e-8632-431fc36beb1b.4368.10:10", 2023-05-24T21:25:32.765 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-14:299a3349-fae0-459e-8632-431fc36beb1b.4368.14:1", 2023-05-24T21:25:32.766 INFO:teuthology.orchestra.run.smithi071.stdout: "sync.error-log.9", 2023-05-24T21:25:32.766 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000086", 2023-05-24T21:25:32.766 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.full-sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-12:299a3349-fae0-459e-8632-431fc36beb1b.4368.12", 2023-05-24T21:25:32.766 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-32:299a3349-fae0-459e-8632-431fc36beb1b.4368.32:7", 2023-05-24T21:25:32.766 INFO:teuthology.orchestra.run.smithi071.stdout: "sync.error-log.5", 2023-05-24T21:25:32.766 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.full-sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-7:299a3349-fae0-459e-8632-431fc36beb1b.4368.7", 2023-05-24T21:25:32.766 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.full-sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-18:299a3349-fae0-459e-8632-431fc36beb1b.4368.18", 2023-05-24T21:25:32.767 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-31:299a3349-fae0-459e-8632-431fc36beb1b.4368.31:7", 2023-05-24T21:25:32.767 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-10:299a3349-fae0-459e-8632-431fc36beb1b.4368.10:6", 2023-05-24T21:25:32.767 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-16:299a3349-fae0-459e-8632-431fc36beb1b.4368.16:7", 2023-05-24T21:25:32.767 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.full-sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-9:299a3349-fae0-459e-8632-431fc36beb1b.4368.9", 2023-05-24T21:25:32.767 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-9:299a3349-fae0-459e-8632-431fc36beb1b.4368.9:10", 2023-05-24T21:25:32.767 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-7:299a3349-fae0-459e-8632-431fc36beb1b.4368.7:3", 2023-05-24T21:25:32.767 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000050", 2023-05-24T21:25:32.768 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-13:299a3349-fae0-459e-8632-431fc36beb1b.4368.13:1", 2023-05-24T21:25:32.768 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000102", 2023-05-24T21:25:32.768 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-34:299a3349-fae0-459e-8632-431fc36beb1b.4368.34:6", 2023-05-24T21:25:32.768 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-31:299a3349-fae0-459e-8632-431fc36beb1b.4368.31:8", 2023-05-24T21:25:32.768 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-16:299a3349-fae0-459e-8632-431fc36beb1b.4368.16:2", 2023-05-24T21:25:32.768 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-34:299a3349-fae0-459e-8632-431fc36beb1b.4368.34:7", 2023-05-24T21:25:32.768 INFO:teuthology.orchestra.run.smithi071.stdout: "data_log.2.2geE6Htnfhl.0", 2023-05-24T21:25:32.769 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000124", 2023-05-24T21:25:32.769 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000001", 2023-05-24T21:25:32.769 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000126", 2023-05-24T21:25:32.769 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-33:299a3349-fae0-459e-8632-431fc36beb1b.4368.33:2", 2023-05-24T21:25:32.769 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-18:299a3349-fae0-459e-8632-431fc36beb1b.4368.18:7", 2023-05-24T21:25:32.769 INFO:teuthology.orchestra.run.smithi071.stdout: "meta.log.5c04936b-70ea-4415-b4eb-3aad3dc612e5.0", 2023-05-24T21:25:32.769 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000055", 2023-05-24T21:25:32.770 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000061", 2023-05-24T21:25:32.770 INFO:teuthology.orchestra.run.smithi071.stdout: "datalog.sync-status.shard.5af1588c-f9b9-4687-a040-ce6e0cc808d9.0", 2023-05-24T21:25:32.770 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-8:299a3349-fae0-459e-8632-431fc36beb1b.4368.8:7", 2023-05-24T21:25:32.770 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-16:299a3349-fae0-459e-8632-431fc36beb1b.4368.16:5", 2023-05-24T21:25:32.770 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-9:299a3349-fae0-459e-8632-431fc36beb1b.4368.9:3", 2023-05-24T21:25:32.770 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-15:299a3349-fae0-459e-8632-431fc36beb1b.4368.15:3", 2023-05-24T21:25:32.771 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000097", 2023-05-24T21:25:32.771 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-33:299a3349-fae0-459e-8632-431fc36beb1b.4368.33:7", 2023-05-24T21:25:32.771 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000042", 2023-05-24T21:25:32.771 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000075", 2023-05-24T21:25:32.771 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000120", 2023-05-24T21:25:32.771 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000030", 2023-05-24T21:25:32.771 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-10:299a3349-fae0-459e-8632-431fc36beb1b.4368.10:0", 2023-05-24T21:25:32.772 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.5af1588c-f9b9-4687-a040-ce6e0cc808d9:qskrps-15:299a3349-fae0-459e-8632-431fc36beb1b.4368.15:9" 2023-05-24T21:25:32.772 INFO:teuthology.orchestra.run.smithi071.stdout:] 2023-05-24T21:25:32.772 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T21:25:32.773 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:25:32.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:25:32.774 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:25:32.821 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:25:32.821 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:25:32.885 INFO:teuthology.orchestra.run.smithi071.stdout:[ 2023-05-24T21:25:32.885 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000078", 2023-05-24T21:25:32.885 INFO:teuthology.orchestra.run.smithi071.stdout: "meta.history", 2023-05-24T21:25:32.885 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000070", 2023-05-24T21:25:32.886 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-12:299a3349-fae0-459e-8632-431fc36beb1b.4368.12:3", 2023-05-24T21:25:32.886 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000104", 2023-05-24T21:25:32.886 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-34:299a3349-fae0-459e-8632-431fc36beb1b.4368.34:5", 2023-05-24T21:25:32.886 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-34:299a3349-fae0-459e-8632-431fc36beb1b.4368.34:10", 2023-05-24T21:25:32.886 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000026", 2023-05-24T21:25:32.886 INFO:teuthology.orchestra.run.smithi071.stdout: "datalog.sync-status.shard.299a3349-fae0-459e-8632-431fc36beb1b.2", 2023-05-24T21:25:32.887 INFO:teuthology.orchestra.run.smithi071.stdout: "data_log.2.7EwG03dWf0p.0", 2023-05-24T21:25:32.887 INFO:teuthology.orchestra.run.smithi071.stdout: "sync.error-log.8", 2023-05-24T21:25:32.887 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-7:299a3349-fae0-459e-8632-431fc36beb1b.4368.7:2", 2023-05-24T21:25:32.887 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000028", 2023-05-24T21:25:32.887 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-16:299a3349-fae0-459e-8632-431fc36beb1b.4368.16:3", 2023-05-24T21:25:32.887 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-7:299a3349-fae0-459e-8632-431fc36beb1b.4368.7:6", 2023-05-24T21:25:32.887 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000040", 2023-05-24T21:25:32.888 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-16:299a3349-fae0-459e-8632-431fc36beb1b.4368.16:2", 2023-05-24T21:25:32.888 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-17:299a3349-fae0-459e-8632-431fc36beb1b.4368.17:6", 2023-05-24T21:25:32.888 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-17:299a3349-fae0-459e-8632-431fc36beb1b.4368.17:4", 2023-05-24T21:25:32.888 INFO:teuthology.orchestra.run.smithi071.stdout: "sync.error-log.6", 2023-05-24T21:25:32.888 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000015", 2023-05-24T21:25:32.888 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000069", 2023-05-24T21:25:32.889 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000095", 2023-05-24T21:25:32.889 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000003", 2023-05-24T21:25:32.889 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000047", 2023-05-24T21:25:32.889 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-35:299a3349-fae0-459e-8632-431fc36beb1b.5347.1:1", 2023-05-24T21:25:32.889 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-31:299a3349-fae0-459e-8632-431fc36beb1b.4368.31:8", 2023-05-24T21:25:32.889 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-12:299a3349-fae0-459e-8632-431fc36beb1b.4368.12:8", 2023-05-24T21:25:32.889 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000035", 2023-05-24T21:25:32.890 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000037", 2023-05-24T21:25:32.890 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-10:299a3349-fae0-459e-8632-431fc36beb1b.4368.10:2", 2023-05-24T21:25:32.890 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-34:299a3349-fae0-459e-8632-431fc36beb1b.4368.34:9", 2023-05-24T21:25:32.890 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-12:299a3349-fae0-459e-8632-431fc36beb1b.4368.12:6", 2023-05-24T21:25:32.890 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-7:299a3349-fae0-459e-8632-431fc36beb1b.4368.7:0", 2023-05-24T21:25:32.890 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000024", 2023-05-24T21:25:32.890 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-13:299a3349-fae0-459e-8632-431fc36beb1b.4368.13:7", 2023-05-24T21:25:32.891 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-32:299a3349-fae0-459e-8632-431fc36beb1b.4368.32:1", 2023-05-24T21:25:32.891 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000000", 2023-05-24T21:25:32.891 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000031", 2023-05-24T21:25:32.891 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-16:299a3349-fae0-459e-8632-431fc36beb1b.4368.16:8", 2023-05-24T21:25:32.891 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-33:299a3349-fae0-459e-8632-431fc36beb1b.4368.33:9", 2023-05-24T21:25:32.891 INFO:teuthology.orchestra.run.smithi071.stdout: "data_log.3", 2023-05-24T21:25:32.891 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-35:299a3349-fae0-459e-8632-431fc36beb1b.5347.1:2", 2023-05-24T21:25:32.892 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-16:299a3349-fae0-459e-8632-431fc36beb1b.4368.16:9", 2023-05-24T21:25:32.892 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-11:299a3349-fae0-459e-8632-431fc36beb1b.4368.11:7", 2023-05-24T21:25:32.892 INFO:teuthology.orchestra.run.smithi071.stdout: "data_loggenerations_metadata", 2023-05-24T21:25:32.892 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-10:299a3349-fae0-459e-8632-431fc36beb1b.4368.10:5", 2023-05-24T21:25:32.892 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-14:299a3349-fae0-459e-8632-431fc36beb1b.4368.14:10", 2023-05-24T21:25:32.892 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000076", 2023-05-24T21:25:32.893 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-17:299a3349-fae0-459e-8632-431fc36beb1b.4368.17:8", 2023-05-24T21:25:32.893 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000113", 2023-05-24T21:25:32.893 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000005", 2023-05-24T21:25:32.893 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000011", 2023-05-24T21:25:32.893 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000058", 2023-05-24T21:25:32.893 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.full-sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-10:299a3349-fae0-459e-8632-431fc36beb1b.4368.10", 2023-05-24T21:25:32.893 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000052", 2023-05-24T21:25:32.893 INFO:teuthology.orchestra.run.smithi071.stdout: "datalog.sync-status.shard.299a3349-fae0-459e-8632-431fc36beb1b.0", 2023-05-24T21:25:32.894 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-12:299a3349-fae0-459e-8632-431fc36beb1b.4368.12:2", 2023-05-24T21:25:32.894 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-35:299a3349-fae0-459e-8632-431fc36beb1b.5347.1:7", 2023-05-24T21:25:32.894 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-12:299a3349-fae0-459e-8632-431fc36beb1b.4368.12:4", 2023-05-24T21:25:32.894 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-11:299a3349-fae0-459e-8632-431fc36beb1b.4368.11:6", 2023-05-24T21:25:32.894 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-8:299a3349-fae0-459e-8632-431fc36beb1b.4368.8:3", 2023-05-24T21:25:32.894 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-12:299a3349-fae0-459e-8632-431fc36beb1b.4368.12:7", 2023-05-24T21:25:32.894 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-32:299a3349-fae0-459e-8632-431fc36beb1b.4368.32:6", 2023-05-24T21:25:32.895 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.full-sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-12:299a3349-fae0-459e-8632-431fc36beb1b.4368.12", 2023-05-24T21:25:32.895 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000088", 2023-05-24T21:25:32.895 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-31:299a3349-fae0-459e-8632-431fc36beb1b.4368.31:2", 2023-05-24T21:25:32.895 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000080", 2023-05-24T21:25:32.895 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000090", 2023-05-24T21:25:32.895 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-15:299a3349-fae0-459e-8632-431fc36beb1b.4368.15:10", 2023-05-24T21:25:32.895 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-31:299a3349-fae0-459e-8632-431fc36beb1b.4368.31:1", 2023-05-24T21:25:32.896 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000110", 2023-05-24T21:25:32.896 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-33:299a3349-fae0-459e-8632-431fc36beb1b.4368.33:6", 2023-05-24T21:25:32.896 INFO:teuthology.orchestra.run.smithi071.stdout: "datalog.sync-status.shard.299a3349-fae0-459e-8632-431fc36beb1b.2.retry", 2023-05-24T21:25:32.896 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-11:299a3349-fae0-459e-8632-431fc36beb1b.4368.11:3", 2023-05-24T21:25:32.896 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-11:299a3349-fae0-459e-8632-431fc36beb1b.4368.11:4", 2023-05-24T21:25:32.896 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000096", 2023-05-24T21:25:32.897 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000087", 2023-05-24T21:25:32.897 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-34:299a3349-fae0-459e-8632-431fc36beb1b.4368.34:0", 2023-05-24T21:25:32.897 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-31:299a3349-fae0-459e-8632-431fc36beb1b.4368.31:7", 2023-05-24T21:25:32.897 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.full-sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-33:299a3349-fae0-459e-8632-431fc36beb1b.4368.33", 2023-05-24T21:25:32.897 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000008", 2023-05-24T21:25:32.897 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-14:299a3349-fae0-459e-8632-431fc36beb1b.4368.14:9", 2023-05-24T21:25:32.897 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-7:299a3349-fae0-459e-8632-431fc36beb1b.4368.7:7", 2023-05-24T21:25:32.897 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.full-sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-35:299a3349-fae0-459e-8632-431fc36beb1b.5347.1", 2023-05-24T21:25:32.898 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000006", 2023-05-24T21:25:32.898 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000029", 2023-05-24T21:25:32.898 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-16:299a3349-fae0-459e-8632-431fc36beb1b.4368.16:1", 2023-05-24T21:25:32.898 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000089", 2023-05-24T21:25:32.898 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000022", 2023-05-24T21:25:32.898 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-17:299a3349-fae0-459e-8632-431fc36beb1b.4368.17:3", 2023-05-24T21:25:32.898 INFO:teuthology.orchestra.run.smithi071.stdout: "bilog.trim", 2023-05-24T21:25:32.899 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000016", 2023-05-24T21:25:32.899 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000041", 2023-05-24T21:25:32.899 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-14:299a3349-fae0-459e-8632-431fc36beb1b.4368.14:1", 2023-05-24T21:25:32.899 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-8:299a3349-fae0-459e-8632-431fc36beb1b.4368.8:5", 2023-05-24T21:25:32.899 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000018", 2023-05-24T21:25:32.899 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-16:299a3349-fae0-459e-8632-431fc36beb1b.4368.16:10", 2023-05-24T21:25:32.899 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000092", 2023-05-24T21:25:32.900 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-9:299a3349-fae0-459e-8632-431fc36beb1b.4368.9:2", 2023-05-24T21:25:32.900 INFO:teuthology.orchestra.run.smithi071.stdout: "sync.error-log.4", 2023-05-24T21:25:32.900 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-10:299a3349-fae0-459e-8632-431fc36beb1b.4368.10:0", 2023-05-24T21:25:32.900 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-13:299a3349-fae0-459e-8632-431fc36beb1b.4368.13:8", 2023-05-24T21:25:32.900 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-15:299a3349-fae0-459e-8632-431fc36beb1b.4368.15:8", 2023-05-24T21:25:32.900 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000014", 2023-05-24T21:25:32.901 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000112", 2023-05-24T21:25:32.901 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000007", 2023-05-24T21:25:32.901 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000021", 2023-05-24T21:25:32.901 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-32:299a3349-fae0-459e-8632-431fc36beb1b.4368.32:10", 2023-05-24T21:25:32.901 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000064", 2023-05-24T21:25:32.901 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000071", 2023-05-24T21:25:32.901 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000074", 2023-05-24T21:25:32.902 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-14:299a3349-fae0-459e-8632-431fc36beb1b.4368.14:7", 2023-05-24T21:25:32.902 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-16:299a3349-fae0-459e-8632-431fc36beb1b.4368.16:7", 2023-05-24T21:25:32.902 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.full-sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-16:299a3349-fae0-459e-8632-431fc36beb1b.4368.16", 2023-05-24T21:25:32.902 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-14:299a3349-fae0-459e-8632-431fc36beb1b.4368.14:8", 2023-05-24T21:25:32.902 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-14:299a3349-fae0-459e-8632-431fc36beb1b.4368.14:0", 2023-05-24T21:25:32.902 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000081", 2023-05-24T21:25:32.902 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000009", 2023-05-24T21:25:32.903 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-13:299a3349-fae0-459e-8632-431fc36beb1b.4368.13:1", 2023-05-24T21:25:32.903 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000121", 2023-05-24T21:25:32.903 INFO:teuthology.orchestra.run.smithi071.stdout: "data_log.1", 2023-05-24T21:25:32.903 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-18:299a3349-fae0-459e-8632-431fc36beb1b.4368.18:3", 2023-05-24T21:25:32.903 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.full-sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-32:299a3349-fae0-459e-8632-431fc36beb1b.4368.32", 2023-05-24T21:25:32.903 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-15:299a3349-fae0-459e-8632-431fc36beb1b.4368.15:9", 2023-05-24T21:25:32.903 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000125", 2023-05-24T21:25:32.904 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000082", 2023-05-24T21:25:32.904 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-7:299a3349-fae0-459e-8632-431fc36beb1b.4368.7:5", 2023-05-24T21:25:32.904 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-16:299a3349-fae0-459e-8632-431fc36beb1b.4368.16:4", 2023-05-24T21:25:32.904 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-18:299a3349-fae0-459e-8632-431fc36beb1b.4368.18:0", 2023-05-24T21:25:32.904 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000105", 2023-05-24T21:25:32.904 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000059", 2023-05-24T21:25:32.904 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-33:299a3349-fae0-459e-8632-431fc36beb1b.4368.33:1", 2023-05-24T21:25:32.905 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-7:299a3349-fae0-459e-8632-431fc36beb1b.4368.7:1", 2023-05-24T21:25:32.905 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-35:299a3349-fae0-459e-8632-431fc36beb1b.5347.1:0", 2023-05-24T21:25:32.905 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-17:299a3349-fae0-459e-8632-431fc36beb1b.4368.17:9", 2023-05-24T21:25:32.905 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-18:299a3349-fae0-459e-8632-431fc36beb1b.4368.18:2", 2023-05-24T21:25:32.905 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000077", 2023-05-24T21:25:32.906 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000032", 2023-05-24T21:25:32.906 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-18:299a3349-fae0-459e-8632-431fc36beb1b.4368.18:6", 2023-05-24T21:25:32.906 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-9:299a3349-fae0-459e-8632-431fc36beb1b.4368.9:10", 2023-05-24T21:25:32.906 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.full-sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-18:299a3349-fae0-459e-8632-431fc36beb1b.4368.18", 2023-05-24T21:25:32.906 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-18:299a3349-fae0-459e-8632-431fc36beb1b.4368.18:7", 2023-05-24T21:25:32.906 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-13:299a3349-fae0-459e-8632-431fc36beb1b.4368.13:5", 2023-05-24T21:25:32.906 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-35:299a3349-fae0-459e-8632-431fc36beb1b.5347.1:5", 2023-05-24T21:25:32.907 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-11:299a3349-fae0-459e-8632-431fc36beb1b.4368.11:0", 2023-05-24T21:25:32.907 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-33:299a3349-fae0-459e-8632-431fc36beb1b.4368.33:10", 2023-05-24T21:25:32.907 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000053", 2023-05-24T21:25:32.907 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000091", 2023-05-24T21:25:32.907 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-34:299a3349-fae0-459e-8632-431fc36beb1b.4368.34:6", 2023-05-24T21:25:32.907 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-8:299a3349-fae0-459e-8632-431fc36beb1b.4368.8:9", 2023-05-24T21:25:32.907 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-9:299a3349-fae0-459e-8632-431fc36beb1b.4368.9:4", 2023-05-24T21:25:32.908 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-33:299a3349-fae0-459e-8632-431fc36beb1b.4368.33:2", 2023-05-24T21:25:32.908 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000065", 2023-05-24T21:25:32.908 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000083", 2023-05-24T21:25:32.908 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000010", 2023-05-24T21:25:32.908 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-32:299a3349-fae0-459e-8632-431fc36beb1b.4368.32:9", 2023-05-24T21:25:32.908 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-10:299a3349-fae0-459e-8632-431fc36beb1b.4368.10:1", 2023-05-24T21:25:32.908 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-33:299a3349-fae0-459e-8632-431fc36beb1b.4368.33:5", 2023-05-24T21:25:32.909 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-31:299a3349-fae0-459e-8632-431fc36beb1b.4368.31:3", 2023-05-24T21:25:32.909 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-13:299a3349-fae0-459e-8632-431fc36beb1b.4368.13:0", 2023-05-24T21:25:32.909 INFO:teuthology.orchestra.run.smithi071.stdout: "sync.error-log.7", 2023-05-24T21:25:32.909 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-7:299a3349-fae0-459e-8632-431fc36beb1b.4368.7:9", 2023-05-24T21:25:32.909 INFO:teuthology.orchestra.run.smithi071.stdout: "sync.error-log.3", 2023-05-24T21:25:32.909 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-31:299a3349-fae0-459e-8632-431fc36beb1b.4368.31:9", 2023-05-24T21:25:32.910 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.full-sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-13:299a3349-fae0-459e-8632-431fc36beb1b.4368.13", 2023-05-24T21:25:32.910 INFO:teuthology.orchestra.run.smithi071.stdout: "data_log.3.1vwPHAKxRuU.0", 2023-05-24T21:25:32.910 INFO:teuthology.orchestra.run.smithi071.stdout: "mdlog.sync-status.shard.2", 2023-05-24T21:25:32.910 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.full-sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-14:299a3349-fae0-459e-8632-431fc36beb1b.4368.14", 2023-05-24T21:25:32.910 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-11:299a3349-fae0-459e-8632-431fc36beb1b.4368.11:10", 2023-05-24T21:25:32.910 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000045", 2023-05-24T21:25:32.910 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-7:299a3349-fae0-459e-8632-431fc36beb1b.4368.7:10", 2023-05-24T21:25:32.911 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-10:299a3349-fae0-459e-8632-431fc36beb1b.4368.10:3", 2023-05-24T21:25:32.911 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000002", 2023-05-24T21:25:32.911 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-11:299a3349-fae0-459e-8632-431fc36beb1b.4368.11:8", 2023-05-24T21:25:32.911 INFO:teuthology.orchestra.run.smithi071.stdout: "sync.error-log.12", 2023-05-24T21:25:32.911 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-34:299a3349-fae0-459e-8632-431fc36beb1b.4368.34:3", 2023-05-24T21:25:32.911 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000116", 2023-05-24T21:25:32.911 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-17:299a3349-fae0-459e-8632-431fc36beb1b.4368.17:0", 2023-05-24T21:25:32.912 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000034", 2023-05-24T21:25:32.912 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-15:299a3349-fae0-459e-8632-431fc36beb1b.4368.15:5", 2023-05-24T21:25:32.912 INFO:teuthology.orchestra.run.smithi071.stdout: "datalog.sync-status.shard.299a3349-fae0-459e-8632-431fc36beb1b.3.retry", 2023-05-24T21:25:32.912 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000101", 2023-05-24T21:25:32.912 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-10:299a3349-fae0-459e-8632-431fc36beb1b.4368.10:9", 2023-05-24T21:25:32.912 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-10:299a3349-fae0-459e-8632-431fc36beb1b.4368.10:6", 2023-05-24T21:25:32.912 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.full-sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-34:299a3349-fae0-459e-8632-431fc36beb1b.4368.34", 2023-05-24T21:25:32.913 INFO:teuthology.orchestra.run.smithi071.stdout: "sync.error-log.13", 2023-05-24T21:25:32.913 INFO:teuthology.orchestra.run.smithi071.stdout: "mdlog.sync-status.shard.1", 2023-05-24T21:25:32.913 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-18:299a3349-fae0-459e-8632-431fc36beb1b.4368.18:1", 2023-05-24T21:25:32.913 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000079", 2023-05-24T21:25:32.913 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000049", 2023-05-24T21:25:32.913 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000117", 2023-05-24T21:25:32.913 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-34:299a3349-fae0-459e-8632-431fc36beb1b.4368.34:1", 2023-05-24T21:25:32.914 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000044", 2023-05-24T21:25:32.914 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-35:299a3349-fae0-459e-8632-431fc36beb1b.5347.1:6", 2023-05-24T21:25:32.914 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-33:299a3349-fae0-459e-8632-431fc36beb1b.4368.33:3", 2023-05-24T21:25:32.914 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-7:299a3349-fae0-459e-8632-431fc36beb1b.4368.7:3", 2023-05-24T21:25:32.914 INFO:teuthology.orchestra.run.smithi071.stdout: "sync.error-log.14", 2023-05-24T21:25:32.914 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-34:299a3349-fae0-459e-8632-431fc36beb1b.4368.34:8", 2023-05-24T21:25:32.915 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000066", 2023-05-24T21:25:32.915 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-14:299a3349-fae0-459e-8632-431fc36beb1b.4368.14:5", 2023-05-24T21:25:32.915 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000068", 2023-05-24T21:25:32.915 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000085", 2023-05-24T21:25:32.915 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-11:299a3349-fae0-459e-8632-431fc36beb1b.4368.11:2", 2023-05-24T21:25:32.915 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.full-sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-11:299a3349-fae0-459e-8632-431fc36beb1b.4368.11", 2023-05-24T21:25:32.915 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-12:299a3349-fae0-459e-8632-431fc36beb1b.4368.12:5", 2023-05-24T21:25:32.916 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-34:299a3349-fae0-459e-8632-431fc36beb1b.4368.34:7", 2023-05-24T21:25:32.916 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000073", 2023-05-24T21:25:32.916 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-11:299a3349-fae0-459e-8632-431fc36beb1b.4368.11:1", 2023-05-24T21:25:32.916 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000038", 2023-05-24T21:25:32.916 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-31:299a3349-fae0-459e-8632-431fc36beb1b.4368.31:5", 2023-05-24T21:25:32.916 INFO:teuthology.orchestra.run.smithi071.stdout: "sync.error-log.10", 2023-05-24T21:25:32.916 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000118", 2023-05-24T21:25:32.917 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-16:299a3349-fae0-459e-8632-431fc36beb1b.4368.16:6", 2023-05-24T21:25:32.917 INFO:teuthology.orchestra.run.smithi071.stdout: "datalog.sync-status.shard.299a3349-fae0-459e-8632-431fc36beb1b.0.retry", 2023-05-24T21:25:32.917 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-10:299a3349-fae0-459e-8632-431fc36beb1b.4368.10:4", 2023-05-24T21:25:32.917 INFO:teuthology.orchestra.run.smithi071.stdout: "datalog.sync-status.299a3349-fae0-459e-8632-431fc36beb1b", 2023-05-24T21:25:32.917 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000036", 2023-05-24T21:25:32.917 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-13:299a3349-fae0-459e-8632-431fc36beb1b.4368.13:9", 2023-05-24T21:25:32.917 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000103", 2023-05-24T21:25:32.918 INFO:teuthology.orchestra.run.smithi071.stdout: "mdlog.sync-status.shard.3", 2023-05-24T21:25:32.918 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000119", 2023-05-24T21:25:32.918 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000098", 2023-05-24T21:25:32.918 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-31:299a3349-fae0-459e-8632-431fc36beb1b.4368.31:4", 2023-05-24T21:25:32.918 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-17:299a3349-fae0-459e-8632-431fc36beb1b.4368.17:7", 2023-05-24T21:25:32.918 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-31:299a3349-fae0-459e-8632-431fc36beb1b.4368.31:0", 2023-05-24T21:25:32.918 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-35:299a3349-fae0-459e-8632-431fc36beb1b.5347.1:9", 2023-05-24T21:25:32.919 INFO:teuthology.orchestra.run.smithi071.stdout: "mdlog.sync-status.shard.0", 2023-05-24T21:25:32.919 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-35:299a3349-fae0-459e-8632-431fc36beb1b.5347.1:8", 2023-05-24T21:25:32.919 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000027", 2023-05-24T21:25:32.919 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-35:299a3349-fae0-459e-8632-431fc36beb1b.5347.1:3", 2023-05-24T21:25:32.919 INFO:teuthology.orchestra.run.smithi071.stdout: "mdlog.sync-status", 2023-05-24T21:25:32.919 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000019", 2023-05-24T21:25:32.919 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-31:299a3349-fae0-459e-8632-431fc36beb1b.4368.31:6", 2023-05-24T21:25:32.920 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-9:299a3349-fae0-459e-8632-431fc36beb1b.4368.9:8", 2023-05-24T21:25:32.920 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000039", 2023-05-24T21:25:32.920 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-12:299a3349-fae0-459e-8632-431fc36beb1b.4368.12:9", 2023-05-24T21:25:32.920 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-17:299a3349-fae0-459e-8632-431fc36beb1b.4368.17:1", 2023-05-24T21:25:32.920 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-16:299a3349-fae0-459e-8632-431fc36beb1b.4368.16:5", 2023-05-24T21:25:32.920 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-8:299a3349-fae0-459e-8632-431fc36beb1b.4368.8:8", 2023-05-24T21:25:32.920 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-18:299a3349-fae0-459e-8632-431fc36beb1b.4368.18:9", 2023-05-24T21:25:32.921 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000100", 2023-05-24T21:25:32.921 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-32:299a3349-fae0-459e-8632-431fc36beb1b.4368.32:0", 2023-05-24T21:25:32.921 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-17:299a3349-fae0-459e-8632-431fc36beb1b.4368.17:10", 2023-05-24T21:25:32.921 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-8:299a3349-fae0-459e-8632-431fc36beb1b.4368.8:2", 2023-05-24T21:25:32.921 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000093", 2023-05-24T21:25:32.921 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.full-sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-7:299a3349-fae0-459e-8632-431fc36beb1b.4368.7", 2023-05-24T21:25:32.922 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000004", 2023-05-24T21:25:32.922 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000063", 2023-05-24T21:25:32.922 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000122", 2023-05-24T21:25:32.922 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000057", 2023-05-24T21:25:32.922 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000054", 2023-05-24T21:25:32.922 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000114", 2023-05-24T21:25:32.922 INFO:teuthology.orchestra.run.smithi071.stdout: "sync.error-log.2", 2023-05-24T21:25:32.923 INFO:teuthology.orchestra.run.smithi071.stdout: "data_log.0", 2023-05-24T21:25:32.923 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-33:299a3349-fae0-459e-8632-431fc36beb1b.4368.33:8", 2023-05-24T21:25:32.923 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000012", 2023-05-24T21:25:32.923 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-13:299a3349-fae0-459e-8632-431fc36beb1b.4368.13:10", 2023-05-24T21:25:32.923 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-11:299a3349-fae0-459e-8632-431fc36beb1b.4368.11:9", 2023-05-24T21:25:32.923 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-33:299a3349-fae0-459e-8632-431fc36beb1b.4368.33:0", 2023-05-24T21:25:32.923 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000084", 2023-05-24T21:25:32.924 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000043", 2023-05-24T21:25:32.924 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-34:299a3349-fae0-459e-8632-431fc36beb1b.4368.34:2", 2023-05-24T21:25:32.924 INFO:teuthology.orchestra.run.smithi071.stdout: "meta.log.5c04936b-70ea-4415-b4eb-3aad3dc612e5.1", 2023-05-24T21:25:32.924 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-8:299a3349-fae0-459e-8632-431fc36beb1b.4368.8:7", 2023-05-24T21:25:32.924 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-18:299a3349-fae0-459e-8632-431fc36beb1b.4368.18:5", 2023-05-24T21:25:32.924 INFO:teuthology.orchestra.run.smithi071.stdout: "sync.error-log.11", 2023-05-24T21:25:32.924 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-32:299a3349-fae0-459e-8632-431fc36beb1b.4368.32:5", 2023-05-24T21:25:32.925 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-14:299a3349-fae0-459e-8632-431fc36beb1b.4368.14:6", 2023-05-24T21:25:32.925 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000111", 2023-05-24T21:25:32.925 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000048", 2023-05-24T21:25:32.925 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000020", 2023-05-24T21:25:32.925 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-32:299a3349-fae0-459e-8632-431fc36beb1b.4368.32:3", 2023-05-24T21:25:32.925 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-32:299a3349-fae0-459e-8632-431fc36beb1b.4368.32:7", 2023-05-24T21:25:32.925 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-15:299a3349-fae0-459e-8632-431fc36beb1b.4368.15:4", 2023-05-24T21:25:32.926 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-15:299a3349-fae0-459e-8632-431fc36beb1b.4368.15:1", 2023-05-24T21:25:32.926 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000099", 2023-05-24T21:25:32.926 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000056", 2023-05-24T21:25:32.926 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-9:299a3349-fae0-459e-8632-431fc36beb1b.4368.9:1", 2023-05-24T21:25:32.926 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000072", 2023-05-24T21:25:32.926 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-8:299a3349-fae0-459e-8632-431fc36beb1b.4368.8:10", 2023-05-24T21:25:32.926 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-13:299a3349-fae0-459e-8632-431fc36beb1b.4368.13:4", 2023-05-24T21:25:32.927 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000062", 2023-05-24T21:25:32.927 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000115", 2023-05-24T21:25:32.927 INFO:teuthology.orchestra.run.smithi071.stdout: "sync.error-log.1", 2023-05-24T21:25:32.927 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000033", 2023-05-24T21:25:32.927 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-14:299a3349-fae0-459e-8632-431fc36beb1b.4368.14:3", 2023-05-24T21:25:32.927 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-32:299a3349-fae0-459e-8632-431fc36beb1b.4368.32:2", 2023-05-24T21:25:32.927 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-16:299a3349-fae0-459e-8632-431fc36beb1b.4368.16:0", 2023-05-24T21:25:32.928 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-9:299a3349-fae0-459e-8632-431fc36beb1b.4368.9:5", 2023-05-24T21:25:32.928 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-31:299a3349-fae0-459e-8632-431fc36beb1b.4368.31:10", 2023-05-24T21:25:32.928 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-18:299a3349-fae0-459e-8632-431fc36beb1b.4368.18:8", 2023-05-24T21:25:32.928 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-10:299a3349-fae0-459e-8632-431fc36beb1b.4368.10:8", 2023-05-24T21:25:32.928 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000025", 2023-05-24T21:25:32.928 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-15:299a3349-fae0-459e-8632-431fc36beb1b.4368.15:2", 2023-05-24T21:25:32.928 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-15:299a3349-fae0-459e-8632-431fc36beb1b.4368.15:7", 2023-05-24T21:25:32.929 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000123", 2023-05-24T21:25:32.929 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-12:299a3349-fae0-459e-8632-431fc36beb1b.4368.12:1", 2023-05-24T21:25:32.929 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000108", 2023-05-24T21:25:32.929 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000094", 2023-05-24T21:25:32.929 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000060", 2023-05-24T21:25:32.929 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000109", 2023-05-24T21:25:32.929 INFO:teuthology.orchestra.run.smithi071.stdout: "sync.error-log.15", 2023-05-24T21:25:32.930 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-10:299a3349-fae0-459e-8632-431fc36beb1b.4368.10:10", 2023-05-24T21:25:32.930 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000013", 2023-05-24T21:25:32.930 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-35:299a3349-fae0-459e-8632-431fc36beb1b.5347.1:4", 2023-05-24T21:25:32.930 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-13:299a3349-fae0-459e-8632-431fc36beb1b.4368.13:6", 2023-05-24T21:25:32.930 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000051", 2023-05-24T21:25:32.930 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-32:299a3349-fae0-459e-8632-431fc36beb1b.4368.32:4", 2023-05-24T21:25:32.930 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-17:299a3349-fae0-459e-8632-431fc36beb1b.4368.17:5", 2023-05-24T21:25:32.931 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000106", 2023-05-24T21:25:32.931 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-18:299a3349-fae0-459e-8632-431fc36beb1b.4368.18:4", 2023-05-24T21:25:32.931 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-9:299a3349-fae0-459e-8632-431fc36beb1b.4368.9:3", 2023-05-24T21:25:32.931 INFO:teuthology.orchestra.run.smithi071.stdout: "data_log.0.658ImFo2plv.0", 2023-05-24T21:25:32.931 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000017", 2023-05-24T21:25:32.931 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.full-sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-17:299a3349-fae0-459e-8632-431fc36beb1b.4368.17", 2023-05-24T21:25:32.932 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-13:299a3349-fae0-459e-8632-431fc36beb1b.4368.13:2", 2023-05-24T21:25:32.932 INFO:teuthology.orchestra.run.smithi071.stdout: "data_log.1.7Shs2LjJOaD.0", 2023-05-24T21:25:32.932 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-33:299a3349-fae0-459e-8632-431fc36beb1b.4368.33:4", 2023-05-24T21:25:32.932 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-9:299a3349-fae0-459e-8632-431fc36beb1b.4368.9:0", 2023-05-24T21:25:32.932 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000023", 2023-05-24T21:25:32.932 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.full-sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-15:299a3349-fae0-459e-8632-431fc36beb1b.4368.15", 2023-05-24T21:25:32.932 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-11:299a3349-fae0-459e-8632-431fc36beb1b.4368.11:5", 2023-05-24T21:25:32.933 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-15:299a3349-fae0-459e-8632-431fc36beb1b.4368.15:6", 2023-05-24T21:25:32.933 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000107", 2023-05-24T21:25:32.933 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.full-sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-31:299a3349-fae0-459e-8632-431fc36beb1b.4368.31", 2023-05-24T21:25:32.933 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000046", 2023-05-24T21:25:32.933 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-13:299a3349-fae0-459e-8632-431fc36beb1b.4368.13:3", 2023-05-24T21:25:32.933 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-8:299a3349-fae0-459e-8632-431fc36beb1b.4368.8:0", 2023-05-24T21:25:32.933 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-8:299a3349-fae0-459e-8632-431fc36beb1b.4368.8:6", 2023-05-24T21:25:32.934 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-17:299a3349-fae0-459e-8632-431fc36beb1b.4368.17:2", 2023-05-24T21:25:32.934 INFO:teuthology.orchestra.run.smithi071.stdout: "data_log.2", 2023-05-24T21:25:32.934 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000067", 2023-05-24T21:25:32.934 INFO:teuthology.orchestra.run.smithi071.stdout: "sync.error-log.9", 2023-05-24T21:25:32.934 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000086", 2023-05-24T21:25:32.934 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-34:299a3349-fae0-459e-8632-431fc36beb1b.4368.34:4", 2023-05-24T21:25:32.935 INFO:teuthology.orchestra.run.smithi071.stdout: "sync.error-log.5", 2023-05-24T21:25:32.935 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-10:299a3349-fae0-459e-8632-431fc36beb1b.4368.10:7", 2023-05-24T21:25:32.935 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-15:299a3349-fae0-459e-8632-431fc36beb1b.4368.15:3", 2023-05-24T21:25:32.935 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-32:299a3349-fae0-459e-8632-431fc36beb1b.4368.32:8", 2023-05-24T21:25:32.935 INFO:teuthology.orchestra.run.smithi071.stdout: "datalog.sync-status.shard.299a3349-fae0-459e-8632-431fc36beb1b.1.retry", 2023-05-24T21:25:32.935 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-9:299a3349-fae0-459e-8632-431fc36beb1b.4368.9:6", 2023-05-24T21:25:32.935 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-18:299a3349-fae0-459e-8632-431fc36beb1b.4368.18:10", 2023-05-24T21:25:32.936 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-14:299a3349-fae0-459e-8632-431fc36beb1b.4368.14:4", 2023-05-24T21:25:32.936 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-8:299a3349-fae0-459e-8632-431fc36beb1b.4368.8:4", 2023-05-24T21:25:32.936 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-8:299a3349-fae0-459e-8632-431fc36beb1b.4368.8:1", 2023-05-24T21:25:32.936 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-14:299a3349-fae0-459e-8632-431fc36beb1b.4368.14:2", 2023-05-24T21:25:32.936 INFO:teuthology.orchestra.run.smithi071.stdout: "datalog.sync-status.shard.299a3349-fae0-459e-8632-431fc36beb1b.1", 2023-05-24T21:25:32.936 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000050", 2023-05-24T21:25:32.936 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.full-sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-9:299a3349-fae0-459e-8632-431fc36beb1b.4368.9", 2023-05-24T21:25:32.937 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000102", 2023-05-24T21:25:32.937 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-33:299a3349-fae0-459e-8632-431fc36beb1b.4368.33:7", 2023-05-24T21:25:32.937 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-7:299a3349-fae0-459e-8632-431fc36beb1b.4368.7:4", 2023-05-24T21:25:32.937 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-9:299a3349-fae0-459e-8632-431fc36beb1b.4368.9:7", 2023-05-24T21:25:32.937 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000124", 2023-05-24T21:25:32.937 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-9:299a3349-fae0-459e-8632-431fc36beb1b.4368.9:9", 2023-05-24T21:25:32.937 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000001", 2023-05-24T21:25:32.938 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-35:299a3349-fae0-459e-8632-431fc36beb1b.5347.1:10", 2023-05-24T21:25:32.938 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000126", 2023-05-24T21:25:32.938 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-7:299a3349-fae0-459e-8632-431fc36beb1b.4368.7:8", 2023-05-24T21:25:32.938 INFO:teuthology.orchestra.run.smithi071.stdout: "meta.log.5c04936b-70ea-4415-b4eb-3aad3dc612e5.0", 2023-05-24T21:25:32.938 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000055", 2023-05-24T21:25:32.938 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000061", 2023-05-24T21:25:32.939 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000097", 2023-05-24T21:25:32.939 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-12:299a3349-fae0-459e-8632-431fc36beb1b.4368.12:10", 2023-05-24T21:25:32.939 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000042", 2023-05-24T21:25:32.939 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-15:299a3349-fae0-459e-8632-431fc36beb1b.4368.15:0", 2023-05-24T21:25:32.939 INFO:teuthology.orchestra.run.smithi071.stdout: "datalog.sync-status.shard.299a3349-fae0-459e-8632-431fc36beb1b.3", 2023-05-24T21:25:32.939 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000075", 2023-05-24T21:25:32.940 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-12:299a3349-fae0-459e-8632-431fc36beb1b.4368.12:0", 2023-05-24T21:25:32.940 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket.full-sync-status.299a3349-fae0-459e-8632-431fc36beb1b:qskrps-8:299a3349-fae0-459e-8632-431fc36beb1b.4368.8", 2023-05-24T21:25:32.940 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000120", 2023-05-24T21:25:32.940 INFO:teuthology.orchestra.run.smithi071.stdout: "obj_delete_at_hint.0000000030" 2023-05-24T21:25:32.940 INFO:teuthology.orchestra.run.smithi071.stdout:] 2023-05-24T21:25:32.940 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-05-24T21:25:32.941 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-05-24T21:25:32.942 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:25:32.942 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:25:32.943 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:25:32.991 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:25:32.991 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:25:33.059 INFO:teuthology.orchestra.run.smithi071.stdout:[ 2023-05-24T21:25:33.059 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:33.059 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000003890", 2023-05-24T21:25:33.059 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:25:32.597538Z" 2023-05-24T21:25:33.060 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:33.060 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:33.060 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000002055", 2023-05-24T21:25:33.060 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:25:32.600117Z" 2023-05-24T21:25:33.060 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:33.061 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:33.061 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000001211", 2023-05-24T21:25:33.061 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:25:32.593101Z" 2023-05-24T21:25:33.061 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:33.061 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:33.061 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000005436", 2023-05-24T21:25:33.061 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:25:32.594676Z" 2023-05-24T21:25:33.062 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:33.062 INFO:teuthology.orchestra.run.smithi071.stdout:] 2023-05-24T21:25:33.063 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000003890', 1: '00000000000000000000:00000000000000002055', 2: '00000000000000000000:00000000000000001211', 3: '00000000000000000000:00000000000000005436'} 2023-05-24T21:25:33.063 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-05-24T21:25:33.064 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:25:33.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:25:33.064 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:25:33.116 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:25:33.117 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:25:33.183 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:25:33.184 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:25:33.184 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:25:33.184 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:25:33.184 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:25:33.185 INFO:teuthology.orchestra.run.smithi071.stdout: "instance_id": 13190874283549403954 2023-05-24T21:25:33.185 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:33.185 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:25:33.185 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:33.185 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:25:33.185 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:33.185 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:25:33.186 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000003048", 2023-05-24T21:25:33.186 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:33.186 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:33.186 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:33.186 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:18:15.513877Z" 2023-05-24T21:25:33.186 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:33.187 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:33.187 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:33.187 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:25:33.187 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:33.187 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:25:33.187 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-05-24T21:25:33.187 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:33.188 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:33.188 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:33.188 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:14:51.314836Z" 2023-05-24T21:25:33.188 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:33.188 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:33.188 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:33.188 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:25:33.189 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:33.189 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:25:33.189 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-05-24T21:25:33.189 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:33.189 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:33.189 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:33.189 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:31.177575Z" 2023-05-24T21:25:33.190 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:33.190 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:33.190 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:33.190 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:25:33.190 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:33.190 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:25:33.190 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000004593", 2023-05-24T21:25:33.191 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:33.191 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:33.191 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:33.191 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:14:52.334363Z" 2023-05-24T21:25:33.191 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:33.191 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:33.191 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:25:33.192 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:33.192 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:25:33.192 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 0, 2023-05-24T21:25:33.192 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 0 2023-05-24T21:25:33.192 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:33.192 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:25:33Z" 2023-05-24T21:25:33.192 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:25:33.193 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13190874283549403954 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003048", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:18:15.513877Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001210", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:14:51.314836Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:31.177575Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004593", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:14:52.334363Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-05-24T21:25:33Z" } 2023-05-24T21:25:33.194 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003048', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:18:15.513877Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:14:51.314836Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:31.177575Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004593', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:14:52.334363Z'}}] 2023-05-24T21:25:33.195 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003890', 1: '00000000000000000000:00000000000000002055', 2: '00000000000000000000:00000000000000001211', 3: '00000000000000000000:00000000000000005436'} 2023-05-24T21:25:33.195 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003048', 1: '00000000000000000000:00000000000000001210', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004593'} 2023-05-24T21:25:33.195 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000003890 target=00000000000000000000:00000000000000003048, shard=1 master=00000000000000000000:00000000000000002055 target=00000000000000000000:00000000000000001210, shard=2 master=00000000000000000000:00000000000000001211 target=00000000000000000000:00000000000000000651, shard=3 master=00000000000000000000:00000000000000005436 target=00000000000000000000:00000000000000004593 2023-05-24T21:25:36.715 INFO:tasks.rgw.c2.client.1.smithi157.stdout:2023-05-24T21:25:36.708+0000 7f37d6f3f700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-05-24T21:25:38.197 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:25:38.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:25:38.198 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:25:38.258 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:25:38.258 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:25:38.321 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:25:38.321 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:25:38.321 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:25:38.322 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:25:38.322 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:25:38.322 INFO:teuthology.orchestra.run.smithi071.stdout: "instance_id": 13190874283549403954 2023-05-24T21:25:38.322 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:38.322 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:25:38.322 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:38.322 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:25:38.323 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:38.323 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:25:38.323 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000003048", 2023-05-24T21:25:38.323 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:38.323 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:38.323 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:38.324 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:18:15.513877Z" 2023-05-24T21:25:38.324 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:38.324 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:38.324 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:38.324 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:25:38.324 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:38.324 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:25:38.325 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-05-24T21:25:38.325 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:38.325 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:38.325 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:38.325 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:14:51.314836Z" 2023-05-24T21:25:38.325 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:38.325 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:38.326 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:38.326 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:25:38.326 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:38.326 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:25:38.326 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-05-24T21:25:38.326 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:38.326 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:38.327 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:38.327 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:31.177575Z" 2023-05-24T21:25:38.327 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:38.327 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:38.327 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:38.327 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:25:38.327 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:38.328 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:25:38.328 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000004593", 2023-05-24T21:25:38.328 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:38.328 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:38.328 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:38.328 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:14:52.334363Z" 2023-05-24T21:25:38.329 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:38.329 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:38.329 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:25:38.329 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:38.329 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:25:38.329 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 0, 2023-05-24T21:25:38.329 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 0 2023-05-24T21:25:38.330 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:38.330 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:25:38Z" 2023-05-24T21:25:38.330 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:25:38.331 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13190874283549403954 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003048", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:18:15.513877Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001210", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:14:51.314836Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:31.177575Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004593", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:14:52.334363Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-05-24T21:25:38Z" } 2023-05-24T21:25:38.331 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003048', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:18:15.513877Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:14:51.314836Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:31.177575Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004593', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:14:52.334363Z'}}] 2023-05-24T21:25:38.332 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003890', 1: '00000000000000000000:00000000000000002055', 2: '00000000000000000000:00000000000000001211', 3: '00000000000000000000:00000000000000005436'} 2023-05-24T21:25:38.332 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003048', 1: '00000000000000000000:00000000000000001210', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004593'} 2023-05-24T21:25:38.332 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000003890 target=00000000000000000000:00000000000000003048, shard=1 master=00000000000000000000:00000000000000002055 target=00000000000000000000:00000000000000001210, shard=2 master=00000000000000000000:00000000000000001211 target=00000000000000000000:00000000000000000651, shard=3 master=00000000000000000000:00000000000000005436 target=00000000000000000000:00000000000000004593 2023-05-24T21:25:43.334 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:25:43.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:25:43.335 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:25:43.394 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:25:43.394 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:25:43.476 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:25:43.476 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:25:43.476 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:25:43.476 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:25:43.477 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:25:43.477 INFO:teuthology.orchestra.run.smithi071.stdout: "instance_id": 13190874283549403954 2023-05-24T21:25:43.477 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:43.477 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:25:43.477 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:43.477 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:25:43.477 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:43.478 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:25:43.478 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000003048", 2023-05-24T21:25:43.478 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:43.478 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:43.478 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:43.478 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:18:15.513877Z" 2023-05-24T21:25:43.479 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:43.479 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:43.479 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:43.479 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:25:43.479 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:43.479 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:25:43.479 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-05-24T21:25:43.480 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:43.480 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:43.480 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:43.480 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:14:51.314836Z" 2023-05-24T21:25:43.480 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:43.480 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:43.480 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:43.481 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:25:43.481 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:43.481 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:25:43.481 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-05-24T21:25:43.481 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:43.481 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:43.482 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:43.482 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:31.177575Z" 2023-05-24T21:25:43.482 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:43.482 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:43.482 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:43.482 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:25:43.482 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:43.483 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:25:43.483 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000004593", 2023-05-24T21:25:43.483 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:43.483 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:43.483 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:43.483 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:14:52.334363Z" 2023-05-24T21:25:43.483 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:43.484 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:43.484 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:25:43.484 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:43.484 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:25:43.484 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 0, 2023-05-24T21:25:43.484 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 0 2023-05-24T21:25:43.484 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:43.485 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:25:43Z" 2023-05-24T21:25:43.485 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:25:43.485 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13190874283549403954 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003048", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:18:15.513877Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001210", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:14:51.314836Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:31.177575Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004593", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:14:52.334363Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-05-24T21:25:43Z" } 2023-05-24T21:25:43.486 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003048', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:18:15.513877Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:14:51.314836Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:31.177575Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004593', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:14:52.334363Z'}}] 2023-05-24T21:25:43.486 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003890', 1: '00000000000000000000:00000000000000002055', 2: '00000000000000000000:00000000000000001211', 3: '00000000000000000000:00000000000000005436'} 2023-05-24T21:25:43.487 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003048', 1: '00000000000000000000:00000000000000001210', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004593'} 2023-05-24T21:25:43.487 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000003890 target=00000000000000000000:00000000000000003048, shard=1 master=00000000000000000000:00000000000000002055 target=00000000000000000000:00000000000000001210, shard=2 master=00000000000000000000:00000000000000001211 target=00000000000000000000:00000000000000000651, shard=3 master=00000000000000000000:00000000000000005436 target=00000000000000000000:00000000000000004593 2023-05-24T21:25:46.429 INFO:tasks.rgw.c1.client.1.smithi071.stdout:2023-05-24T21:25:46.422+0000 7f1a4f21b700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-05-24T21:25:48.488 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:25:48.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:25:48.489 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:25:48.548 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:25:48.548 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:25:48.599 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:25:48.600 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:25:48.600 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:25:48.600 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:25:48.600 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:25:48.600 INFO:teuthology.orchestra.run.smithi071.stdout: "instance_id": 13190874283549403954 2023-05-24T21:25:48.601 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:48.601 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:25:48.601 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:48.601 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:25:48.601 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:48.602 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:25:48.602 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000003890", 2023-05-24T21:25:48.602 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:48.602 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:48.602 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:48.602 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:25:32.597538Z" 2023-05-24T21:25:48.602 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:48.603 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:48.603 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:48.603 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:25:48.603 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:48.603 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:25:48.603 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000002055", 2023-05-24T21:25:48.604 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:48.604 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:48.604 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:48.604 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:25:32.600117Z" 2023-05-24T21:25:48.604 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:48.604 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:48.605 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:48.605 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:25:48.605 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:48.605 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:25:48.605 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000001211", 2023-05-24T21:25:48.605 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:48.605 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:48.606 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:48.606 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:25:32.593101Z" 2023-05-24T21:25:48.606 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:48.606 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:48.606 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:48.606 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:25:48.607 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:48.607 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:25:48.607 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000005436", 2023-05-24T21:25:48.607 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:48.607 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:48.607 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:48.608 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:25:32.594676Z" 2023-05-24T21:25:48.608 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:48.608 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:48.608 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:25:48.608 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:48.608 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:25:48.608 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 0, 2023-05-24T21:25:48.609 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 0 2023-05-24T21:25:48.609 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:48.609 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:25:48Z" 2023-05-24T21:25:48.609 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:25:48.610 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13190874283549403954 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003890", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:25:32.597538Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002055", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:25:32.600117Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001211", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:25:32.593101Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005436", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:25:32.594676Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-05-24T21:25:48Z" } 2023-05-24T21:25:48.610 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003890', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:25:32.597538Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002055', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:25:32.600117Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001211', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:25:32.593101Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005436', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:25:32.594676Z'}}] 2023-05-24T21:25:48.611 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003890', 1: '00000000000000000000:00000000000000002055', 2: '00000000000000000000:00000000000000001211', 3: '00000000000000000000:00000000000000005436'} 2023-05-24T21:25:48.611 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003890', 1: '00000000000000000000:00000000000000002055', 2: '00000000000000000000:00000000000000001211', 3: '00000000000000000000:00000000000000005436'} 2023-05-24T21:25:48.612 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-05-24T21:25:48.612 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-05-24T21:25:48.612 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:25:48.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:25:48.613 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:25:48.662 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:25:48.662 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:25:48.715 INFO:teuthology.orchestra.run.smithi071.stdout:[ 2023-05-24T21:25:48.716 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:48.716 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000002486", 2023-05-24T21:25:48.716 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:18:15.945505Z" 2023-05-24T21:25:48.716 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:48.717 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:48.717 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-05-24T21:25:48.717 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:14:51.309908Z" 2023-05-24T21:25:48.717 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:48.717 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:48.717 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-05-24T21:25:48.717 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:03:11.225401Z" 2023-05-24T21:25:48.718 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:48.718 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:48.718 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000004734", 2023-05-24T21:25:48.718 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:18:15.892968Z" 2023-05-24T21:25:48.718 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:48.718 INFO:teuthology.orchestra.run.smithi071.stdout:] 2023-05-24T21:25:48.719 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000001210', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004734'} 2023-05-24T21:25:48.720 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-05-24T21:25:48.720 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:25:48.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:25:48.721 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:25:48.772 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:25:48.772 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:25:48.836 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:25:48.836 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:25:48.836 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:25:48.836 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:25:48.836 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:25:48.836 INFO:teuthology.orchestra.run.smithi071.stdout: "instance_id": 6585121019900467362 2023-05-24T21:25:48.837 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:48.837 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:25:48.837 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:48.837 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:25:48.837 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:48.837 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:25:48.838 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000002486", 2023-05-24T21:25:48.838 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:48.838 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:48.838 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:48.838 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:18:15.945505Z" 2023-05-24T21:25:48.838 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:48.838 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:48.839 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:48.839 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:25:48.839 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:48.839 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:25:48.839 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-05-24T21:25:48.839 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:48.839 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:48.840 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:48.840 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:14:51.309908Z" 2023-05-24T21:25:48.840 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:48.840 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:48.840 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:48.840 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:25:48.840 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:48.841 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:25:48.841 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-05-24T21:25:48.841 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:48.841 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:48.841 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:48.841 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:03:11.225401Z" 2023-05-24T21:25:48.841 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:48.842 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:48.842 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:48.842 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:25:48.842 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:48.842 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:25:48.842 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000004734", 2023-05-24T21:25:48.842 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:48.843 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:48.843 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:48.843 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:18:15.892968Z" 2023-05-24T21:25:48.843 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:48.843 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:48.843 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:25:48.843 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:48.844 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:25:48.844 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 0, 2023-05-24T21:25:48.844 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 0 2023-05-24T21:25:48.844 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:48.844 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:25:48Z" 2023-05-24T21:25:48.844 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:25:48.845 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6585121019900467362 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002486", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:18:15.945505Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001210", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:14:51.309908Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:03:11.225401Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004734", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:18:15.892968Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-05-24T21:25:48Z" } 2023-05-24T21:25:48.846 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002486', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:18:15.945505Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:14:51.309908Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:03:11.225401Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004734', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:18:15.892968Z'}}] 2023-05-24T21:25:48.846 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000001210', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004734'} 2023-05-24T21:25:48.846 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000001210', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004734'} 2023-05-24T21:25:48.847 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-05-24T21:25:48.847 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable ... ok 2023-05-24T21:25:48.849 INFO:rgw_multi.tests:create bucket zone=a1 name=qskrps-46 2023-05-24T21:25:48.849 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:25:48.850 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:25:48.850 DEBUG:boto:Method: PUT 2023-05-24T21:25:48.851 DEBUG:boto:Path: /qskrps-46/ 2023-05-24T21:25:48.851 DEBUG:boto:Data: 2023-05-24T21:25:48.851 DEBUG:boto:Headers: {} 2023-05-24T21:25:48.852 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:25:48.852 DEBUG:boto:Port: 8000 2023-05-24T21:25:48.852 DEBUG:boto:Params: {} 2023-05-24T21:25:48.853 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T21:25:48.854 DEBUG:boto:Token: None 2023-05-24T21:25:48.854 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:25:48 GMT /qskrps-46/ 2023-05-24T21:25:48.854 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:AedJyTXE2vb4RJWyaNVBzxQOGgA= 2023-05-24T21:25:48.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:25:48 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:AedJyTXE2vb4RJWyaNVBzxQOGgA=', 'Content-Length': '0'} 2023-05-24T21:25:48.872 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b9eeb9f7e82e7598-00646e80dc-1627-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:25:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:25:48.872 INFO:rgw_multi.tests:create bucket zone=a2 name=qskrps-47 2023-05-24T21:25:48.873 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:25:48.873 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:25:48.873 DEBUG:boto:Method: PUT 2023-05-24T21:25:48.874 DEBUG:boto:Path: /qskrps-47/ 2023-05-24T21:25:48.874 DEBUG:boto:Data: 2023-05-24T21:25:48.874 DEBUG:boto:Headers: {} 2023-05-24T21:25:48.875 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:25:48.875 DEBUG:boto:Port: 8001 2023-05-24T21:25:48.875 DEBUG:boto:Params: {} 2023-05-24T21:25:48.876 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-24T21:25:48.876 DEBUG:boto:Token: None 2023-05-24T21:25:48.877 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:25:48 GMT /qskrps-47/ 2023-05-24T21:25:48.877 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:6oJm6ktO5jreKI3PCjN8oP9ZLVQ= 2023-05-24T21:25:48.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:25:48 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:6oJm6ktO5jreKI3PCjN8oP9ZLVQ=', 'Content-Length': '0'} 2023-05-24T21:25:48.915 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d02d2cc74afc6e31-00646e80dc-1625-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:25:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:25:48.915 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:25:48.916 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:25:48.916 DEBUG:boto:Method: HEAD 2023-05-24T21:25:48.916 DEBUG:boto:Path: /qskrps-46/ 2023-05-24T21:25:48.917 DEBUG:boto:Data: 2023-05-24T21:25:48.917 DEBUG:boto:Headers: {} 2023-05-24T21:25:48.917 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:25:48.918 DEBUG:boto:Port: 8000 2023-05-24T21:25:48.918 DEBUG:boto:Params: {} 2023-05-24T21:25:48.918 DEBUG:boto:Token: None 2023-05-24T21:25:48.919 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:25:48 GMT /qskrps-46/ 2023-05-24T21:25:48.919 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:rkIuPw9DxmC1cEwmErv/vy7SeoA= 2023-05-24T21:25:48.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:25:48 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:rkIuPw9DxmC1cEwmErv/vy7SeoA=', 'Content-Length': '0'} 2023-05-24T21:25:48.922 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001240ab935e7e6782-00646e80dc-1627-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:25:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:25:48.922 DEBUG:boto:path=/qskrps-46/obj1 2023-05-24T21:25:48.923 DEBUG:boto:auth_path=/qskrps-46/obj1 2023-05-24T21:25:48.923 DEBUG:boto:Method: PUT 2023-05-24T21:25:48.923 DEBUG:boto:Path: /qskrps-46/obj1 2023-05-24T21:25:48.924 DEBUG:boto:Data: 2023-05-24T21:25:48.924 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-24T21:25:48.924 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:25:48.925 DEBUG:boto:Port: 8000 2023-05-24T21:25:48.925 DEBUG:boto:Params: {} 2023-05-24T21:25:48.925 DEBUG:boto:Token: None 2023-05-24T21:25:48.926 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 24 May 2023 21:25:48 GMT /qskrps-46/obj1 2023-05-24T21:25:48.926 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:T8CiQaDc+LPZPYpIPf9R/xrt0Es= 2023-05-24T21:25:48.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:25:48 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:T8CiQaDc+LPZPYpIPf9R/xrt0Es='} 2023-05-24T21:25:48.933 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000051daaf3d90e47ad5-00646e80dc-1627-a1'), ('Date', 'Wed, 24 May 2023 21:25:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:25:48.933 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:25:48.934 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:25:48.934 DEBUG:boto:Method: HEAD 2023-05-24T21:25:48.934 DEBUG:boto:Path: /qskrps-46/ 2023-05-24T21:25:48.935 DEBUG:boto:Data: 2023-05-24T21:25:48.935 DEBUG:boto:Headers: {} 2023-05-24T21:25:48.935 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:25:48.936 DEBUG:boto:Port: 8000 2023-05-24T21:25:48.936 DEBUG:boto:Params: {} 2023-05-24T21:25:48.936 DEBUG:boto:Token: None 2023-05-24T21:25:48.937 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:25:48 GMT /qskrps-46/ 2023-05-24T21:25:48.937 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:rkIuPw9DxmC1cEwmErv/vy7SeoA= 2023-05-24T21:25:48.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:25:48 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:rkIuPw9DxmC1cEwmErv/vy7SeoA=', 'Content-Length': '0'} 2023-05-24T21:25:48.940 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000549275636469369b-00646e80dc-1627-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:25:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:25:48.940 DEBUG:boto:path=/qskrps-46/obj2 2023-05-24T21:25:48.941 DEBUG:boto:auth_path=/qskrps-46/obj2 2023-05-24T21:25:48.941 DEBUG:boto:Method: PUT 2023-05-24T21:25:48.942 DEBUG:boto:Path: /qskrps-46/obj2 2023-05-24T21:25:48.942 DEBUG:boto:Data: 2023-05-24T21:25:48.942 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-24T21:25:48.943 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:25:48.943 DEBUG:boto:Port: 8000 2023-05-24T21:25:48.943 DEBUG:boto:Params: {} 2023-05-24T21:25:48.944 DEBUG:boto:Token: None 2023-05-24T21:25:48.944 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 24 May 2023 21:25:48 GMT /qskrps-46/obj2 2023-05-24T21:25:48.944 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:yIdSz/xV+eISngLVaqBmq0NGW90= 2023-05-24T21:25:48.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:25:48 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:yIdSz/xV+eISngLVaqBmq0NGW90='} 2023-05-24T21:25:48.960 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008d9c059e45fbd677-00646e80dc-1627-a1'), ('Date', 'Wed, 24 May 2023 21:25:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:25:48.960 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:25:48.961 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:25:48.961 DEBUG:boto:Method: HEAD 2023-05-24T21:25:48.961 DEBUG:boto:Path: /qskrps-46/ 2023-05-24T21:25:48.962 DEBUG:boto:Data: 2023-05-24T21:25:48.962 DEBUG:boto:Headers: {} 2023-05-24T21:25:48.962 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:25:48.963 DEBUG:boto:Port: 8000 2023-05-24T21:25:48.963 DEBUG:boto:Params: {} 2023-05-24T21:25:48.963 DEBUG:boto:Token: None 2023-05-24T21:25:48.964 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:25:48 GMT /qskrps-46/ 2023-05-24T21:25:48.964 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:rkIuPw9DxmC1cEwmErv/vy7SeoA= 2023-05-24T21:25:48.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:25:48 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:rkIuPw9DxmC1cEwmErv/vy7SeoA=', 'Content-Length': '0'} 2023-05-24T21:25:48.966 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000457f45fe379c834d-00646e80dc-1627-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:25:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:25:48.967 DEBUG:boto:path=/qskrps-46/obj3 2023-05-24T21:25:48.967 DEBUG:boto:auth_path=/qskrps-46/obj3 2023-05-24T21:25:48.968 DEBUG:boto:Method: PUT 2023-05-24T21:25:48.968 DEBUG:boto:Path: /qskrps-46/obj3 2023-05-24T21:25:48.968 DEBUG:boto:Data: 2023-05-24T21:25:48.969 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-24T21:25:48.969 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:25:48.969 DEBUG:boto:Port: 8000 2023-05-24T21:25:48.970 DEBUG:boto:Params: {} 2023-05-24T21:25:48.970 DEBUG:boto:Token: None 2023-05-24T21:25:48.970 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 24 May 2023 21:25:48 GMT /qskrps-46/obj3 2023-05-24T21:25:48.971 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:C6SoxXBMYJ46d+WxlIRttqWurIU= 2023-05-24T21:25:48.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:25:48 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:C6SoxXBMYJ46d+WxlIRttqWurIU='} 2023-05-24T21:25:48.987 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007be883024c41ed90-00646e80dc-1627-a1'), ('Date', 'Wed, 24 May 2023 21:25:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:25:48.988 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:25:48.988 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:25:48.989 DEBUG:boto:Method: HEAD 2023-05-24T21:25:48.989 DEBUG:boto:Path: /qskrps-46/ 2023-05-24T21:25:48.989 DEBUG:boto:Data: 2023-05-24T21:25:48.989 DEBUG:boto:Headers: {} 2023-05-24T21:25:48.990 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:25:48.990 DEBUG:boto:Port: 8000 2023-05-24T21:25:48.990 DEBUG:boto:Params: {} 2023-05-24T21:25:48.991 DEBUG:boto:Token: None 2023-05-24T21:25:48.991 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:25:48 GMT /qskrps-46/ 2023-05-24T21:25:48.992 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:rkIuPw9DxmC1cEwmErv/vy7SeoA= 2023-05-24T21:25:48.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:25:48 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:rkIuPw9DxmC1cEwmErv/vy7SeoA=', 'Content-Length': '0'} 2023-05-24T21:25:48.994 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004cf4998ee949e5de-00646e80dc-1627-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:25:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:25:48.994 DEBUG:boto:path=/qskrps-46/obj4 2023-05-24T21:25:48.995 DEBUG:boto:auth_path=/qskrps-46/obj4 2023-05-24T21:25:48.995 DEBUG:boto:Method: PUT 2023-05-24T21:25:48.996 DEBUG:boto:Path: /qskrps-46/obj4 2023-05-24T21:25:48.996 DEBUG:boto:Data: 2023-05-24T21:25:48.996 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-24T21:25:48.997 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:25:48.997 DEBUG:boto:Port: 8000 2023-05-24T21:25:48.997 DEBUG:boto:Params: {} 2023-05-24T21:25:48.998 DEBUG:boto:Token: None 2023-05-24T21:25:48.998 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 24 May 2023 21:25:48 GMT /qskrps-46/obj4 2023-05-24T21:25:48.998 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:0RpkIDlTXmyoSDM6uAcm3rygIqI= 2023-05-24T21:25:48.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:25:48 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:0RpkIDlTXmyoSDM6uAcm3rygIqI='} 2023-05-24T21:25:49.040 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000064022aef849aa68e-00646e80dc-1627-a1'), ('Date', 'Wed, 24 May 2023 21:25:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:25:49.041 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:25:49.041 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:25:49.041 DEBUG:boto:Method: HEAD 2023-05-24T21:25:49.042 DEBUG:boto:Path: /qskrps-47/ 2023-05-24T21:25:49.042 DEBUG:boto:Data: 2023-05-24T21:25:49.042 DEBUG:boto:Headers: {} 2023-05-24T21:25:49.043 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:25:49.043 DEBUG:boto:Port: 8001 2023-05-24T21:25:49.043 DEBUG:boto:Params: {} 2023-05-24T21:25:49.044 DEBUG:boto:Token: None 2023-05-24T21:25:49.044 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:25:49 GMT /qskrps-47/ 2023-05-24T21:25:49.044 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:3rJjwjU4AfKi1XzBtS6P8h0bhn0= 2023-05-24T21:25:49.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:25:49 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:3rJjwjU4AfKi1XzBtS6P8h0bhn0=', 'Content-Length': '0'} 2023-05-24T21:25:49.047 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009d1c0fe8c643410c-00646e80dd-1625-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:25:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:25:49.048 DEBUG:boto:path=/qskrps-47/obj1 2023-05-24T21:25:49.048 DEBUG:boto:auth_path=/qskrps-47/obj1 2023-05-24T21:25:49.048 DEBUG:boto:Method: PUT 2023-05-24T21:25:49.049 DEBUG:boto:Path: /qskrps-47/obj1 2023-05-24T21:25:49.049 DEBUG:boto:Data: 2023-05-24T21:25:49.049 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-24T21:25:49.050 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:25:49.050 DEBUG:boto:Port: 8001 2023-05-24T21:25:49.050 DEBUG:boto:Params: {} 2023-05-24T21:25:49.051 DEBUG:boto:Token: None 2023-05-24T21:25:49.051 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 24 May 2023 21:25:49 GMT /qskrps-47/obj1 2023-05-24T21:25:49.051 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:b59FpiIf9hczRtVbxLJZLgjFROU= 2023-05-24T21:25:49.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:25:49 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:b59FpiIf9hczRtVbxLJZLgjFROU='} 2023-05-24T21:25:49.059 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007af1a25aafef45a8-00646e80dd-1625-a2'), ('Date', 'Wed, 24 May 2023 21:25:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:25:49.059 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:25:49.060 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:25:49.060 DEBUG:boto:Method: HEAD 2023-05-24T21:25:49.060 DEBUG:boto:Path: /qskrps-47/ 2023-05-24T21:25:49.061 DEBUG:boto:Data: 2023-05-24T21:25:49.061 DEBUG:boto:Headers: {} 2023-05-24T21:25:49.061 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:25:49.062 DEBUG:boto:Port: 8001 2023-05-24T21:25:49.062 DEBUG:boto:Params: {} 2023-05-24T21:25:49.062 DEBUG:boto:Token: None 2023-05-24T21:25:49.063 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:25:49 GMT /qskrps-47/ 2023-05-24T21:25:49.063 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:3rJjwjU4AfKi1XzBtS6P8h0bhn0= 2023-05-24T21:25:49.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:25:49 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:3rJjwjU4AfKi1XzBtS6P8h0bhn0=', 'Content-Length': '0'} 2023-05-24T21:25:49.066 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003ff24b8cf8b3c2a5-00646e80dd-1625-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:25:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:25:49.066 DEBUG:boto:path=/qskrps-47/obj2 2023-05-24T21:25:49.067 DEBUG:boto:auth_path=/qskrps-47/obj2 2023-05-24T21:25:49.067 DEBUG:boto:Method: PUT 2023-05-24T21:25:49.067 DEBUG:boto:Path: /qskrps-47/obj2 2023-05-24T21:25:49.068 DEBUG:boto:Data: 2023-05-24T21:25:49.068 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-24T21:25:49.068 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:25:49.069 DEBUG:boto:Port: 8001 2023-05-24T21:25:49.069 DEBUG:boto:Params: {} 2023-05-24T21:25:49.069 DEBUG:boto:Token: None 2023-05-24T21:25:49.070 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 24 May 2023 21:25:49 GMT /qskrps-47/obj2 2023-05-24T21:25:49.070 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:RPhK7tVFaQengxMDEXOMS+QVRUw= 2023-05-24T21:25:49.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:25:49 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:RPhK7tVFaQengxMDEXOMS+QVRUw='} 2023-05-24T21:25:49.076 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e7af70a63f5f41c4-00646e80dd-1625-a2'), ('Date', 'Wed, 24 May 2023 21:25:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:25:49.076 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:25:49.077 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:25:49.077 DEBUG:boto:Method: HEAD 2023-05-24T21:25:49.077 DEBUG:boto:Path: /qskrps-47/ 2023-05-24T21:25:49.078 DEBUG:boto:Data: 2023-05-24T21:25:49.078 DEBUG:boto:Headers: {} 2023-05-24T21:25:49.078 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:25:49.079 DEBUG:boto:Port: 8001 2023-05-24T21:25:49.079 DEBUG:boto:Params: {} 2023-05-24T21:25:49.079 DEBUG:boto:Token: None 2023-05-24T21:25:49.080 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:25:49 GMT /qskrps-47/ 2023-05-24T21:25:49.080 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:3rJjwjU4AfKi1XzBtS6P8h0bhn0= 2023-05-24T21:25:49.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:25:49 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:3rJjwjU4AfKi1XzBtS6P8h0bhn0=', 'Content-Length': '0'} 2023-05-24T21:25:49.082 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000014ad32fe1b290e47-00646e80dd-1625-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:25:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:25:49.083 DEBUG:boto:path=/qskrps-47/obj3 2023-05-24T21:25:49.083 DEBUG:boto:auth_path=/qskrps-47/obj3 2023-05-24T21:25:49.084 DEBUG:boto:Method: PUT 2023-05-24T21:25:49.084 DEBUG:boto:Path: /qskrps-47/obj3 2023-05-24T21:25:49.084 DEBUG:boto:Data: 2023-05-24T21:25:49.085 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-24T21:25:49.085 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:25:49.085 DEBUG:boto:Port: 8001 2023-05-24T21:25:49.086 DEBUG:boto:Params: {} 2023-05-24T21:25:49.086 DEBUG:boto:Token: None 2023-05-24T21:25:49.086 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 24 May 2023 21:25:49 GMT /qskrps-47/obj3 2023-05-24T21:25:49.087 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:/FWgw1PP3NoNj/1mTRRHcwLWrFg= 2023-05-24T21:25:49.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:25:49 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:/FWgw1PP3NoNj/1mTRRHcwLWrFg='} 2023-05-24T21:25:49.093 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008d4f85230a1fc19b-00646e80dd-1625-a2'), ('Date', 'Wed, 24 May 2023 21:25:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:25:49.094 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:25:49.094 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:25:49.094 DEBUG:boto:Method: HEAD 2023-05-24T21:25:49.095 DEBUG:boto:Path: /qskrps-47/ 2023-05-24T21:25:49.095 DEBUG:boto:Data: 2023-05-24T21:25:49.095 DEBUG:boto:Headers: {} 2023-05-24T21:25:49.096 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:25:49.096 DEBUG:boto:Port: 8001 2023-05-24T21:25:49.096 DEBUG:boto:Params: {} 2023-05-24T21:25:49.097 DEBUG:boto:Token: None 2023-05-24T21:25:49.097 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:25:49 GMT /qskrps-47/ 2023-05-24T21:25:49.097 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:3rJjwjU4AfKi1XzBtS6P8h0bhn0= 2023-05-24T21:25:49.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:25:49 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:3rJjwjU4AfKi1XzBtS6P8h0bhn0=', 'Content-Length': '0'} 2023-05-24T21:25:49.100 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000046aed2c2e463c2d1-00646e80dd-1625-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:25:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:25:49.100 DEBUG:boto:path=/qskrps-47/obj4 2023-05-24T21:25:49.100 DEBUG:boto:auth_path=/qskrps-47/obj4 2023-05-24T21:25:49.101 DEBUG:boto:Method: PUT 2023-05-24T21:25:49.101 DEBUG:boto:Path: /qskrps-47/obj4 2023-05-24T21:25:49.101 DEBUG:boto:Data: 2023-05-24T21:25:49.102 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-24T21:25:49.102 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:25:49.102 DEBUG:boto:Port: 8001 2023-05-24T21:25:49.103 DEBUG:boto:Params: {} 2023-05-24T21:25:49.103 DEBUG:boto:Token: None 2023-05-24T21:25:49.103 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 24 May 2023 21:25:49 GMT /qskrps-47/obj4 2023-05-24T21:25:49.104 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:03/0+oJRmtxkGHl3mTpyk4klVEM= 2023-05-24T21:25:49.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:25:49 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:03/0+oJRmtxkGHl3mTpyk4klVEM='} 2023-05-24T21:25:49.148 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c234f24ebb4f8de2-00646e80dd-1625-a2'), ('Date', 'Wed, 24 May 2023 21:25:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:25:49.149 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:25:49.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:25:49.150 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:25:49.198 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:25:49.198 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:25:49.233 INFO:teuthology.orchestra.run.smithi071.stderr:No --period given, using current period=5c04936b-70ea-4415-b4eb-3aad3dc612e5 2023-05-24T21:25:49.250 INFO:teuthology.orchestra.run.smithi071.stdout:[ 2023-05-24T21:25:49.250 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:49.251 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963548.896085_485.1", 2023-05-24T21:25:49.251 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:25:48.896085Z" 2023-05-24T21:25:49.251 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:49.251 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:49.251 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:25:49.251 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:06:32.333844Z" 2023-05-24T21:25:49.252 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:49.252 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:49.252 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:25:49.252 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:25:49.252 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:49.252 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:49.253 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:25:49.253 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:25:49.253 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:49.253 INFO:teuthology.orchestra.run.smithi071.stdout:] 2023-05-24T21:25:49.254 DEBUG:rgw_multi.tests:master meta markers={0: '1_1684963548.896085_485.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:25:49.254 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-24T21:25:49.254 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:25:49.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:25:49.255 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:25:49.305 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:25:49.305 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:25:49.358 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:25:49.351+0000 7fb582d84780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:25:49.359 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:25:49.352+0000 7fb582d84780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:25:49.380 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:25:49.380 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:25:49.380 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:25:49.380 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:25:49.380 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:25:49.380 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:25:49.381 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:25:49.381 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:49.381 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:25:49.381 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:49.381 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:25:49.381 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:49.382 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:25:49.382 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963229.148253_462.1", 2023-05-24T21:25:49.382 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:49.382 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:25:49.382 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:49.382 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:20:29.148253Z", 2023-05-24T21:25:49.382 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:25:49.383 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:49.383 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:49.383 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:49.383 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:25:49.383 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:49.383 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:25:49.383 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:25:49.384 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:49.384 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:49.384 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:49.384 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:25:49.384 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:25:49.384 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:49.384 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:49.385 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:49.385 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:25:49.385 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:49.385 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:25:49.385 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:25:49.385 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:49.385 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:49.386 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:49.386 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:25:49.386 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:25:49.386 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:49.386 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:49.386 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:49.386 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:25:49.387 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:49.387 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:25:49.387 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:25:49.387 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:49.387 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:49.387 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:49.387 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:25:49.388 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:25:49.388 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:49.388 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:49.388 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:25:49.388 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:49.388 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:25:49.389 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:25:49.389 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:25:49.389 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:49.389 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:25:49Z" 2023-05-24T21:25:49.389 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:25:49.390 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684963229.148253_462.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:20:29.148253Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:25:49Z" } 2023-05-24T21:25:49.390 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963229.148253_462.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:20:29.148253Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:25:49.391 DEBUG:rgw_multi.tests:log_status={0: '1_1684963548.896085_485.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:25:49.391 DEBUG:rgw_multi.tests:sync_status={0: '1_1684963229.148253_462.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:25:49.391 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963548.896085_485.1 target=1_1684963229.148253_462.1 2023-05-24T21:25:54.392 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:25:54.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:25:54.394 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:25:54.450 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:25:54.451 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:25:54.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:25:54.497+0000 7f2e234fe780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:25:54.503 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:25:54.498+0000 7f2e234fe780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:25:54.526 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:25:54.526 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:25:54.526 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:25:54.526 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:25:54.526 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:25:54.526 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:25:54.527 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:25:54.527 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:54.527 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:25:54.527 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:54.527 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:25:54.527 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:54.528 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:25:54.528 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963548.896085_485.1", 2023-05-24T21:25:54.528 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:54.528 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:25:54.528 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:54.528 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:25:48.896085Z", 2023-05-24T21:25:54.528 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:25:54.529 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:54.529 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:54.529 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:54.529 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:25:54.529 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:54.529 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:25:54.529 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:25:54.530 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:54.530 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:54.530 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:54.530 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:25:54.530 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:25:54.530 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:54.530 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:54.531 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:54.531 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:25:54.531 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:54.531 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:25:54.531 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:25:54.531 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:54.531 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:54.532 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:54.532 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:25:54.532 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:25:54.532 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:54.532 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:54.532 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:25:54.533 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:25:54.533 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:25:54.533 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:25:54.533 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:25:54.533 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:25:54.533 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:25:54.533 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:25:54.534 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:25:54.534 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:25:54.534 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:54.534 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:25:54.534 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:25:54.534 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:54.534 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:25:54.535 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:25:54.535 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:25:54.535 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:25:54.535 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:25:54Z" 2023-05-24T21:25:54.535 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:25:54.536 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684963548.896085_485.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:25:48.896085Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:25:54Z" } 2023-05-24T21:25:54.536 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963548.896085_485.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:25:48.896085Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:25:54.537 DEBUG:rgw_multi.tests:log_status={0: '1_1684963548.896085_485.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:25:54.537 DEBUG:rgw_multi.tests:sync_status={0: '1_1684963548.896085_485.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:25:54.537 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-24T21:25:54.538 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=qskrps-46 2023-05-24T21:25:54.538 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-46', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:25:54.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-46', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:25:54.539 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-46 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:25:54.595 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:25:54.596 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:25:54.652 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:25:54.647+0000 7f9947e83780 1 waiting to reach incremental sync.. 2023-05-24T21:25:56.426 INFO:tasks.rgw.c1.client.0.smithi071.stdout:2023-05-24T21:25:56.420+0000 7f4a397b6700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-05-24T21:25:59.653 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:25:59.647+0000 7f9947e83780 1 waiting to reach incremental sync.. 2023-05-24T21:26:04.654 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:26:04.648+0000 7f9947e83780 1 waiting to reach incremental sync.. 2023-05-24T21:26:09.655 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:26:09.650+0000 7f9947e83780 1 bucket sync caught up with source: 2023-05-24T21:26:09.656 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [00000000001.23.6, , 00000000001.101.6, , 00000000001.14.6, , , 00000000001.19.6, , , ] 2023-05-24T21:26:09.656 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [00000000001.23.6, , 00000000001.101.6, , 00000000001.14.6, , , 00000000001.19.6, , , ] 2023-05-24T21:26:09.656 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:26:09.650+0000 7f9947e83780 0 bucket checkpoint complete 2023-05-24T21:26:09.673 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=qskrps-46 2023-05-24T21:26:09.674 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-46', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:26:09.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-46', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:26:09.674 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-46 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:26:09.736 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:26:09.736 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:26:09.777 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:26:09.771+0000 7feaed286780 1 waiting for incremental sync to catch up: 2023-05-24T21:26:09.777 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [00000000001.386.6, , 00000000001.1699.6, , 00000000001.508.6, , , , , , ] 2023-05-24T21:26:09.778 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [00000000001.386.6, , 00000000001.1699.6, , 00000000001.508.6, , , 00000000001.664.6, , , ] 2023-05-24T21:26:14.777 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:26:14.772+0000 7feaed286780 1 waiting for incremental sync to catch up: 2023-05-24T21:26:14.778 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [00000000001.386.6, , 00000000001.1699.6, , 00000000001.508.6, , , , , , ] 2023-05-24T21:26:14.778 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [00000000001.386.6, , 00000000001.1699.6, , 00000000001.508.6, , , 00000000001.664.6, , , ] 2023-05-24T21:26:19.778 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:26:19.773+0000 7feaed286780 1 bucket sync caught up with source: 2023-05-24T21:26:19.779 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [00000000001.386.6, , 00000000001.1699.6, , 00000000001.508.6, , , 00000000001.664.6, , , ] 2023-05-24T21:26:19.779 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [00000000001.386.6, , 00000000001.1699.6, , 00000000001.508.6, , , 00000000001.664.6, , , ] 2023-05-24T21:26:19.779 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:26:19.773+0000 7feaed286780 0 bucket checkpoint complete 2023-05-24T21:26:19.801 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qskrps-46 zones={a2, a1} 2023-05-24T21:26:19.802 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:26:19.802 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:26:19.803 DEBUG:boto:Method: HEAD 2023-05-24T21:26:19.803 DEBUG:boto:Path: /qskrps-46/ 2023-05-24T21:26:19.803 DEBUG:boto:Data: 2023-05-24T21:26:19.804 DEBUG:boto:Headers: {} 2023-05-24T21:26:19.804 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:19.804 DEBUG:boto:Port: 8001 2023-05-24T21:26:19.805 DEBUG:boto:Params: {} 2023-05-24T21:26:19.805 DEBUG:boto:Token: None 2023-05-24T21:26:19.805 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:26:19 GMT /qskrps-46/ 2023-05-24T21:26:19.806 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4GhFWDICaStWzafYRrP8s75SdIg= 2023-05-24T21:26:19.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:19 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4GhFWDICaStWzafYRrP8s75SdIg=', 'Content-Length': '0'} 2023-05-24T21:26:19.808 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b42801190c22e235-00646e80fb-1625-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:26:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:19.809 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:26:19.809 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:26:19.810 DEBUG:boto:Method: HEAD 2023-05-24T21:26:19.810 DEBUG:boto:Path: /qskrps-46/ 2023-05-24T21:26:19.810 DEBUG:boto:Data: 2023-05-24T21:26:19.811 DEBUG:boto:Headers: {} 2023-05-24T21:26:19.811 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:19.811 DEBUG:boto:Port: 8000 2023-05-24T21:26:19.812 DEBUG:boto:Params: {} 2023-05-24T21:26:19.812 DEBUG:boto:Token: None 2023-05-24T21:26:19.813 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:26:19 GMT /qskrps-46/ 2023-05-24T21:26:19.813 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4GhFWDICaStWzafYRrP8s75SdIg= 2023-05-24T21:26:19.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:19 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4GhFWDICaStWzafYRrP8s75SdIg=', 'Content-Length': '0'} 2023-05-24T21:26:19.815 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000388e6d293b0887ce-00646e80fb-1627-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:26:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:19.816 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-24T21:26:19.816 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:26:19.817 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:26:19.817 DEBUG:boto:path=/qskrps-46/?versions&max-keys=999 2023-05-24T21:26:19.818 DEBUG:boto:auth_path=/qskrps-46/?versions&max-keys=999 2023-05-24T21:26:19.818 DEBUG:boto:Method: GET 2023-05-24T21:26:19.818 DEBUG:boto:Path: /qskrps-46/?versions&max-keys=999 2023-05-24T21:26:19.819 DEBUG:boto:Data: 2023-05-24T21:26:19.819 DEBUG:boto:Headers: {} 2023-05-24T21:26:19.819 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:19.820 DEBUG:boto:Port: 8001 2023-05-24T21:26:19.820 DEBUG:boto:Params: {} 2023-05-24T21:26:19.820 DEBUG:boto:Token: None 2023-05-24T21:26:19.821 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:19 GMT /qskrps-46/?versions 2023-05-24T21:26:19.821 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:BDTqV2kEhSm3jsniedKMmhVnmeA= 2023-05-24T21:26:19.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:19 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:BDTqV2kEhSm3jsniedKMmhVnmeA=', 'Content-Length': '0'} 2023-05-24T21:26:19.824 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a128659bc1aea3f1-00646e80fb-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:26:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:19.824 DEBUG:boto:b'qskrps-46999falseobj1nulltrue2023-05-24T21:25:48.924Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-24T21:25:48.942Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-24T21:25:48.968Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-24T21:25:48.995Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:26:19.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-05-24T21:26:19.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-05-24T21:26:19.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-05-24T21:26:19.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-05-24T21:26:19.827 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-24T21:26:19.827 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:26:19.827 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:26:19.828 DEBUG:boto:path=/qskrps-46/?versions&max-keys=999 2023-05-24T21:26:19.828 DEBUG:boto:auth_path=/qskrps-46/?versions&max-keys=999 2023-05-24T21:26:19.828 DEBUG:boto:Method: GET 2023-05-24T21:26:19.829 DEBUG:boto:Path: /qskrps-46/?versions&max-keys=999 2023-05-24T21:26:19.829 DEBUG:boto:Data: 2023-05-24T21:26:19.829 DEBUG:boto:Headers: {} 2023-05-24T21:26:19.830 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:19.830 DEBUG:boto:Port: 8000 2023-05-24T21:26:19.830 DEBUG:boto:Params: {} 2023-05-24T21:26:19.831 DEBUG:boto:Token: None 2023-05-24T21:26:19.831 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:19 GMT /qskrps-46/?versions 2023-05-24T21:26:19.832 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:BDTqV2kEhSm3jsniedKMmhVnmeA= 2023-05-24T21:26:19.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:19 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:BDTqV2kEhSm3jsniedKMmhVnmeA=', 'Content-Length': '0'} 2023-05-24T21:26:19.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000390136ee19766890-00646e80fb-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:26:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:19.835 DEBUG:boto:b'qskrps-46999falseobj1nulltrue2023-05-24T21:25:48.924Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-24T21:25:48.942Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-24T21:25:48.968Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-24T21:25:48.995Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:26:19.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-05-24T21:26:19.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-05-24T21:26:19.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-05-24T21:26:19.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-05-24T21:26:19.837 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:26:19.837 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:26:19.837 DEBUG:boto:path=/qskrps-46/?versions&max-keys=999 2023-05-24T21:26:19.838 DEBUG:boto:auth_path=/qskrps-46/?versions&max-keys=999 2023-05-24T21:26:19.838 DEBUG:boto:Method: GET 2023-05-24T21:26:19.838 DEBUG:boto:Path: /qskrps-46/?versions&max-keys=999 2023-05-24T21:26:19.839 DEBUG:boto:Data: 2023-05-24T21:26:19.839 DEBUG:boto:Headers: {} 2023-05-24T21:26:19.839 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:19.840 DEBUG:boto:Port: 8001 2023-05-24T21:26:19.840 DEBUG:boto:Params: {} 2023-05-24T21:26:19.841 DEBUG:boto:Token: None 2023-05-24T21:26:19.841 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:19 GMT /qskrps-46/?versions 2023-05-24T21:26:19.841 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:BDTqV2kEhSm3jsniedKMmhVnmeA= 2023-05-24T21:26:19.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:19 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:BDTqV2kEhSm3jsniedKMmhVnmeA=', 'Content-Length': '0'} 2023-05-24T21:26:19.844 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095bd263e93416eb3-00646e80fb-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:26:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:19.845 DEBUG:boto:b'qskrps-46999falseobj1nulltrue2023-05-24T21:25:48.924Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-24T21:25:48.942Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-24T21:25:48.968Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-24T21:25:48.995Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:26:19.845 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:26:19.846 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:26:19.846 DEBUG:boto:path=/qskrps-46/?versions&max-keys=999 2023-05-24T21:26:19.846 DEBUG:boto:auth_path=/qskrps-46/?versions&max-keys=999 2023-05-24T21:26:19.847 DEBUG:boto:Method: GET 2023-05-24T21:26:19.847 DEBUG:boto:Path: /qskrps-46/?versions&max-keys=999 2023-05-24T21:26:19.847 DEBUG:boto:Data: 2023-05-24T21:26:19.848 DEBUG:boto:Headers: {} 2023-05-24T21:26:19.848 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:19.848 DEBUG:boto:Port: 8000 2023-05-24T21:26:19.849 DEBUG:boto:Params: {} 2023-05-24T21:26:19.849 DEBUG:boto:Token: None 2023-05-24T21:26:19.850 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:19 GMT /qskrps-46/?versions 2023-05-24T21:26:19.850 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:BDTqV2kEhSm3jsniedKMmhVnmeA= 2023-05-24T21:26:19.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:19 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:BDTqV2kEhSm3jsniedKMmhVnmeA=', 'Content-Length': '0'} 2023-05-24T21:26:19.852 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000692f3d99942f0f01-00646e80fb-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:26:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:19.853 DEBUG:boto:b'qskrps-46999falseobj1nulltrue2023-05-24T21:25:48.924Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-24T21:25:48.942Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-24T21:25:48.968Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-24T21:25:48.995Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:26:19.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-24T21:26:19.854 DEBUG:boto:path=/qskrps-46/obj1 2023-05-24T21:26:19.854 DEBUG:boto:auth_path=/qskrps-46/obj1 2023-05-24T21:26:19.855 DEBUG:boto:path=/qskrps-46/obj1?versionId=null 2023-05-24T21:26:19.855 DEBUG:boto:auth_path=/qskrps-46/obj1?versionId=null 2023-05-24T21:26:19.856 DEBUG:boto:Method: GET 2023-05-24T21:26:19.856 DEBUG:boto:Path: /qskrps-46/obj1?versionId=null 2023-05-24T21:26:19.856 DEBUG:boto:Data: 2023-05-24T21:26:19.857 DEBUG:boto:Headers: {} 2023-05-24T21:26:19.857 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:19.857 DEBUG:boto:Port: 8001 2023-05-24T21:26:19.858 DEBUG:boto:Params: {} 2023-05-24T21:26:19.858 DEBUG:boto:Token: None 2023-05-24T21:26:19.859 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:19 GMT /qskrps-46/obj1?versionId=null 2023-05-24T21:26:19.859 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:SXzKXJUUmHpbd+gZhG0H0sLfZLg= 2023-05-24T21:26:19.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:19 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:SXzKXJUUmHpbd+gZhG0H0sLfZLg=', 'Content-Length': '0'} 2023-05-24T21:26:19.861 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c67af022486e2be7-00646e80fb-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:19.862 DEBUG:boto:path=/qskrps-46/obj1 2023-05-24T21:26:19.863 DEBUG:boto:auth_path=/qskrps-46/obj1 2023-05-24T21:26:19.863 DEBUG:boto:path=/qskrps-46/obj1?versionId=null 2023-05-24T21:26:19.864 DEBUG:boto:auth_path=/qskrps-46/obj1?versionId=null 2023-05-24T21:26:19.864 DEBUG:boto:Method: GET 2023-05-24T21:26:19.864 DEBUG:boto:Path: /qskrps-46/obj1?versionId=null 2023-05-24T21:26:19.865 DEBUG:boto:Data: 2023-05-24T21:26:19.865 DEBUG:boto:Headers: {} 2023-05-24T21:26:19.865 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:19.866 DEBUG:boto:Port: 8000 2023-05-24T21:26:19.866 DEBUG:boto:Params: {} 2023-05-24T21:26:19.867 DEBUG:boto:Token: None 2023-05-24T21:26:19.867 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:19 GMT /qskrps-46/obj1?versionId=null 2023-05-24T21:26:19.867 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:SXzKXJUUmHpbd+gZhG0H0sLfZLg= 2023-05-24T21:26:19.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:19 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:SXzKXJUUmHpbd+gZhG0H0sLfZLg=', 'Content-Length': '0'} 2023-05-24T21:26:19.870 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000942b306d6e466bad-00646e80fb-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:19.871 DEBUG:boto:path=/qskrps-46/obj1 2023-05-24T21:26:19.871 DEBUG:boto:auth_path=/qskrps-46/obj1 2023-05-24T21:26:19.871 DEBUG:boto:path=/qskrps-46/obj1?versionId=null 2023-05-24T21:26:19.872 DEBUG:boto:auth_path=/qskrps-46/obj1?versionId=null 2023-05-24T21:26:19.872 DEBUG:boto:Method: HEAD 2023-05-24T21:26:19.872 DEBUG:boto:Path: /qskrps-46/obj1?versionId=null 2023-05-24T21:26:19.873 DEBUG:boto:Data: 2023-05-24T21:26:19.873 DEBUG:boto:Headers: {} 2023-05-24T21:26:19.873 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:19.874 DEBUG:boto:Port: 8001 2023-05-24T21:26:19.874 DEBUG:boto:Params: {} 2023-05-24T21:26:19.874 DEBUG:boto:Token: None 2023-05-24T21:26:19.875 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:26:19 GMT /qskrps-46/obj1?versionId=null 2023-05-24T21:26:19.875 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4d19MbfQRKBc96SbFf6vhOLxrJ4= 2023-05-24T21:26:19.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:19 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4d19MbfQRKBc96SbFf6vhOLxrJ4=', 'Content-Length': '0'} 2023-05-24T21:26:19.877 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f62ff6fb3543e5dc-00646e80fb-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:19.878 DEBUG:boto:path=/qskrps-46/obj1 2023-05-24T21:26:19.878 DEBUG:boto:auth_path=/qskrps-46/obj1 2023-05-24T21:26:19.879 DEBUG:boto:path=/qskrps-46/obj1?versionId=null 2023-05-24T21:26:19.879 DEBUG:boto:auth_path=/qskrps-46/obj1?versionId=null 2023-05-24T21:26:19.879 DEBUG:boto:Method: HEAD 2023-05-24T21:26:19.880 DEBUG:boto:Path: /qskrps-46/obj1?versionId=null 2023-05-24T21:26:19.880 DEBUG:boto:Data: 2023-05-24T21:26:19.880 DEBUG:boto:Headers: {} 2023-05-24T21:26:19.881 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:19.881 DEBUG:boto:Port: 8000 2023-05-24T21:26:19.881 DEBUG:boto:Params: {} 2023-05-24T21:26:19.882 DEBUG:boto:Token: None 2023-05-24T21:26:19.882 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:26:19 GMT /qskrps-46/obj1?versionId=null 2023-05-24T21:26:19.882 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4d19MbfQRKBc96SbFf6vhOLxrJ4= 2023-05-24T21:26:19.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:19 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4d19MbfQRKBc96SbFf6vhOLxrJ4=', 'Content-Length': '0'} 2023-05-24T21:26:19.885 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008b97489cc3e56d76-00646e80fb-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:19.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-24T21:26:19.886 DEBUG:boto:path=/qskrps-46/obj1 2023-05-24T21:26:19.886 DEBUG:boto:auth_path=/qskrps-46/obj1 2023-05-24T21:26:19.886 DEBUG:boto:path=/qskrps-46/obj1?versionId=null 2023-05-24T21:26:19.887 DEBUG:boto:auth_path=/qskrps-46/obj1?versionId=null 2023-05-24T21:26:19.887 DEBUG:boto:Method: GET 2023-05-24T21:26:19.887 DEBUG:boto:Path: /qskrps-46/obj1?versionId=null 2023-05-24T21:26:19.888 DEBUG:boto:Data: 2023-05-24T21:26:19.888 DEBUG:boto:Headers: {} 2023-05-24T21:26:19.889 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:19.889 DEBUG:boto:Port: 8001 2023-05-24T21:26:19.889 DEBUG:boto:Params: {} 2023-05-24T21:26:19.890 DEBUG:boto:Token: None 2023-05-24T21:26:19.890 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:19 GMT /qskrps-46/obj1?versionId=null 2023-05-24T21:26:19.891 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:SXzKXJUUmHpbd+gZhG0H0sLfZLg= 2023-05-24T21:26:19.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:19 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:SXzKXJUUmHpbd+gZhG0H0sLfZLg=', 'Content-Length': '0'} 2023-05-24T21:26:19.893 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d969ac596bc10dde-00646e80fb-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:19.936 DEBUG:boto:path=/qskrps-46/obj1 2023-05-24T21:26:19.937 DEBUG:boto:auth_path=/qskrps-46/obj1 2023-05-24T21:26:19.937 DEBUG:boto:path=/qskrps-46/obj1?versionId=null 2023-05-24T21:26:19.937 DEBUG:boto:auth_path=/qskrps-46/obj1?versionId=null 2023-05-24T21:26:19.938 DEBUG:boto:Method: GET 2023-05-24T21:26:19.938 DEBUG:boto:Path: /qskrps-46/obj1?versionId=null 2023-05-24T21:26:19.939 DEBUG:boto:Data: 2023-05-24T21:26:19.939 DEBUG:boto:Headers: {} 2023-05-24T21:26:19.939 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:19.940 DEBUG:boto:Port: 8000 2023-05-24T21:26:19.940 DEBUG:boto:Params: {} 2023-05-24T21:26:19.940 DEBUG:boto:Token: None 2023-05-24T21:26:19.941 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:19 GMT /qskrps-46/obj1?versionId=null 2023-05-24T21:26:19.941 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:SXzKXJUUmHpbd+gZhG0H0sLfZLg= 2023-05-24T21:26:19.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:19 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:SXzKXJUUmHpbd+gZhG0H0sLfZLg=', 'Content-Length': '0'} 2023-05-24T21:26:19.944 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d037fc80566824b6-00646e80fb-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:19.944 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:26:19.945 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:26:19.945 DEBUG:boto:path=/qskrps-46/?prefix=obj1 2023-05-24T21:26:19.945 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj1 2023-05-24T21:26:19.946 DEBUG:boto:Method: GET 2023-05-24T21:26:19.946 DEBUG:boto:Path: /qskrps-46/?prefix=obj1 2023-05-24T21:26:19.946 DEBUG:boto:Data: 2023-05-24T21:26:19.947 DEBUG:boto:Headers: {} 2023-05-24T21:26:19.947 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:19.948 DEBUG:boto:Port: 8001 2023-05-24T21:26:19.948 DEBUG:boto:Params: {} 2023-05-24T21:26:19.948 DEBUG:boto:Token: None 2023-05-24T21:26:19.949 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:19 GMT /qskrps-46/ 2023-05-24T21:26:19.949 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:RAUkulMKwZXyhcB7+dFdB6afj7E= 2023-05-24T21:26:19.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:19 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:RAUkulMKwZXyhcB7+dFdB6afj7E=', 'Content-Length': '0'} 2023-05-24T21:26:19.952 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009774c63c8934a9e2-00646e80fb-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:26:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:19.952 DEBUG:boto:b'qskrps-46obj11000falseobj12023-05-24T21:25:48.924Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:26:19.953 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:26:19.953 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:26:19.954 DEBUG:boto:path=/qskrps-46/?prefix=obj1 2023-05-24T21:26:19.954 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj1 2023-05-24T21:26:19.954 DEBUG:boto:Method: GET 2023-05-24T21:26:19.955 DEBUG:boto:Path: /qskrps-46/?prefix=obj1 2023-05-24T21:26:19.955 DEBUG:boto:Data: 2023-05-24T21:26:19.956 DEBUG:boto:Headers: {} 2023-05-24T21:26:19.956 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:19.956 DEBUG:boto:Port: 8000 2023-05-24T21:26:19.957 DEBUG:boto:Params: {} 2023-05-24T21:26:19.957 DEBUG:boto:Token: None 2023-05-24T21:26:19.957 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:19 GMT /qskrps-46/ 2023-05-24T21:26:19.958 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:RAUkulMKwZXyhcB7+dFdB6afj7E= 2023-05-24T21:26:19.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:19 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:RAUkulMKwZXyhcB7+dFdB6afj7E=', 'Content-Length': '0'} 2023-05-24T21:26:19.960 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad124e206547f59c-00646e80fb-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:26:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:19.961 DEBUG:boto:b'qskrps-46obj11000falseobj12023-05-24T21:25:48.924Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:26:19.962 DEBUG:boto:path=/qskrps-46/obj1 2023-05-24T21:26:19.962 DEBUG:boto:auth_path=/qskrps-46/obj1 2023-05-24T21:26:19.962 DEBUG:boto:Method: HEAD 2023-05-24T21:26:19.963 DEBUG:boto:Path: /qskrps-46/obj1 2023-05-24T21:26:19.963 DEBUG:boto:Data: 2023-05-24T21:26:19.963 DEBUG:boto:Headers: {} 2023-05-24T21:26:19.964 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:19.964 DEBUG:boto:Port: 8001 2023-05-24T21:26:19.964 DEBUG:boto:Params: {} 2023-05-24T21:26:19.965 DEBUG:boto:Token: None 2023-05-24T21:26:19.965 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:26:19 GMT /qskrps-46/obj1 2023-05-24T21:26:19.966 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:9Itm1BUSR1p0nrfpM5JwZ2rxm8U= 2023-05-24T21:26:19.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:19 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:9Itm1BUSR1p0nrfpM5JwZ2rxm8U=', 'Content-Length': '0'} 2023-05-24T21:26:19.968 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004f484f846b9921b8-00646e80fb-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:19.968 DEBUG:boto:path=/qskrps-46/obj1 2023-05-24T21:26:19.969 DEBUG:boto:auth_path=/qskrps-46/obj1 2023-05-24T21:26:19.969 DEBUG:boto:Method: HEAD 2023-05-24T21:26:19.969 DEBUG:boto:Path: /qskrps-46/obj1 2023-05-24T21:26:19.970 DEBUG:boto:Data: 2023-05-24T21:26:19.970 DEBUG:boto:Headers: {} 2023-05-24T21:26:19.970 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:19.971 DEBUG:boto:Port: 8000 2023-05-24T21:26:19.971 DEBUG:boto:Params: {} 2023-05-24T21:26:19.971 DEBUG:boto:Token: None 2023-05-24T21:26:19.972 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:26:19 GMT /qskrps-46/obj1 2023-05-24T21:26:19.972 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:9Itm1BUSR1p0nrfpM5JwZ2rxm8U= 2023-05-24T21:26:19.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:19 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:9Itm1BUSR1p0nrfpM5JwZ2rxm8U=', 'Content-Length': '0'} 2023-05-24T21:26:19.974 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b306f665d01c304f-00646e80fb-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:19.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-24T21:26:19.975 DEBUG:boto:path=/qskrps-46/obj1 2023-05-24T21:26:19.976 DEBUG:boto:auth_path=/qskrps-46/obj1 2023-05-24T21:26:19.976 DEBUG:boto:Method: GET 2023-05-24T21:26:19.976 DEBUG:boto:Path: /qskrps-46/obj1 2023-05-24T21:26:19.977 DEBUG:boto:Data: 2023-05-24T21:26:19.977 DEBUG:boto:Headers: {} 2023-05-24T21:26:19.977 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:19.978 DEBUG:boto:Port: 8001 2023-05-24T21:26:19.978 DEBUG:boto:Params: {} 2023-05-24T21:26:19.978 DEBUG:boto:Token: None 2023-05-24T21:26:19.979 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:19 GMT /qskrps-46/obj1 2023-05-24T21:26:19.979 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:GcRc/9aO5MX4sSicUWxaTywfuUs= 2023-05-24T21:26:19.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:19 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:GcRc/9aO5MX4sSicUWxaTywfuUs=', 'Content-Length': '0'} 2023-05-24T21:26:19.981 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006682a89efe0fcc5a-00646e80fb-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:19.982 DEBUG:boto:path=/qskrps-46/obj1 2023-05-24T21:26:19.982 DEBUG:boto:auth_path=/qskrps-46/obj1 2023-05-24T21:26:19.983 DEBUG:boto:Method: GET 2023-05-24T21:26:19.983 DEBUG:boto:Path: /qskrps-46/obj1 2023-05-24T21:26:19.983 DEBUG:boto:Data: 2023-05-24T21:26:19.984 DEBUG:boto:Headers: {} 2023-05-24T21:26:19.984 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:19.985 DEBUG:boto:Port: 8000 2023-05-24T21:26:19.985 DEBUG:boto:Params: {} 2023-05-24T21:26:19.985 DEBUG:boto:Token: None 2023-05-24T21:26:19.986 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:19 GMT /qskrps-46/obj1 2023-05-24T21:26:19.986 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:GcRc/9aO5MX4sSicUWxaTywfuUs= 2023-05-24T21:26:19.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:19 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:GcRc/9aO5MX4sSicUWxaTywfuUs=', 'Content-Length': '0'} 2023-05-24T21:26:19.988 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f8b4ba4e5f97d76f-00646e80fb-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:19.989 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:26:19.989 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:26:19.990 DEBUG:boto:path=/qskrps-46/?prefix=obj1 2023-05-24T21:26:19.990 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj1 2023-05-24T21:26:19.991 DEBUG:boto:Method: GET 2023-05-24T21:26:19.991 DEBUG:boto:Path: /qskrps-46/?prefix=obj1 2023-05-24T21:26:19.991 DEBUG:boto:Data: 2023-05-24T21:26:19.992 DEBUG:boto:Headers: {} 2023-05-24T21:26:19.992 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:19.992 DEBUG:boto:Port: 8001 2023-05-24T21:26:19.993 DEBUG:boto:Params: {} 2023-05-24T21:26:19.993 DEBUG:boto:Token: None 2023-05-24T21:26:19.994 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:19 GMT /qskrps-46/ 2023-05-24T21:26:19.994 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:RAUkulMKwZXyhcB7+dFdB6afj7E= 2023-05-24T21:26:19.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:19 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:RAUkulMKwZXyhcB7+dFdB6afj7E=', 'Content-Length': '0'} 2023-05-24T21:26:19.996 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d8f119181594aff-00646e80fb-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:26:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:19.997 DEBUG:boto:b'qskrps-46obj11000falseobj12023-05-24T21:25:48.924Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:26:19.997 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:26:19.998 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:26:19.998 DEBUG:boto:path=/qskrps-46/?prefix=obj1 2023-05-24T21:26:19.999 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj1 2023-05-24T21:26:19.999 DEBUG:boto:Method: GET 2023-05-24T21:26:19.999 DEBUG:boto:Path: /qskrps-46/?prefix=obj1 2023-05-24T21:26:20.000 DEBUG:boto:Data: 2023-05-24T21:26:20.000 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.000 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:20.001 DEBUG:boto:Port: 8000 2023-05-24T21:26:20.001 DEBUG:boto:Params: {} 2023-05-24T21:26:20.002 DEBUG:boto:Token: None 2023-05-24T21:26:20.002 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:20 GMT /qskrps-46/ 2023-05-24T21:26:20.002 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Elhi2CO6BWtJwuCficzl3xxxGkw= 2023-05-24T21:26:20.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Elhi2CO6BWtJwuCficzl3xxxGkw=', 'Content-Length': '0'} 2023-05-24T21:26:20.005 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008341e5bfa99fba08-00646e80fb-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.048 DEBUG:boto:b'qskrps-46obj11000falseobj12023-05-24T21:25:48.924Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:26:20.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-24T21:26:20.049 DEBUG:boto:path=/qskrps-46/obj2 2023-05-24T21:26:20.049 DEBUG:boto:auth_path=/qskrps-46/obj2 2023-05-24T21:26:20.050 DEBUG:boto:path=/qskrps-46/obj2?versionId=null 2023-05-24T21:26:20.050 DEBUG:boto:auth_path=/qskrps-46/obj2?versionId=null 2023-05-24T21:26:20.051 DEBUG:boto:Method: GET 2023-05-24T21:26:20.051 DEBUG:boto:Path: /qskrps-46/obj2?versionId=null 2023-05-24T21:26:20.051 DEBUG:boto:Data: 2023-05-24T21:26:20.052 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.052 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:20.052 DEBUG:boto:Port: 8001 2023-05-24T21:26:20.053 DEBUG:boto:Params: {} 2023-05-24T21:26:20.053 DEBUG:boto:Token: None 2023-05-24T21:26:20.054 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:20 GMT /qskrps-46/obj2?versionId=null 2023-05-24T21:26:20.054 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:TNSIcJkG235CZmiU6yyHyWIWdLY= 2023-05-24T21:26:20.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:TNSIcJkG235CZmiU6yyHyWIWdLY=', 'Content-Length': '0'} 2023-05-24T21:26:20.056 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f91645ef663d975d-00646e80fc-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.057 DEBUG:boto:path=/qskrps-46/obj2 2023-05-24T21:26:20.057 DEBUG:boto:auth_path=/qskrps-46/obj2 2023-05-24T21:26:20.058 DEBUG:boto:path=/qskrps-46/obj2?versionId=null 2023-05-24T21:26:20.058 DEBUG:boto:auth_path=/qskrps-46/obj2?versionId=null 2023-05-24T21:26:20.058 DEBUG:boto:Method: GET 2023-05-24T21:26:20.059 DEBUG:boto:Path: /qskrps-46/obj2?versionId=null 2023-05-24T21:26:20.059 DEBUG:boto:Data: 2023-05-24T21:26:20.059 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.060 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:20.060 DEBUG:boto:Port: 8000 2023-05-24T21:26:20.061 DEBUG:boto:Params: {} 2023-05-24T21:26:20.061 DEBUG:boto:Token: None 2023-05-24T21:26:20.061 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:20 GMT /qskrps-46/obj2?versionId=null 2023-05-24T21:26:20.062 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:TNSIcJkG235CZmiU6yyHyWIWdLY= 2023-05-24T21:26:20.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:TNSIcJkG235CZmiU6yyHyWIWdLY=', 'Content-Length': '0'} 2023-05-24T21:26:20.064 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000da14ac5f9e3263ef-00646e80fc-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.065 DEBUG:boto:path=/qskrps-46/obj2 2023-05-24T21:26:20.065 DEBUG:boto:auth_path=/qskrps-46/obj2 2023-05-24T21:26:20.065 DEBUG:boto:path=/qskrps-46/obj2?versionId=null 2023-05-24T21:26:20.066 DEBUG:boto:auth_path=/qskrps-46/obj2?versionId=null 2023-05-24T21:26:20.066 DEBUG:boto:Method: HEAD 2023-05-24T21:26:20.066 DEBUG:boto:Path: /qskrps-46/obj2?versionId=null 2023-05-24T21:26:20.067 DEBUG:boto:Data: 2023-05-24T21:26:20.067 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.067 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:20.068 DEBUG:boto:Port: 8001 2023-05-24T21:26:20.068 DEBUG:boto:Params: {} 2023-05-24T21:26:20.068 DEBUG:boto:Token: None 2023-05-24T21:26:20.069 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:26:20 GMT /qskrps-46/obj2?versionId=null 2023-05-24T21:26:20.069 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ugY1LWccy/0qtN6pT5zPa0sPW48= 2023-05-24T21:26:20.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ugY1LWccy/0qtN6pT5zPa0sPW48=', 'Content-Length': '0'} 2023-05-24T21:26:20.071 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004b601e4a794a34f2-00646e80fc-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.072 DEBUG:boto:path=/qskrps-46/obj2 2023-05-24T21:26:20.072 DEBUG:boto:auth_path=/qskrps-46/obj2 2023-05-24T21:26:20.072 DEBUG:boto:path=/qskrps-46/obj2?versionId=null 2023-05-24T21:26:20.073 DEBUG:boto:auth_path=/qskrps-46/obj2?versionId=null 2023-05-24T21:26:20.073 DEBUG:boto:Method: HEAD 2023-05-24T21:26:20.073 DEBUG:boto:Path: /qskrps-46/obj2?versionId=null 2023-05-24T21:26:20.074 DEBUG:boto:Data: 2023-05-24T21:26:20.074 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.075 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:20.075 DEBUG:boto:Port: 8000 2023-05-24T21:26:20.075 DEBUG:boto:Params: {} 2023-05-24T21:26:20.076 DEBUG:boto:Token: None 2023-05-24T21:26:20.076 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:26:20 GMT /qskrps-46/obj2?versionId=null 2023-05-24T21:26:20.076 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ugY1LWccy/0qtN6pT5zPa0sPW48= 2023-05-24T21:26:20.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ugY1LWccy/0qtN6pT5zPa0sPW48=', 'Content-Length': '0'} 2023-05-24T21:26:20.078 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008eefd0e1893823d7-00646e80fc-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-24T21:26:20.079 DEBUG:boto:path=/qskrps-46/obj2 2023-05-24T21:26:20.080 DEBUG:boto:auth_path=/qskrps-46/obj2 2023-05-24T21:26:20.080 DEBUG:boto:path=/qskrps-46/obj2?versionId=null 2023-05-24T21:26:20.080 DEBUG:boto:auth_path=/qskrps-46/obj2?versionId=null 2023-05-24T21:26:20.081 DEBUG:boto:Method: GET 2023-05-24T21:26:20.081 DEBUG:boto:Path: /qskrps-46/obj2?versionId=null 2023-05-24T21:26:20.082 DEBUG:boto:Data: 2023-05-24T21:26:20.082 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.082 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:20.083 DEBUG:boto:Port: 8001 2023-05-24T21:26:20.083 DEBUG:boto:Params: {} 2023-05-24T21:26:20.083 DEBUG:boto:Token: None 2023-05-24T21:26:20.084 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:20 GMT /qskrps-46/obj2?versionId=null 2023-05-24T21:26:20.084 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:TNSIcJkG235CZmiU6yyHyWIWdLY= 2023-05-24T21:26:20.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:TNSIcJkG235CZmiU6yyHyWIWdLY=', 'Content-Length': '0'} 2023-05-24T21:26:20.086 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e151a1d5676c7fd2-00646e80fc-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.087 DEBUG:boto:path=/qskrps-46/obj2 2023-05-24T21:26:20.087 DEBUG:boto:auth_path=/qskrps-46/obj2 2023-05-24T21:26:20.088 DEBUG:boto:path=/qskrps-46/obj2?versionId=null 2023-05-24T21:26:20.088 DEBUG:boto:auth_path=/qskrps-46/obj2?versionId=null 2023-05-24T21:26:20.088 DEBUG:boto:Method: GET 2023-05-24T21:26:20.089 DEBUG:boto:Path: /qskrps-46/obj2?versionId=null 2023-05-24T21:26:20.089 DEBUG:boto:Data: 2023-05-24T21:26:20.090 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.090 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:20.090 DEBUG:boto:Port: 8000 2023-05-24T21:26:20.091 DEBUG:boto:Params: {} 2023-05-24T21:26:20.091 DEBUG:boto:Token: None 2023-05-24T21:26:20.091 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:20 GMT /qskrps-46/obj2?versionId=null 2023-05-24T21:26:20.092 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:TNSIcJkG235CZmiU6yyHyWIWdLY= 2023-05-24T21:26:20.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:TNSIcJkG235CZmiU6yyHyWIWdLY=', 'Content-Length': '0'} 2023-05-24T21:26:20.094 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea121b46ed34f9c1-00646e80fc-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.095 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:26:20.095 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:26:20.096 DEBUG:boto:path=/qskrps-46/?prefix=obj2 2023-05-24T21:26:20.096 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj2 2023-05-24T21:26:20.096 DEBUG:boto:Method: GET 2023-05-24T21:26:20.097 DEBUG:boto:Path: /qskrps-46/?prefix=obj2 2023-05-24T21:26:20.097 DEBUG:boto:Data: 2023-05-24T21:26:20.097 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.098 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:20.098 DEBUG:boto:Port: 8001 2023-05-24T21:26:20.098 DEBUG:boto:Params: {} 2023-05-24T21:26:20.099 DEBUG:boto:Token: None 2023-05-24T21:26:20.099 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:20 GMT /qskrps-46/ 2023-05-24T21:26:20.100 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Elhi2CO6BWtJwuCficzl3xxxGkw= 2023-05-24T21:26:20.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Elhi2CO6BWtJwuCficzl3xxxGkw=', 'Content-Length': '0'} 2023-05-24T21:26:20.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8a266d57994cb62-00646e80fc-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.103 DEBUG:boto:b'qskrps-46obj21000falseobj22023-05-24T21:25:48.942Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:26:20.103 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:26:20.104 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:26:20.104 DEBUG:boto:path=/qskrps-46/?prefix=obj2 2023-05-24T21:26:20.105 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj2 2023-05-24T21:26:20.105 DEBUG:boto:Method: GET 2023-05-24T21:26:20.105 DEBUG:boto:Path: /qskrps-46/?prefix=obj2 2023-05-24T21:26:20.106 DEBUG:boto:Data: 2023-05-24T21:26:20.106 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.106 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:20.107 DEBUG:boto:Port: 8000 2023-05-24T21:26:20.107 DEBUG:boto:Params: {} 2023-05-24T21:26:20.107 DEBUG:boto:Token: None 2023-05-24T21:26:20.108 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:20 GMT /qskrps-46/ 2023-05-24T21:26:20.108 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Elhi2CO6BWtJwuCficzl3xxxGkw= 2023-05-24T21:26:20.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Elhi2CO6BWtJwuCficzl3xxxGkw=', 'Content-Length': '0'} 2023-05-24T21:26:20.111 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a485d6280b4b22b-00646e80fc-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.112 DEBUG:boto:b'qskrps-46obj21000falseobj22023-05-24T21:25:48.942Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:26:20.112 DEBUG:boto:path=/qskrps-46/obj2 2023-05-24T21:26:20.113 DEBUG:boto:auth_path=/qskrps-46/obj2 2023-05-24T21:26:20.113 DEBUG:boto:Method: HEAD 2023-05-24T21:26:20.113 DEBUG:boto:Path: /qskrps-46/obj2 2023-05-24T21:26:20.114 DEBUG:boto:Data: 2023-05-24T21:26:20.114 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.114 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:20.115 DEBUG:boto:Port: 8001 2023-05-24T21:26:20.115 DEBUG:boto:Params: {} 2023-05-24T21:26:20.115 DEBUG:boto:Token: None 2023-05-24T21:26:20.116 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:26:20 GMT /qskrps-46/obj2 2023-05-24T21:26:20.116 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:mYFV9ivYO4o2TzVsPa8v0M+0jag= 2023-05-24T21:26:20.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:mYFV9ivYO4o2TzVsPa8v0M+0jag=', 'Content-Length': '0'} 2023-05-24T21:26:20.118 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009a24ce05e4130963-00646e80fc-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.119 DEBUG:boto:path=/qskrps-46/obj2 2023-05-24T21:26:20.119 DEBUG:boto:auth_path=/qskrps-46/obj2 2023-05-24T21:26:20.119 DEBUG:boto:Method: HEAD 2023-05-24T21:26:20.120 DEBUG:boto:Path: /qskrps-46/obj2 2023-05-24T21:26:20.120 DEBUG:boto:Data: 2023-05-24T21:26:20.120 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.121 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:20.121 DEBUG:boto:Port: 8000 2023-05-24T21:26:20.121 DEBUG:boto:Params: {} 2023-05-24T21:26:20.122 DEBUG:boto:Token: None 2023-05-24T21:26:20.122 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:26:20 GMT /qskrps-46/obj2 2023-05-24T21:26:20.123 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:mYFV9ivYO4o2TzVsPa8v0M+0jag= 2023-05-24T21:26:20.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:mYFV9ivYO4o2TzVsPa8v0M+0jag=', 'Content-Length': '0'} 2023-05-24T21:26:20.125 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d5067b81651662ee-00646e80fc-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-24T21:26:20.125 DEBUG:boto:path=/qskrps-46/obj2 2023-05-24T21:26:20.126 DEBUG:boto:auth_path=/qskrps-46/obj2 2023-05-24T21:26:20.126 DEBUG:boto:Method: GET 2023-05-24T21:26:20.127 DEBUG:boto:Path: /qskrps-46/obj2 2023-05-24T21:26:20.127 DEBUG:boto:Data: 2023-05-24T21:26:20.127 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.128 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:20.128 DEBUG:boto:Port: 8001 2023-05-24T21:26:20.128 DEBUG:boto:Params: {} 2023-05-24T21:26:20.129 DEBUG:boto:Token: None 2023-05-24T21:26:20.129 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:20 GMT /qskrps-46/obj2 2023-05-24T21:26:20.130 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:0nTIaIWFUHL5Bft5RqeiDauqHrk= 2023-05-24T21:26:20.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:0nTIaIWFUHL5Bft5RqeiDauqHrk=', 'Content-Length': '0'} 2023-05-24T21:26:20.132 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007a57b81d409416c3-00646e80fc-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.132 DEBUG:boto:path=/qskrps-46/obj2 2023-05-24T21:26:20.133 DEBUG:boto:auth_path=/qskrps-46/obj2 2023-05-24T21:26:20.133 DEBUG:boto:Method: GET 2023-05-24T21:26:20.134 DEBUG:boto:Path: /qskrps-46/obj2 2023-05-24T21:26:20.134 DEBUG:boto:Data: 2023-05-24T21:26:20.134 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.135 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:20.135 DEBUG:boto:Port: 8000 2023-05-24T21:26:20.135 DEBUG:boto:Params: {} 2023-05-24T21:26:20.136 DEBUG:boto:Token: None 2023-05-24T21:26:20.136 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:20 GMT /qskrps-46/obj2 2023-05-24T21:26:20.137 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:0nTIaIWFUHL5Bft5RqeiDauqHrk= 2023-05-24T21:26:20.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:0nTIaIWFUHL5Bft5RqeiDauqHrk=', 'Content-Length': '0'} 2023-05-24T21:26:20.139 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000960b99034ea96af6-00646e80fc-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.180 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:26:20.181 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:26:20.181 DEBUG:boto:path=/qskrps-46/?prefix=obj2 2023-05-24T21:26:20.181 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj2 2023-05-24T21:26:20.182 DEBUG:boto:Method: GET 2023-05-24T21:26:20.182 DEBUG:boto:Path: /qskrps-46/?prefix=obj2 2023-05-24T21:26:20.182 DEBUG:boto:Data: 2023-05-24T21:26:20.183 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.183 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:20.183 DEBUG:boto:Port: 8001 2023-05-24T21:26:20.184 DEBUG:boto:Params: {} 2023-05-24T21:26:20.184 DEBUG:boto:Token: None 2023-05-24T21:26:20.185 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:20 GMT /qskrps-46/ 2023-05-24T21:26:20.185 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Elhi2CO6BWtJwuCficzl3xxxGkw= 2023-05-24T21:26:20.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Elhi2CO6BWtJwuCficzl3xxxGkw=', 'Content-Length': '0'} 2023-05-24T21:26:20.188 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002132bafc0e59a631-00646e80fc-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.188 DEBUG:boto:b'qskrps-46obj21000falseobj22023-05-24T21:25:48.942Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:26:20.189 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:26:20.189 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:26:20.190 DEBUG:boto:path=/qskrps-46/?prefix=obj2 2023-05-24T21:26:20.190 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj2 2023-05-24T21:26:20.191 DEBUG:boto:Method: GET 2023-05-24T21:26:20.191 DEBUG:boto:Path: /qskrps-46/?prefix=obj2 2023-05-24T21:26:20.191 DEBUG:boto:Data: 2023-05-24T21:26:20.192 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.192 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:20.192 DEBUG:boto:Port: 8000 2023-05-24T21:26:20.193 DEBUG:boto:Params: {} 2023-05-24T21:26:20.193 DEBUG:boto:Token: None 2023-05-24T21:26:20.194 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:20 GMT /qskrps-46/ 2023-05-24T21:26:20.194 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Elhi2CO6BWtJwuCficzl3xxxGkw= 2023-05-24T21:26:20.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Elhi2CO6BWtJwuCficzl3xxxGkw=', 'Content-Length': '0'} 2023-05-24T21:26:20.197 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c06cb08a3dc4fbc1-00646e80fc-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.197 DEBUG:boto:b'qskrps-46obj21000falseobj22023-05-24T21:25:48.942Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:26:20.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-24T21:26:20.198 DEBUG:boto:path=/qskrps-46/obj3 2023-05-24T21:26:20.199 DEBUG:boto:auth_path=/qskrps-46/obj3 2023-05-24T21:26:20.199 DEBUG:boto:path=/qskrps-46/obj3?versionId=null 2023-05-24T21:26:20.199 DEBUG:boto:auth_path=/qskrps-46/obj3?versionId=null 2023-05-24T21:26:20.200 DEBUG:boto:Method: GET 2023-05-24T21:26:20.200 DEBUG:boto:Path: /qskrps-46/obj3?versionId=null 2023-05-24T21:26:20.200 DEBUG:boto:Data: 2023-05-24T21:26:20.201 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.201 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:20.201 DEBUG:boto:Port: 8001 2023-05-24T21:26:20.202 DEBUG:boto:Params: {} 2023-05-24T21:26:20.202 DEBUG:boto:Token: None 2023-05-24T21:26:20.203 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:20 GMT /qskrps-46/obj3?versionId=null 2023-05-24T21:26:20.203 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:f7XNOQSczGzv/u0shq0/jBO9ESc= 2023-05-24T21:26:20.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:f7XNOQSczGzv/u0shq0/jBO9ESc=', 'Content-Length': '0'} 2023-05-24T21:26:20.205 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000228d20beb48d994d-00646e80fc-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.206 DEBUG:boto:path=/qskrps-46/obj3 2023-05-24T21:26:20.206 DEBUG:boto:auth_path=/qskrps-46/obj3 2023-05-24T21:26:20.206 DEBUG:boto:path=/qskrps-46/obj3?versionId=null 2023-05-24T21:26:20.207 DEBUG:boto:auth_path=/qskrps-46/obj3?versionId=null 2023-05-24T21:26:20.207 DEBUG:boto:Method: GET 2023-05-24T21:26:20.208 DEBUG:boto:Path: /qskrps-46/obj3?versionId=null 2023-05-24T21:26:20.208 DEBUG:boto:Data: 2023-05-24T21:26:20.208 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.209 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:20.209 DEBUG:boto:Port: 8000 2023-05-24T21:26:20.209 DEBUG:boto:Params: {} 2023-05-24T21:26:20.210 DEBUG:boto:Token: None 2023-05-24T21:26:20.210 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:20 GMT /qskrps-46/obj3?versionId=null 2023-05-24T21:26:20.211 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:f7XNOQSczGzv/u0shq0/jBO9ESc= 2023-05-24T21:26:20.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:f7XNOQSczGzv/u0shq0/jBO9ESc=', 'Content-Length': '0'} 2023-05-24T21:26:20.213 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e9f4615dae7abfa3-00646e80fc-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.213 DEBUG:boto:path=/qskrps-46/obj3 2023-05-24T21:26:20.214 DEBUG:boto:auth_path=/qskrps-46/obj3 2023-05-24T21:26:20.214 DEBUG:boto:path=/qskrps-46/obj3?versionId=null 2023-05-24T21:26:20.214 DEBUG:boto:auth_path=/qskrps-46/obj3?versionId=null 2023-05-24T21:26:20.215 DEBUG:boto:Method: HEAD 2023-05-24T21:26:20.215 DEBUG:boto:Path: /qskrps-46/obj3?versionId=null 2023-05-24T21:26:20.215 DEBUG:boto:Data: 2023-05-24T21:26:20.216 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.216 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:20.216 DEBUG:boto:Port: 8001 2023-05-24T21:26:20.217 DEBUG:boto:Params: {} 2023-05-24T21:26:20.217 DEBUG:boto:Token: None 2023-05-24T21:26:20.217 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:26:20 GMT /qskrps-46/obj3?versionId=null 2023-05-24T21:26:20.218 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Y3jq7MtpGJDAUITn1L/O9OIfOP8= 2023-05-24T21:26:20.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Y3jq7MtpGJDAUITn1L/O9OIfOP8=', 'Content-Length': '0'} 2023-05-24T21:26:20.220 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000045c685b041cd030a-00646e80fc-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.220 DEBUG:boto:path=/qskrps-46/obj3 2023-05-24T21:26:20.221 DEBUG:boto:auth_path=/qskrps-46/obj3 2023-05-24T21:26:20.221 DEBUG:boto:path=/qskrps-46/obj3?versionId=null 2023-05-24T21:26:20.221 DEBUG:boto:auth_path=/qskrps-46/obj3?versionId=null 2023-05-24T21:26:20.222 DEBUG:boto:Method: HEAD 2023-05-24T21:26:20.222 DEBUG:boto:Path: /qskrps-46/obj3?versionId=null 2023-05-24T21:26:20.222 DEBUG:boto:Data: 2023-05-24T21:26:20.223 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.223 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:20.223 DEBUG:boto:Port: 8000 2023-05-24T21:26:20.224 DEBUG:boto:Params: {} 2023-05-24T21:26:20.224 DEBUG:boto:Token: None 2023-05-24T21:26:20.225 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:26:20 GMT /qskrps-46/obj3?versionId=null 2023-05-24T21:26:20.225 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Y3jq7MtpGJDAUITn1L/O9OIfOP8= 2023-05-24T21:26:20.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Y3jq7MtpGJDAUITn1L/O9OIfOP8=', 'Content-Length': '0'} 2023-05-24T21:26:20.227 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000046ea00ae3dfe8ba-00646e80fc-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-24T21:26:20.228 DEBUG:boto:path=/qskrps-46/obj3 2023-05-24T21:26:20.228 DEBUG:boto:auth_path=/qskrps-46/obj3 2023-05-24T21:26:20.229 DEBUG:boto:path=/qskrps-46/obj3?versionId=null 2023-05-24T21:26:20.229 DEBUG:boto:auth_path=/qskrps-46/obj3?versionId=null 2023-05-24T21:26:20.229 DEBUG:boto:Method: GET 2023-05-24T21:26:20.230 DEBUG:boto:Path: /qskrps-46/obj3?versionId=null 2023-05-24T21:26:20.230 DEBUG:boto:Data: 2023-05-24T21:26:20.230 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.231 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:20.231 DEBUG:boto:Port: 8001 2023-05-24T21:26:20.231 DEBUG:boto:Params: {} 2023-05-24T21:26:20.232 DEBUG:boto:Token: None 2023-05-24T21:26:20.232 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:20 GMT /qskrps-46/obj3?versionId=null 2023-05-24T21:26:20.233 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:f7XNOQSczGzv/u0shq0/jBO9ESc= 2023-05-24T21:26:20.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:f7XNOQSczGzv/u0shq0/jBO9ESc=', 'Content-Length': '0'} 2023-05-24T21:26:20.235 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006aec50edd5bae4b7-00646e80fc-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.276 DEBUG:boto:path=/qskrps-46/obj3 2023-05-24T21:26:20.277 DEBUG:boto:auth_path=/qskrps-46/obj3 2023-05-24T21:26:20.277 DEBUG:boto:path=/qskrps-46/obj3?versionId=null 2023-05-24T21:26:20.277 DEBUG:boto:auth_path=/qskrps-46/obj3?versionId=null 2023-05-24T21:26:20.278 DEBUG:boto:Method: GET 2023-05-24T21:26:20.278 DEBUG:boto:Path: /qskrps-46/obj3?versionId=null 2023-05-24T21:26:20.278 DEBUG:boto:Data: 2023-05-24T21:26:20.279 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.279 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:20.279 DEBUG:boto:Port: 8000 2023-05-24T21:26:20.280 DEBUG:boto:Params: {} 2023-05-24T21:26:20.280 DEBUG:boto:Token: None 2023-05-24T21:26:20.281 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:20 GMT /qskrps-46/obj3?versionId=null 2023-05-24T21:26:20.281 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:f7XNOQSczGzv/u0shq0/jBO9ESc= 2023-05-24T21:26:20.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:f7XNOQSczGzv/u0shq0/jBO9ESc=', 'Content-Length': '0'} 2023-05-24T21:26:20.283 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000295d10a2146a628d-00646e80fc-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.284 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:26:20.284 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:26:20.285 DEBUG:boto:path=/qskrps-46/?prefix=obj3 2023-05-24T21:26:20.285 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj3 2023-05-24T21:26:20.285 DEBUG:boto:Method: GET 2023-05-24T21:26:20.286 DEBUG:boto:Path: /qskrps-46/?prefix=obj3 2023-05-24T21:26:20.286 DEBUG:boto:Data: 2023-05-24T21:26:20.287 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.287 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:20.287 DEBUG:boto:Port: 8001 2023-05-24T21:26:20.288 DEBUG:boto:Params: {} 2023-05-24T21:26:20.288 DEBUG:boto:Token: None 2023-05-24T21:26:20.288 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:20 GMT /qskrps-46/ 2023-05-24T21:26:20.289 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Elhi2CO6BWtJwuCficzl3xxxGkw= 2023-05-24T21:26:20.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Elhi2CO6BWtJwuCficzl3xxxGkw=', 'Content-Length': '0'} 2023-05-24T21:26:20.292 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1146acf8292c77d-00646e80fc-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.292 DEBUG:boto:b'qskrps-46obj31000falseobj32023-05-24T21:25:48.968Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:26:20.293 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:26:20.293 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:26:20.293 DEBUG:boto:path=/qskrps-46/?prefix=obj3 2023-05-24T21:26:20.294 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj3 2023-05-24T21:26:20.294 DEBUG:boto:Method: GET 2023-05-24T21:26:20.295 DEBUG:boto:Path: /qskrps-46/?prefix=obj3 2023-05-24T21:26:20.295 DEBUG:boto:Data: 2023-05-24T21:26:20.295 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.296 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:20.296 DEBUG:boto:Port: 8000 2023-05-24T21:26:20.296 DEBUG:boto:Params: {} 2023-05-24T21:26:20.297 DEBUG:boto:Token: None 2023-05-24T21:26:20.297 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:20 GMT /qskrps-46/ 2023-05-24T21:26:20.298 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Elhi2CO6BWtJwuCficzl3xxxGkw= 2023-05-24T21:26:20.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Elhi2CO6BWtJwuCficzl3xxxGkw=', 'Content-Length': '0'} 2023-05-24T21:26:20.300 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f057e6907485188-00646e80fc-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.301 DEBUG:boto:b'qskrps-46obj31000falseobj32023-05-24T21:25:48.968Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:26:20.302 DEBUG:boto:path=/qskrps-46/obj3 2023-05-24T21:26:20.302 DEBUG:boto:auth_path=/qskrps-46/obj3 2023-05-24T21:26:20.302 DEBUG:boto:Method: HEAD 2023-05-24T21:26:20.303 DEBUG:boto:Path: /qskrps-46/obj3 2023-05-24T21:26:20.303 DEBUG:boto:Data: 2023-05-24T21:26:20.303 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.304 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:20.304 DEBUG:boto:Port: 8001 2023-05-24T21:26:20.304 DEBUG:boto:Params: {} 2023-05-24T21:26:20.305 DEBUG:boto:Token: None 2023-05-24T21:26:20.305 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:26:20 GMT /qskrps-46/obj3 2023-05-24T21:26:20.305 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:uXXgZepxYrESsSVm3dAeRkG2JzE= 2023-05-24T21:26:20.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:uXXgZepxYrESsSVm3dAeRkG2JzE=', 'Content-Length': '0'} 2023-05-24T21:26:20.308 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000520da725347b9a74-00646e80fc-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.308 DEBUG:boto:path=/qskrps-46/obj3 2023-05-24T21:26:20.309 DEBUG:boto:auth_path=/qskrps-46/obj3 2023-05-24T21:26:20.309 DEBUG:boto:Method: HEAD 2023-05-24T21:26:20.309 DEBUG:boto:Path: /qskrps-46/obj3 2023-05-24T21:26:20.310 DEBUG:boto:Data: 2023-05-24T21:26:20.310 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.310 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:20.311 DEBUG:boto:Port: 8000 2023-05-24T21:26:20.311 DEBUG:boto:Params: {} 2023-05-24T21:26:20.311 DEBUG:boto:Token: None 2023-05-24T21:26:20.312 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:26:20 GMT /qskrps-46/obj3 2023-05-24T21:26:20.312 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:uXXgZepxYrESsSVm3dAeRkG2JzE= 2023-05-24T21:26:20.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:uXXgZepxYrESsSVm3dAeRkG2JzE=', 'Content-Length': '0'} 2023-05-24T21:26:20.314 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008f759d332772e86f-00646e80fc-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-24T21:26:20.315 DEBUG:boto:path=/qskrps-46/obj3 2023-05-24T21:26:20.315 DEBUG:boto:auth_path=/qskrps-46/obj3 2023-05-24T21:26:20.316 DEBUG:boto:Method: GET 2023-05-24T21:26:20.316 DEBUG:boto:Path: /qskrps-46/obj3 2023-05-24T21:26:20.317 DEBUG:boto:Data: 2023-05-24T21:26:20.317 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.317 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:20.318 DEBUG:boto:Port: 8001 2023-05-24T21:26:20.318 DEBUG:boto:Params: {} 2023-05-24T21:26:20.318 DEBUG:boto:Token: None 2023-05-24T21:26:20.319 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:20 GMT /qskrps-46/obj3 2023-05-24T21:26:20.319 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:BgtfKbO7b2LYi1xVPN4xL1wlQVE= 2023-05-24T21:26:20.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:BgtfKbO7b2LYi1xVPN4xL1wlQVE=', 'Content-Length': '0'} 2023-05-24T21:26:20.321 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000046818c122c21021e-00646e80fc-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.322 DEBUG:boto:path=/qskrps-46/obj3 2023-05-24T21:26:20.322 DEBUG:boto:auth_path=/qskrps-46/obj3 2023-05-24T21:26:20.323 DEBUG:boto:Method: GET 2023-05-24T21:26:20.323 DEBUG:boto:Path: /qskrps-46/obj3 2023-05-24T21:26:20.324 DEBUG:boto:Data: 2023-05-24T21:26:20.324 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.324 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:20.325 DEBUG:boto:Port: 8000 2023-05-24T21:26:20.325 DEBUG:boto:Params: {} 2023-05-24T21:26:20.325 DEBUG:boto:Token: None 2023-05-24T21:26:20.326 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:20 GMT /qskrps-46/obj3 2023-05-24T21:26:20.326 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:BgtfKbO7b2LYi1xVPN4xL1wlQVE= 2023-05-24T21:26:20.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:BgtfKbO7b2LYi1xVPN4xL1wlQVE=', 'Content-Length': '0'} 2023-05-24T21:26:20.328 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001112912eefa957bf-00646e80fc-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.329 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:26:20.329 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:26:20.330 DEBUG:boto:path=/qskrps-46/?prefix=obj3 2023-05-24T21:26:20.330 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj3 2023-05-24T21:26:20.330 DEBUG:boto:Method: GET 2023-05-24T21:26:20.331 DEBUG:boto:Path: /qskrps-46/?prefix=obj3 2023-05-24T21:26:20.331 DEBUG:boto:Data: 2023-05-24T21:26:20.331 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.332 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:20.332 DEBUG:boto:Port: 8001 2023-05-24T21:26:20.333 DEBUG:boto:Params: {} 2023-05-24T21:26:20.333 DEBUG:boto:Token: None 2023-05-24T21:26:20.333 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:20 GMT /qskrps-46/ 2023-05-24T21:26:20.334 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Elhi2CO6BWtJwuCficzl3xxxGkw= 2023-05-24T21:26:20.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Elhi2CO6BWtJwuCficzl3xxxGkw=', 'Content-Length': '0'} 2023-05-24T21:26:20.336 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efacba012b200f44-00646e80fc-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.337 DEBUG:boto:b'qskrps-46obj31000falseobj32023-05-24T21:25:48.968Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:26:20.337 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:26:20.338 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:26:20.338 DEBUG:boto:path=/qskrps-46/?prefix=obj3 2023-05-24T21:26:20.339 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj3 2023-05-24T21:26:20.339 DEBUG:boto:Method: GET 2023-05-24T21:26:20.339 DEBUG:boto:Path: /qskrps-46/?prefix=obj3 2023-05-24T21:26:20.340 DEBUG:boto:Data: 2023-05-24T21:26:20.340 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.340 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:20.341 DEBUG:boto:Port: 8000 2023-05-24T21:26:20.341 DEBUG:boto:Params: {} 2023-05-24T21:26:20.342 DEBUG:boto:Token: None 2023-05-24T21:26:20.342 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:20 GMT /qskrps-46/ 2023-05-24T21:26:20.342 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Elhi2CO6BWtJwuCficzl3xxxGkw= 2023-05-24T21:26:20.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Elhi2CO6BWtJwuCficzl3xxxGkw=', 'Content-Length': '0'} 2023-05-24T21:26:20.345 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087ff2131ddbe43d8-00646e80fc-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.345 DEBUG:boto:b'qskrps-46obj31000falseobj32023-05-24T21:25:48.968Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:26:20.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-24T21:26:20.346 DEBUG:boto:path=/qskrps-46/obj4 2023-05-24T21:26:20.347 DEBUG:boto:auth_path=/qskrps-46/obj4 2023-05-24T21:26:20.347 DEBUG:boto:path=/qskrps-46/obj4?versionId=null 2023-05-24T21:26:20.347 DEBUG:boto:auth_path=/qskrps-46/obj4?versionId=null 2023-05-24T21:26:20.348 DEBUG:boto:Method: GET 2023-05-24T21:26:20.348 DEBUG:boto:Path: /qskrps-46/obj4?versionId=null 2023-05-24T21:26:20.349 DEBUG:boto:Data: 2023-05-24T21:26:20.349 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.349 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:20.350 DEBUG:boto:Port: 8001 2023-05-24T21:26:20.350 DEBUG:boto:Params: {} 2023-05-24T21:26:20.350 DEBUG:boto:Token: None 2023-05-24T21:26:20.351 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:20 GMT /qskrps-46/obj4?versionId=null 2023-05-24T21:26:20.351 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:AAtyLatxKroPNjkRaWhtPnignFU= 2023-05-24T21:26:20.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:AAtyLatxKroPNjkRaWhtPnignFU=', 'Content-Length': '0'} 2023-05-24T21:26:20.353 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000014e42477c89c49c4-00646e80fc-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.354 DEBUG:boto:path=/qskrps-46/obj4 2023-05-24T21:26:20.354 DEBUG:boto:auth_path=/qskrps-46/obj4 2023-05-24T21:26:20.355 DEBUG:boto:path=/qskrps-46/obj4?versionId=null 2023-05-24T21:26:20.355 DEBUG:boto:auth_path=/qskrps-46/obj4?versionId=null 2023-05-24T21:26:20.355 DEBUG:boto:Method: GET 2023-05-24T21:26:20.356 DEBUG:boto:Path: /qskrps-46/obj4?versionId=null 2023-05-24T21:26:20.356 DEBUG:boto:Data: 2023-05-24T21:26:20.357 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.357 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:20.357 DEBUG:boto:Port: 8000 2023-05-24T21:26:20.358 DEBUG:boto:Params: {} 2023-05-24T21:26:20.358 DEBUG:boto:Token: None 2023-05-24T21:26:20.358 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:20 GMT /qskrps-46/obj4?versionId=null 2023-05-24T21:26:20.359 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:AAtyLatxKroPNjkRaWhtPnignFU= 2023-05-24T21:26:20.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:AAtyLatxKroPNjkRaWhtPnignFU=', 'Content-Length': '0'} 2023-05-24T21:26:20.361 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000284f239d261a4f06-00646e80fc-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.404 DEBUG:boto:path=/qskrps-46/obj4 2023-05-24T21:26:20.404 DEBUG:boto:auth_path=/qskrps-46/obj4 2023-05-24T21:26:20.405 DEBUG:boto:path=/qskrps-46/obj4?versionId=null 2023-05-24T21:26:20.405 DEBUG:boto:auth_path=/qskrps-46/obj4?versionId=null 2023-05-24T21:26:20.405 DEBUG:boto:Method: HEAD 2023-05-24T21:26:20.406 DEBUG:boto:Path: /qskrps-46/obj4?versionId=null 2023-05-24T21:26:20.406 DEBUG:boto:Data: 2023-05-24T21:26:20.406 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.407 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:20.407 DEBUG:boto:Port: 8001 2023-05-24T21:26:20.407 DEBUG:boto:Params: {} 2023-05-24T21:26:20.408 DEBUG:boto:Token: None 2023-05-24T21:26:20.408 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:26:20 GMT /qskrps-46/obj4?versionId=null 2023-05-24T21:26:20.409 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:VNzIuF4Kwf0oNiUI1tMVbR8PNmw= 2023-05-24T21:26:20.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:VNzIuF4Kwf0oNiUI1tMVbR8PNmw=', 'Content-Length': '0'} 2023-05-24T21:26:20.411 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f8992f3a3f76678b-00646e80fc-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.411 DEBUG:boto:path=/qskrps-46/obj4 2023-05-24T21:26:20.412 DEBUG:boto:auth_path=/qskrps-46/obj4 2023-05-24T21:26:20.412 DEBUG:boto:path=/qskrps-46/obj4?versionId=null 2023-05-24T21:26:20.412 DEBUG:boto:auth_path=/qskrps-46/obj4?versionId=null 2023-05-24T21:26:20.413 DEBUG:boto:Method: HEAD 2023-05-24T21:26:20.413 DEBUG:boto:Path: /qskrps-46/obj4?versionId=null 2023-05-24T21:26:20.413 DEBUG:boto:Data: 2023-05-24T21:26:20.414 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.414 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:20.414 DEBUG:boto:Port: 8000 2023-05-24T21:26:20.415 DEBUG:boto:Params: {} 2023-05-24T21:26:20.415 DEBUG:boto:Token: None 2023-05-24T21:26:20.415 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:26:20 GMT /qskrps-46/obj4?versionId=null 2023-05-24T21:26:20.416 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:VNzIuF4Kwf0oNiUI1tMVbR8PNmw= 2023-05-24T21:26:20.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:VNzIuF4Kwf0oNiUI1tMVbR8PNmw=', 'Content-Length': '0'} 2023-05-24T21:26:20.418 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000246cb9942a650970-00646e80fc-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-24T21:26:20.419 DEBUG:boto:path=/qskrps-46/obj4 2023-05-24T21:26:20.419 DEBUG:boto:auth_path=/qskrps-46/obj4 2023-05-24T21:26:20.419 DEBUG:boto:path=/qskrps-46/obj4?versionId=null 2023-05-24T21:26:20.420 DEBUG:boto:auth_path=/qskrps-46/obj4?versionId=null 2023-05-24T21:26:20.420 DEBUG:boto:Method: GET 2023-05-24T21:26:20.421 DEBUG:boto:Path: /qskrps-46/obj4?versionId=null 2023-05-24T21:26:20.421 DEBUG:boto:Data: 2023-05-24T21:26:20.421 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.422 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:20.422 DEBUG:boto:Port: 8001 2023-05-24T21:26:20.422 DEBUG:boto:Params: {} 2023-05-24T21:26:20.423 DEBUG:boto:Token: None 2023-05-24T21:26:20.423 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:20 GMT /qskrps-46/obj4?versionId=null 2023-05-24T21:26:20.423 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:AAtyLatxKroPNjkRaWhtPnignFU= 2023-05-24T21:26:20.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:AAtyLatxKroPNjkRaWhtPnignFU=', 'Content-Length': '0'} 2023-05-24T21:26:20.426 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d804c351e156d9c5-00646e80fc-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.426 DEBUG:boto:path=/qskrps-46/obj4 2023-05-24T21:26:20.427 DEBUG:boto:auth_path=/qskrps-46/obj4 2023-05-24T21:26:20.427 DEBUG:boto:path=/qskrps-46/obj4?versionId=null 2023-05-24T21:26:20.427 DEBUG:boto:auth_path=/qskrps-46/obj4?versionId=null 2023-05-24T21:26:20.428 DEBUG:boto:Method: GET 2023-05-24T21:26:20.428 DEBUG:boto:Path: /qskrps-46/obj4?versionId=null 2023-05-24T21:26:20.428 DEBUG:boto:Data: 2023-05-24T21:26:20.429 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.429 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:20.429 DEBUG:boto:Port: 8000 2023-05-24T21:26:20.430 DEBUG:boto:Params: {} 2023-05-24T21:26:20.430 DEBUG:boto:Token: None 2023-05-24T21:26:20.431 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:20 GMT /qskrps-46/obj4?versionId=null 2023-05-24T21:26:20.431 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:AAtyLatxKroPNjkRaWhtPnignFU= 2023-05-24T21:26:20.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:AAtyLatxKroPNjkRaWhtPnignFU=', 'Content-Length': '0'} 2023-05-24T21:26:20.433 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000014018ff5951e93a2-00646e80fc-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.434 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:26:20.434 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:26:20.435 DEBUG:boto:path=/qskrps-46/?prefix=obj4 2023-05-24T21:26:20.435 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj4 2023-05-24T21:26:20.435 DEBUG:boto:Method: GET 2023-05-24T21:26:20.436 DEBUG:boto:Path: /qskrps-46/?prefix=obj4 2023-05-24T21:26:20.436 DEBUG:boto:Data: 2023-05-24T21:26:20.436 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.437 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:20.437 DEBUG:boto:Port: 8001 2023-05-24T21:26:20.437 DEBUG:boto:Params: {} 2023-05-24T21:26:20.438 DEBUG:boto:Token: None 2023-05-24T21:26:20.438 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:20 GMT /qskrps-46/ 2023-05-24T21:26:20.439 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Elhi2CO6BWtJwuCficzl3xxxGkw= 2023-05-24T21:26:20.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Elhi2CO6BWtJwuCficzl3xxxGkw=', 'Content-Length': '0'} 2023-05-24T21:26:20.441 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082db60bb15ebf8d2-00646e80fc-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.442 DEBUG:boto:b'qskrps-46obj41000falseobj42023-05-24T21:25:48.995Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:26:20.442 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:26:20.443 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:26:20.443 DEBUG:boto:path=/qskrps-46/?prefix=obj4 2023-05-24T21:26:20.444 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj4 2023-05-24T21:26:20.444 DEBUG:boto:Method: GET 2023-05-24T21:26:20.444 DEBUG:boto:Path: /qskrps-46/?prefix=obj4 2023-05-24T21:26:20.445 DEBUG:boto:Data: 2023-05-24T21:26:20.445 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.445 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:20.446 DEBUG:boto:Port: 8000 2023-05-24T21:26:20.446 DEBUG:boto:Params: {} 2023-05-24T21:26:20.446 DEBUG:boto:Token: None 2023-05-24T21:26:20.447 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:20 GMT /qskrps-46/ 2023-05-24T21:26:20.447 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Elhi2CO6BWtJwuCficzl3xxxGkw= 2023-05-24T21:26:20.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Elhi2CO6BWtJwuCficzl3xxxGkw=', 'Content-Length': '0'} 2023-05-24T21:26:20.452 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023b3b2a0465a2a2a-00646e80fc-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.452 DEBUG:boto:b'qskrps-46obj41000falseobj42023-05-24T21:25:48.995Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:26:20.453 DEBUG:boto:path=/qskrps-46/obj4 2023-05-24T21:26:20.453 DEBUG:boto:auth_path=/qskrps-46/obj4 2023-05-24T21:26:20.453 DEBUG:boto:Method: HEAD 2023-05-24T21:26:20.454 DEBUG:boto:Path: /qskrps-46/obj4 2023-05-24T21:26:20.454 DEBUG:boto:Data: 2023-05-24T21:26:20.454 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.455 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:20.455 DEBUG:boto:Port: 8001 2023-05-24T21:26:20.455 DEBUG:boto:Params: {} 2023-05-24T21:26:20.456 DEBUG:boto:Token: None 2023-05-24T21:26:20.456 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:26:20 GMT /qskrps-46/obj4 2023-05-24T21:26:20.457 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:6jDBb0i+Dm/HHk9xf0JE2KHgrOs= 2023-05-24T21:26:20.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:6jDBb0i+Dm/HHk9xf0JE2KHgrOs=', 'Content-Length': '0'} 2023-05-24T21:26:20.458 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000847a1cb8c8010a42-00646e80fc-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.459 DEBUG:boto:path=/qskrps-46/obj4 2023-05-24T21:26:20.459 DEBUG:boto:auth_path=/qskrps-46/obj4 2023-05-24T21:26:20.460 DEBUG:boto:Method: HEAD 2023-05-24T21:26:20.460 DEBUG:boto:Path: /qskrps-46/obj4 2023-05-24T21:26:20.460 DEBUG:boto:Data: 2023-05-24T21:26:20.461 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.461 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:20.461 DEBUG:boto:Port: 8000 2023-05-24T21:26:20.462 DEBUG:boto:Params: {} 2023-05-24T21:26:20.462 DEBUG:boto:Token: None 2023-05-24T21:26:20.463 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:26:20 GMT /qskrps-46/obj4 2023-05-24T21:26:20.463 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:6jDBb0i+Dm/HHk9xf0JE2KHgrOs= 2023-05-24T21:26:20.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:6jDBb0i+Dm/HHk9xf0JE2KHgrOs=', 'Content-Length': '0'} 2023-05-24T21:26:20.465 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000040501b23950d939-00646e80fc-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-24T21:26:20.466 DEBUG:boto:path=/qskrps-46/obj4 2023-05-24T21:26:20.466 DEBUG:boto:auth_path=/qskrps-46/obj4 2023-05-24T21:26:20.467 DEBUG:boto:Method: GET 2023-05-24T21:26:20.467 DEBUG:boto:Path: /qskrps-46/obj4 2023-05-24T21:26:20.467 DEBUG:boto:Data: 2023-05-24T21:26:20.468 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.468 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:20.469 DEBUG:boto:Port: 8001 2023-05-24T21:26:20.469 DEBUG:boto:Params: {} 2023-05-24T21:26:20.469 DEBUG:boto:Token: None 2023-05-24T21:26:20.470 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:20 GMT /qskrps-46/obj4 2023-05-24T21:26:20.470 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:7qkUgMCUQObO7wVFXugXx9Q2ULQ= 2023-05-24T21:26:20.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:7qkUgMCUQObO7wVFXugXx9Q2ULQ=', 'Content-Length': '0'} 2023-05-24T21:26:20.472 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f0b67095705cd4c2-00646e80fc-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.516 DEBUG:boto:path=/qskrps-46/obj4 2023-05-24T21:26:20.517 DEBUG:boto:auth_path=/qskrps-46/obj4 2023-05-24T21:26:20.517 DEBUG:boto:Method: GET 2023-05-24T21:26:20.517 DEBUG:boto:Path: /qskrps-46/obj4 2023-05-24T21:26:20.518 DEBUG:boto:Data: 2023-05-24T21:26:20.518 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.518 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:20.519 DEBUG:boto:Port: 8000 2023-05-24T21:26:20.519 DEBUG:boto:Params: {} 2023-05-24T21:26:20.519 DEBUG:boto:Token: None 2023-05-24T21:26:20.520 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:20 GMT /qskrps-46/obj4 2023-05-24T21:26:20.520 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:7qkUgMCUQObO7wVFXugXx9Q2ULQ= 2023-05-24T21:26:20.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:7qkUgMCUQObO7wVFXugXx9Q2ULQ=', 'Content-Length': '0'} 2023-05-24T21:26:20.522 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006f7a433149bb7b66-00646e80fc-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.523 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:26:20.523 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:26:20.524 DEBUG:boto:path=/qskrps-46/?prefix=obj4 2023-05-24T21:26:20.524 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj4 2023-05-24T21:26:20.525 DEBUG:boto:Method: GET 2023-05-24T21:26:20.525 DEBUG:boto:Path: /qskrps-46/?prefix=obj4 2023-05-24T21:26:20.525 DEBUG:boto:Data: 2023-05-24T21:26:20.526 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.526 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:20.526 DEBUG:boto:Port: 8001 2023-05-24T21:26:20.527 DEBUG:boto:Params: {} 2023-05-24T21:26:20.527 DEBUG:boto:Token: None 2023-05-24T21:26:20.527 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:20 GMT /qskrps-46/ 2023-05-24T21:26:20.528 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Elhi2CO6BWtJwuCficzl3xxxGkw= 2023-05-24T21:26:20.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Elhi2CO6BWtJwuCficzl3xxxGkw=', 'Content-Length': '0'} 2023-05-24T21:26:20.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0986a660a83b5d0-00646e80fc-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.531 DEBUG:boto:b'qskrps-46obj41000falseobj42023-05-24T21:25:48.995Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:26:20.532 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:26:20.532 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:26:20.532 DEBUG:boto:path=/qskrps-46/?prefix=obj4 2023-05-24T21:26:20.533 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj4 2023-05-24T21:26:20.533 DEBUG:boto:Method: GET 2023-05-24T21:26:20.533 DEBUG:boto:Path: /qskrps-46/?prefix=obj4 2023-05-24T21:26:20.534 DEBUG:boto:Data: 2023-05-24T21:26:20.534 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.535 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:20.535 DEBUG:boto:Port: 8000 2023-05-24T21:26:20.535 DEBUG:boto:Params: {} 2023-05-24T21:26:20.536 DEBUG:boto:Token: None 2023-05-24T21:26:20.536 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:20 GMT /qskrps-46/ 2023-05-24T21:26:20.536 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Elhi2CO6BWtJwuCficzl3xxxGkw= 2023-05-24T21:26:20.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Elhi2CO6BWtJwuCficzl3xxxGkw=', 'Content-Length': '0'} 2023-05-24T21:26:20.539 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0ae6067adbf866c-00646e80fc-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.540 DEBUG:boto:b'qskrps-46obj41000falseobj42023-05-24T21:25:48.995Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:26:20.540 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qskrps-46 zones={a2, a1} 2023-05-24T21:26:20.541 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=qskrps-47 2023-05-24T21:26:20.541 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-47', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:26:20.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-47', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:26:20.542 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-47 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:26:20.602 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:26:20.603 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:26:20.650 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:26:20.645+0000 7fa2eb5cf780 1 bucket sync caught up with source: 2023-05-24T21:26:20.651 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [00000000001.11.6, , 00000000001.16.6, , 00000000001.48.6, , , 00000000001.26.6, , , ] 2023-05-24T21:26:20.651 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [00000000001.11.6, , 00000000001.16.6, , 00000000001.48.6, , , 00000000001.26.6, , , ] 2023-05-24T21:26:20.651 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:26:20.645+0000 7fa2eb5cf780 0 bucket checkpoint complete 2023-05-24T21:26:20.672 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=qskrps-47 2023-05-24T21:26:20.673 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-47', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:26:20.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-47', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:26:20.673 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-47 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:26:20.729 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:26:20.729 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:26:20.771 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:26:20.765+0000 7f4df4c17780 1 bucket sync caught up with source: 2023-05-24T21:26:20.771 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [00000000001.325.6, , 00000000001.502.6, , 00000000001.533.6, , , 00000000001.844.6, , , ] 2023-05-24T21:26:20.771 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [00000000001.325.6, , 00000000001.502.6, , 00000000001.533.6, , , 00000000001.844.6, , , ] 2023-05-24T21:26:20.771 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:26:20.765+0000 7f4df4c17780 0 bucket checkpoint complete 2023-05-24T21:26:20.784 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qskrps-47 zones={a2, a1} 2023-05-24T21:26:20.784 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:26:20.784 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:26:20.785 DEBUG:boto:Method: HEAD 2023-05-24T21:26:20.785 DEBUG:boto:Path: /qskrps-47/ 2023-05-24T21:26:20.786 DEBUG:boto:Data: 2023-05-24T21:26:20.786 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.786 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:20.787 DEBUG:boto:Port: 8001 2023-05-24T21:26:20.787 DEBUG:boto:Params: {} 2023-05-24T21:26:20.787 DEBUG:boto:Token: None 2023-05-24T21:26:20.788 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:26:20 GMT /qskrps-47/ 2023-05-24T21:26:20.788 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ndgjP3uHQvVhXYej6MOGnnsPwsc= 2023-05-24T21:26:20.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ndgjP3uHQvVhXYej6MOGnnsPwsc=', 'Content-Length': '0'} 2023-05-24T21:26:20.791 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ef4c36742be8e11f-00646e80fc-1625-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.791 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:26:20.792 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:26:20.792 DEBUG:boto:Method: HEAD 2023-05-24T21:26:20.792 DEBUG:boto:Path: /qskrps-47/ 2023-05-24T21:26:20.793 DEBUG:boto:Data: 2023-05-24T21:26:20.793 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.793 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:20.794 DEBUG:boto:Port: 8000 2023-05-24T21:26:20.794 DEBUG:boto:Params: {} 2023-05-24T21:26:20.794 DEBUG:boto:Token: None 2023-05-24T21:26:20.795 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:26:20 GMT /qskrps-47/ 2023-05-24T21:26:20.795 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ndgjP3uHQvVhXYej6MOGnnsPwsc= 2023-05-24T21:26:20.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ndgjP3uHQvVhXYej6MOGnnsPwsc=', 'Content-Length': '0'} 2023-05-24T21:26:20.798 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000042941e870a7132e9-00646e80fc-1627-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.798 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-24T21:26:20.799 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:26:20.799 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:26:20.800 DEBUG:boto:path=/qskrps-47/?versions&max-keys=999 2023-05-24T21:26:20.800 DEBUG:boto:auth_path=/qskrps-47/?versions&max-keys=999 2023-05-24T21:26:20.800 DEBUG:boto:Method: GET 2023-05-24T21:26:20.801 DEBUG:boto:Path: /qskrps-47/?versions&max-keys=999 2023-05-24T21:26:20.801 DEBUG:boto:Data: 2023-05-24T21:26:20.802 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.802 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:20.802 DEBUG:boto:Port: 8001 2023-05-24T21:26:20.803 DEBUG:boto:Params: {} 2023-05-24T21:26:20.803 DEBUG:boto:Token: None 2023-05-24T21:26:20.804 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:20 GMT /qskrps-47/?versions 2023-05-24T21:26:20.804 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:RwTAYVgOd7yfR2QJ7tyEzr6Y2Dw= 2023-05-24T21:26:20.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:RwTAYVgOd7yfR2QJ7tyEzr6Y2Dw=', 'Content-Length': '0'} 2023-05-24T21:26:20.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006072550ee294acdc-00646e80fc-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.807 DEBUG:boto:b'qskrps-47999falseobj1nulltrue2023-05-24T21:25:49.049Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-24T21:25:49.067Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-24T21:25:49.084Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-24T21:25:49.101Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:26:20.808 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-05-24T21:26:20.808 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-05-24T21:26:20.808 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-05-24T21:26:20.809 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-05-24T21:26:20.809 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-24T21:26:20.810 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:26:20.810 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:26:20.810 DEBUG:boto:path=/qskrps-47/?versions&max-keys=999 2023-05-24T21:26:20.811 DEBUG:boto:auth_path=/qskrps-47/?versions&max-keys=999 2023-05-24T21:26:20.811 DEBUG:boto:Method: GET 2023-05-24T21:26:20.811 DEBUG:boto:Path: /qskrps-47/?versions&max-keys=999 2023-05-24T21:26:20.812 DEBUG:boto:Data: 2023-05-24T21:26:20.812 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.813 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:20.813 DEBUG:boto:Port: 8000 2023-05-24T21:26:20.813 DEBUG:boto:Params: {} 2023-05-24T21:26:20.814 DEBUG:boto:Token: None 2023-05-24T21:26:20.814 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:20 GMT /qskrps-47/?versions 2023-05-24T21:26:20.814 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:RwTAYVgOd7yfR2QJ7tyEzr6Y2Dw= 2023-05-24T21:26:20.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:RwTAYVgOd7yfR2QJ7tyEzr6Y2Dw=', 'Content-Length': '0'} 2023-05-24T21:26:20.817 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000712f661cf8f617b9-00646e80fc-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.818 DEBUG:boto:b'qskrps-47999falseobj1nulltrue2023-05-24T21:25:49.049Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-24T21:25:49.067Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-24T21:25:49.084Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-24T21:25:49.101Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:26:20.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-05-24T21:26:20.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-05-24T21:26:20.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-05-24T21:26:20.820 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-05-24T21:26:20.820 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:26:20.820 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:26:20.821 DEBUG:boto:path=/qskrps-47/?versions&max-keys=999 2023-05-24T21:26:20.821 DEBUG:boto:auth_path=/qskrps-47/?versions&max-keys=999 2023-05-24T21:26:20.821 DEBUG:boto:Method: GET 2023-05-24T21:26:20.822 DEBUG:boto:Path: /qskrps-47/?versions&max-keys=999 2023-05-24T21:26:20.822 DEBUG:boto:Data: 2023-05-24T21:26:20.822 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.823 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:20.823 DEBUG:boto:Port: 8001 2023-05-24T21:26:20.823 DEBUG:boto:Params: {} 2023-05-24T21:26:20.824 DEBUG:boto:Token: None 2023-05-24T21:26:20.824 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:20 GMT /qskrps-47/?versions 2023-05-24T21:26:20.825 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:RwTAYVgOd7yfR2QJ7tyEzr6Y2Dw= 2023-05-24T21:26:20.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:RwTAYVgOd7yfR2QJ7tyEzr6Y2Dw=', 'Content-Length': '0'} 2023-05-24T21:26:20.827 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d6602be72e60f6f-00646e80fc-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.828 DEBUG:boto:b'qskrps-47999falseobj1nulltrue2023-05-24T21:25:49.049Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-24T21:25:49.067Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-24T21:25:49.084Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-24T21:25:49.101Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:26:20.828 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:26:20.829 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:26:20.829 DEBUG:boto:path=/qskrps-47/?versions&max-keys=999 2023-05-24T21:26:20.830 DEBUG:boto:auth_path=/qskrps-47/?versions&max-keys=999 2023-05-24T21:26:20.830 DEBUG:boto:Method: GET 2023-05-24T21:26:20.830 DEBUG:boto:Path: /qskrps-47/?versions&max-keys=999 2023-05-24T21:26:20.831 DEBUG:boto:Data: 2023-05-24T21:26:20.831 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.831 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:20.832 DEBUG:boto:Port: 8000 2023-05-24T21:26:20.832 DEBUG:boto:Params: {} 2023-05-24T21:26:20.832 DEBUG:boto:Token: None 2023-05-24T21:26:20.833 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:20 GMT /qskrps-47/?versions 2023-05-24T21:26:20.833 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:RwTAYVgOd7yfR2QJ7tyEzr6Y2Dw= 2023-05-24T21:26:20.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:RwTAYVgOd7yfR2QJ7tyEzr6Y2Dw=', 'Content-Length': '0'} 2023-05-24T21:26:20.836 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b9b2e9243b18059-00646e80fc-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.836 DEBUG:boto:b'qskrps-47999falseobj1nulltrue2023-05-24T21:25:49.049Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-24T21:25:49.067Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-24T21:25:49.084Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-24T21:25:49.101Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:26:20.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-24T21:26:20.837 DEBUG:boto:path=/qskrps-47/obj1 2023-05-24T21:26:20.838 DEBUG:boto:auth_path=/qskrps-47/obj1 2023-05-24T21:26:20.838 DEBUG:boto:path=/qskrps-47/obj1?versionId=null 2023-05-24T21:26:20.838 DEBUG:boto:auth_path=/qskrps-47/obj1?versionId=null 2023-05-24T21:26:20.839 DEBUG:boto:Method: GET 2023-05-24T21:26:20.839 DEBUG:boto:Path: /qskrps-47/obj1?versionId=null 2023-05-24T21:26:20.839 DEBUG:boto:Data: 2023-05-24T21:26:20.840 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.840 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:20.841 DEBUG:boto:Port: 8001 2023-05-24T21:26:20.841 DEBUG:boto:Params: {} 2023-05-24T21:26:20.841 DEBUG:boto:Token: None 2023-05-24T21:26:20.842 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:20 GMT /qskrps-47/obj1?versionId=null 2023-05-24T21:26:20.842 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:3HB3A5u5jT4lZDc6/oqdHY8Z7EQ= 2023-05-24T21:26:20.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:3HB3A5u5jT4lZDc6/oqdHY8Z7EQ=', 'Content-Length': '0'} 2023-05-24T21:26:20.844 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005f8f9074166cc7cb-00646e80fc-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.845 DEBUG:boto:path=/qskrps-47/obj1 2023-05-24T21:26:20.845 DEBUG:boto:auth_path=/qskrps-47/obj1 2023-05-24T21:26:20.846 DEBUG:boto:path=/qskrps-47/obj1?versionId=null 2023-05-24T21:26:20.846 DEBUG:boto:auth_path=/qskrps-47/obj1?versionId=null 2023-05-24T21:26:20.846 DEBUG:boto:Method: GET 2023-05-24T21:26:20.847 DEBUG:boto:Path: /qskrps-47/obj1?versionId=null 2023-05-24T21:26:20.847 DEBUG:boto:Data: 2023-05-24T21:26:20.847 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.848 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:20.848 DEBUG:boto:Port: 8000 2023-05-24T21:26:20.849 DEBUG:boto:Params: {} 2023-05-24T21:26:20.849 DEBUG:boto:Token: None 2023-05-24T21:26:20.849 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:20 GMT /qskrps-47/obj1?versionId=null 2023-05-24T21:26:20.850 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:3HB3A5u5jT4lZDc6/oqdHY8Z7EQ= 2023-05-24T21:26:20.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:3HB3A5u5jT4lZDc6/oqdHY8Z7EQ=', 'Content-Length': '0'} 2023-05-24T21:26:20.852 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000828aec4ad315c51-00646e80fc-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.853 DEBUG:boto:path=/qskrps-47/obj1 2023-05-24T21:26:20.853 DEBUG:boto:auth_path=/qskrps-47/obj1 2023-05-24T21:26:20.854 DEBUG:boto:path=/qskrps-47/obj1?versionId=null 2023-05-24T21:26:20.854 DEBUG:boto:auth_path=/qskrps-47/obj1?versionId=null 2023-05-24T21:26:20.854 DEBUG:boto:Method: HEAD 2023-05-24T21:26:20.855 DEBUG:boto:Path: /qskrps-47/obj1?versionId=null 2023-05-24T21:26:20.855 DEBUG:boto:Data: 2023-05-24T21:26:20.855 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.856 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:20.856 DEBUG:boto:Port: 8001 2023-05-24T21:26:20.856 DEBUG:boto:Params: {} 2023-05-24T21:26:20.857 DEBUG:boto:Token: None 2023-05-24T21:26:20.857 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:26:20 GMT /qskrps-47/obj1?versionId=null 2023-05-24T21:26:20.858 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:/ukvzlqVJc4E7o/zzsT0CWxH/+o= 2023-05-24T21:26:20.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:/ukvzlqVJc4E7o/zzsT0CWxH/+o=', 'Content-Length': '0'} 2023-05-24T21:26:20.859 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008825e62abfbe9efa-00646e80fc-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.860 DEBUG:boto:path=/qskrps-47/obj1 2023-05-24T21:26:20.860 DEBUG:boto:auth_path=/qskrps-47/obj1 2023-05-24T21:26:20.861 DEBUG:boto:path=/qskrps-47/obj1?versionId=null 2023-05-24T21:26:20.861 DEBUG:boto:auth_path=/qskrps-47/obj1?versionId=null 2023-05-24T21:26:20.861 DEBUG:boto:Method: HEAD 2023-05-24T21:26:20.862 DEBUG:boto:Path: /qskrps-47/obj1?versionId=null 2023-05-24T21:26:20.862 DEBUG:boto:Data: 2023-05-24T21:26:20.862 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.863 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:20.863 DEBUG:boto:Port: 8000 2023-05-24T21:26:20.863 DEBUG:boto:Params: {} 2023-05-24T21:26:20.864 DEBUG:boto:Token: None 2023-05-24T21:26:20.864 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:26:20 GMT /qskrps-47/obj1?versionId=null 2023-05-24T21:26:20.865 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:/ukvzlqVJc4E7o/zzsT0CWxH/+o= 2023-05-24T21:26:20.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:/ukvzlqVJc4E7o/zzsT0CWxH/+o=', 'Content-Length': '0'} 2023-05-24T21:26:20.867 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aec085ff883b3acb-00646e80fc-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-24T21:26:20.867 DEBUG:boto:path=/qskrps-47/obj1 2023-05-24T21:26:20.868 DEBUG:boto:auth_path=/qskrps-47/obj1 2023-05-24T21:26:20.868 DEBUG:boto:path=/qskrps-47/obj1?versionId=null 2023-05-24T21:26:20.869 DEBUG:boto:auth_path=/qskrps-47/obj1?versionId=null 2023-05-24T21:26:20.869 DEBUG:boto:Method: GET 2023-05-24T21:26:20.869 DEBUG:boto:Path: /qskrps-47/obj1?versionId=null 2023-05-24T21:26:20.870 DEBUG:boto:Data: 2023-05-24T21:26:20.870 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.870 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:20.871 DEBUG:boto:Port: 8001 2023-05-24T21:26:20.871 DEBUG:boto:Params: {} 2023-05-24T21:26:20.871 DEBUG:boto:Token: None 2023-05-24T21:26:20.872 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:20 GMT /qskrps-47/obj1?versionId=null 2023-05-24T21:26:20.872 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:3HB3A5u5jT4lZDc6/oqdHY8Z7EQ= 2023-05-24T21:26:20.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:3HB3A5u5jT4lZDc6/oqdHY8Z7EQ=', 'Content-Length': '0'} 2023-05-24T21:26:20.874 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a4de7031fbb8bac1-00646e80fc-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.875 DEBUG:boto:path=/qskrps-47/obj1 2023-05-24T21:26:20.875 DEBUG:boto:auth_path=/qskrps-47/obj1 2023-05-24T21:26:20.876 DEBUG:boto:path=/qskrps-47/obj1?versionId=null 2023-05-24T21:26:20.876 DEBUG:boto:auth_path=/qskrps-47/obj1?versionId=null 2023-05-24T21:26:20.877 DEBUG:boto:Method: GET 2023-05-24T21:26:20.877 DEBUG:boto:Path: /qskrps-47/obj1?versionId=null 2023-05-24T21:26:20.877 DEBUG:boto:Data: 2023-05-24T21:26:20.878 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.878 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:20.878 DEBUG:boto:Port: 8000 2023-05-24T21:26:20.879 DEBUG:boto:Params: {} 2023-05-24T21:26:20.879 DEBUG:boto:Token: None 2023-05-24T21:26:20.880 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:20 GMT /qskrps-47/obj1?versionId=null 2023-05-24T21:26:20.880 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:3HB3A5u5jT4lZDc6/oqdHY8Z7EQ= 2023-05-24T21:26:20.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:3HB3A5u5jT4lZDc6/oqdHY8Z7EQ=', 'Content-Length': '0'} 2023-05-24T21:26:20.882 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000468d384ea8a3c127-00646e80fc-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.924 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:26:20.925 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:26:20.925 DEBUG:boto:path=/qskrps-47/?prefix=obj1 2023-05-24T21:26:20.926 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj1 2023-05-24T21:26:20.926 DEBUG:boto:Method: GET 2023-05-24T21:26:20.926 DEBUG:boto:Path: /qskrps-47/?prefix=obj1 2023-05-24T21:26:20.927 DEBUG:boto:Data: 2023-05-24T21:26:20.927 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.928 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:20.928 DEBUG:boto:Port: 8001 2023-05-24T21:26:20.928 DEBUG:boto:Params: {} 2023-05-24T21:26:20.929 DEBUG:boto:Token: None 2023-05-24T21:26:20.929 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:20 GMT /qskrps-47/ 2023-05-24T21:26:20.930 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:IXlnL0AyK0dIp1pEenuja93erls= 2023-05-24T21:26:20.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:IXlnL0AyK0dIp1pEenuja93erls=', 'Content-Length': '0'} 2023-05-24T21:26:20.932 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e7b548d2beca362-00646e80fc-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.933 DEBUG:boto:b'qskrps-47obj11000falseobj12023-05-24T21:25:49.049Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:26:20.933 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:26:20.934 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:26:20.934 DEBUG:boto:path=/qskrps-47/?prefix=obj1 2023-05-24T21:26:20.935 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj1 2023-05-24T21:26:20.935 DEBUG:boto:Method: GET 2023-05-24T21:26:20.935 DEBUG:boto:Path: /qskrps-47/?prefix=obj1 2023-05-24T21:26:20.936 DEBUG:boto:Data: 2023-05-24T21:26:20.936 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.937 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:20.937 DEBUG:boto:Port: 8000 2023-05-24T21:26:20.937 DEBUG:boto:Params: {} 2023-05-24T21:26:20.938 DEBUG:boto:Token: None 2023-05-24T21:26:20.938 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:20 GMT /qskrps-47/ 2023-05-24T21:26:20.939 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:IXlnL0AyK0dIp1pEenuja93erls= 2023-05-24T21:26:20.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:IXlnL0AyK0dIp1pEenuja93erls=', 'Content-Length': '0'} 2023-05-24T21:26:20.941 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8cf44833bfc8764-00646e80fc-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.941 DEBUG:boto:b'qskrps-47obj11000falseobj12023-05-24T21:25:49.049Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:26:20.942 DEBUG:boto:path=/qskrps-47/obj1 2023-05-24T21:26:20.943 DEBUG:boto:auth_path=/qskrps-47/obj1 2023-05-24T21:26:20.943 DEBUG:boto:Method: HEAD 2023-05-24T21:26:20.943 DEBUG:boto:Path: /qskrps-47/obj1 2023-05-24T21:26:20.944 DEBUG:boto:Data: 2023-05-24T21:26:20.944 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.944 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:20.945 DEBUG:boto:Port: 8001 2023-05-24T21:26:20.945 DEBUG:boto:Params: {} 2023-05-24T21:26:20.945 DEBUG:boto:Token: None 2023-05-24T21:26:20.946 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:26:20 GMT /qskrps-47/obj1 2023-05-24T21:26:20.946 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:xEZyfrBGcgYQj3qgehNBxtZEw7Q= 2023-05-24T21:26:20.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:xEZyfrBGcgYQj3qgehNBxtZEw7Q=', 'Content-Length': '0'} 2023-05-24T21:26:20.948 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c03bd4f82f49cd5b-00646e80fc-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.949 DEBUG:boto:path=/qskrps-47/obj1 2023-05-24T21:26:20.949 DEBUG:boto:auth_path=/qskrps-47/obj1 2023-05-24T21:26:20.950 DEBUG:boto:Method: HEAD 2023-05-24T21:26:20.950 DEBUG:boto:Path: /qskrps-47/obj1 2023-05-24T21:26:20.950 DEBUG:boto:Data: 2023-05-24T21:26:20.951 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.951 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:20.951 DEBUG:boto:Port: 8000 2023-05-24T21:26:20.952 DEBUG:boto:Params: {} 2023-05-24T21:26:20.952 DEBUG:boto:Token: None 2023-05-24T21:26:20.952 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:26:20 GMT /qskrps-47/obj1 2023-05-24T21:26:20.953 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:xEZyfrBGcgYQj3qgehNBxtZEw7Q= 2023-05-24T21:26:20.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:xEZyfrBGcgYQj3qgehNBxtZEw7Q=', 'Content-Length': '0'} 2023-05-24T21:26:20.955 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f0f929fb28855cdd-00646e80fc-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:20.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-24T21:26:20.956 DEBUG:boto:path=/qskrps-47/obj1 2023-05-24T21:26:20.956 DEBUG:boto:auth_path=/qskrps-47/obj1 2023-05-24T21:26:20.957 DEBUG:boto:Method: GET 2023-05-24T21:26:20.957 DEBUG:boto:Path: /qskrps-47/obj1 2023-05-24T21:26:20.957 DEBUG:boto:Data: 2023-05-24T21:26:20.958 DEBUG:boto:Headers: {} 2023-05-24T21:26:20.958 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:20.958 DEBUG:boto:Port: 8001 2023-05-24T21:26:20.959 DEBUG:boto:Params: {} 2023-05-24T21:26:20.959 DEBUG:boto:Token: None 2023-05-24T21:26:20.960 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:20 GMT /qskrps-47/obj1 2023-05-24T21:26:20.960 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:1nGbDPYp2499m3xBEkOhR9vZfUo= 2023-05-24T21:26:20.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:1nGbDPYp2499m3xBEkOhR9vZfUo=', 'Content-Length': '0'} 2023-05-24T21:26:20.962 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f5f7e1c2f231d9cd-00646e80fc-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:21.004 DEBUG:boto:path=/qskrps-47/obj1 2023-05-24T21:26:21.005 DEBUG:boto:auth_path=/qskrps-47/obj1 2023-05-24T21:26:21.005 DEBUG:boto:Method: GET 2023-05-24T21:26:21.005 DEBUG:boto:Path: /qskrps-47/obj1 2023-05-24T21:26:21.006 DEBUG:boto:Data: 2023-05-24T21:26:21.006 DEBUG:boto:Headers: {} 2023-05-24T21:26:21.006 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:21.007 DEBUG:boto:Port: 8000 2023-05-24T21:26:21.007 DEBUG:boto:Params: {} 2023-05-24T21:26:21.008 DEBUG:boto:Token: None 2023-05-24T21:26:21.008 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:21 GMT /qskrps-47/obj1 2023-05-24T21:26:21.008 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:o/Ei/1N1ONChJ7xxVm+3lbYz4rs= 2023-05-24T21:26:21.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:o/Ei/1N1ONChJ7xxVm+3lbYz4rs=', 'Content-Length': '0'} 2023-05-24T21:26:21.010 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008172ab1818b50d42-00646e80fd-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:21.011 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:26:21.012 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:26:21.012 DEBUG:boto:path=/qskrps-47/?prefix=obj1 2023-05-24T21:26:21.012 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj1 2023-05-24T21:26:21.013 DEBUG:boto:Method: GET 2023-05-24T21:26:21.013 DEBUG:boto:Path: /qskrps-47/?prefix=obj1 2023-05-24T21:26:21.013 DEBUG:boto:Data: 2023-05-24T21:26:21.014 DEBUG:boto:Headers: {} 2023-05-24T21:26:21.014 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:21.015 DEBUG:boto:Port: 8001 2023-05-24T21:26:21.015 DEBUG:boto:Params: {} 2023-05-24T21:26:21.015 DEBUG:boto:Token: None 2023-05-24T21:26:21.016 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:21 GMT /qskrps-47/ 2023-05-24T21:26:21.016 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:tQhrcjy8y3G2ohCCC0wseAtKmDI= 2023-05-24T21:26:21.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:tQhrcjy8y3G2ohCCC0wseAtKmDI=', 'Content-Length': '0'} 2023-05-24T21:26:21.019 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1b33dce9674533f-00646e80fd-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:26:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:21.019 DEBUG:boto:b'qskrps-47obj11000falseobj12023-05-24T21:25:49.049Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:26:21.020 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:26:21.020 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:26:21.021 DEBUG:boto:path=/qskrps-47/?prefix=obj1 2023-05-24T21:26:21.021 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj1 2023-05-24T21:26:21.021 DEBUG:boto:Method: GET 2023-05-24T21:26:21.022 DEBUG:boto:Path: /qskrps-47/?prefix=obj1 2023-05-24T21:26:21.022 DEBUG:boto:Data: 2023-05-24T21:26:21.023 DEBUG:boto:Headers: {} 2023-05-24T21:26:21.023 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:21.023 DEBUG:boto:Port: 8000 2023-05-24T21:26:21.024 DEBUG:boto:Params: {} 2023-05-24T21:26:21.024 DEBUG:boto:Token: None 2023-05-24T21:26:21.024 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:21 GMT /qskrps-47/ 2023-05-24T21:26:21.025 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:tQhrcjy8y3G2ohCCC0wseAtKmDI= 2023-05-24T21:26:21.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:tQhrcjy8y3G2ohCCC0wseAtKmDI=', 'Content-Length': '0'} 2023-05-24T21:26:21.028 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075bca87712bf5868-00646e80fd-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:26:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:21.028 DEBUG:boto:b'qskrps-47obj11000falseobj12023-05-24T21:25:49.049Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:26:21.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-24T21:26:21.029 DEBUG:boto:path=/qskrps-47/obj2 2023-05-24T21:26:21.030 DEBUG:boto:auth_path=/qskrps-47/obj2 2023-05-24T21:26:21.030 DEBUG:boto:path=/qskrps-47/obj2?versionId=null 2023-05-24T21:26:21.030 DEBUG:boto:auth_path=/qskrps-47/obj2?versionId=null 2023-05-24T21:26:21.031 DEBUG:boto:Method: GET 2023-05-24T21:26:21.031 DEBUG:boto:Path: /qskrps-47/obj2?versionId=null 2023-05-24T21:26:21.031 DEBUG:boto:Data: 2023-05-24T21:26:21.032 DEBUG:boto:Headers: {} 2023-05-24T21:26:21.032 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:21.033 DEBUG:boto:Port: 8001 2023-05-24T21:26:21.033 DEBUG:boto:Params: {} 2023-05-24T21:26:21.033 DEBUG:boto:Token: None 2023-05-24T21:26:21.034 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:21 GMT /qskrps-47/obj2?versionId=null 2023-05-24T21:26:21.034 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:WyzB+dC78Pd7n84g63j9H5HbgfM= 2023-05-24T21:26:21.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WyzB+dC78Pd7n84g63j9H5HbgfM=', 'Content-Length': '0'} 2023-05-24T21:26:21.036 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006ca0a83f9c8ad885-00646e80fd-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:21.037 DEBUG:boto:path=/qskrps-47/obj2 2023-05-24T21:26:21.037 DEBUG:boto:auth_path=/qskrps-47/obj2 2023-05-24T21:26:21.037 DEBUG:boto:path=/qskrps-47/obj2?versionId=null 2023-05-24T21:26:21.038 DEBUG:boto:auth_path=/qskrps-47/obj2?versionId=null 2023-05-24T21:26:21.038 DEBUG:boto:Method: GET 2023-05-24T21:26:21.039 DEBUG:boto:Path: /qskrps-47/obj2?versionId=null 2023-05-24T21:26:21.039 DEBUG:boto:Data: 2023-05-24T21:26:21.039 DEBUG:boto:Headers: {} 2023-05-24T21:26:21.040 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:21.040 DEBUG:boto:Port: 8000 2023-05-24T21:26:21.040 DEBUG:boto:Params: {} 2023-05-24T21:26:21.041 DEBUG:boto:Token: None 2023-05-24T21:26:21.041 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:21 GMT /qskrps-47/obj2?versionId=null 2023-05-24T21:26:21.042 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:WyzB+dC78Pd7n84g63j9H5HbgfM= 2023-05-24T21:26:21.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WyzB+dC78Pd7n84g63j9H5HbgfM=', 'Content-Length': '0'} 2023-05-24T21:26:21.044 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ad49402e930717e0-00646e80fd-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:21.044 DEBUG:boto:path=/qskrps-47/obj2 2023-05-24T21:26:21.045 DEBUG:boto:auth_path=/qskrps-47/obj2 2023-05-24T21:26:21.045 DEBUG:boto:path=/qskrps-47/obj2?versionId=null 2023-05-24T21:26:21.045 DEBUG:boto:auth_path=/qskrps-47/obj2?versionId=null 2023-05-24T21:26:21.046 DEBUG:boto:Method: HEAD 2023-05-24T21:26:21.046 DEBUG:boto:Path: /qskrps-47/obj2?versionId=null 2023-05-24T21:26:21.047 DEBUG:boto:Data: 2023-05-24T21:26:21.047 DEBUG:boto:Headers: {} 2023-05-24T21:26:21.047 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:21.048 DEBUG:boto:Port: 8001 2023-05-24T21:26:21.048 DEBUG:boto:Params: {} 2023-05-24T21:26:21.048 DEBUG:boto:Token: None 2023-05-24T21:26:21.049 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:26:21 GMT /qskrps-47/obj2?versionId=null 2023-05-24T21:26:21.049 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:yuWe9BxKe10N8CavP/YqHO7QzZU= 2023-05-24T21:26:21.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:yuWe9BxKe10N8CavP/YqHO7QzZU=', 'Content-Length': '0'} 2023-05-24T21:26:21.051 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000023f38b5204b1e3e0-00646e80fd-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:21.052 DEBUG:boto:path=/qskrps-47/obj2 2023-05-24T21:26:21.052 DEBUG:boto:auth_path=/qskrps-47/obj2 2023-05-24T21:26:21.052 DEBUG:boto:path=/qskrps-47/obj2?versionId=null 2023-05-24T21:26:21.053 DEBUG:boto:auth_path=/qskrps-47/obj2?versionId=null 2023-05-24T21:26:21.053 DEBUG:boto:Method: HEAD 2023-05-24T21:26:21.053 DEBUG:boto:Path: /qskrps-47/obj2?versionId=null 2023-05-24T21:26:21.054 DEBUG:boto:Data: 2023-05-24T21:26:21.054 DEBUG:boto:Headers: {} 2023-05-24T21:26:21.054 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:21.055 DEBUG:boto:Port: 8000 2023-05-24T21:26:21.055 DEBUG:boto:Params: {} 2023-05-24T21:26:21.056 DEBUG:boto:Token: None 2023-05-24T21:26:21.056 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:26:21 GMT /qskrps-47/obj2?versionId=null 2023-05-24T21:26:21.056 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:yuWe9BxKe10N8CavP/YqHO7QzZU= 2023-05-24T21:26:21.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:yuWe9BxKe10N8CavP/YqHO7QzZU=', 'Content-Length': '0'} 2023-05-24T21:26:21.058 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003ec5aaa9cb7916d3-00646e80fd-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:21.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-24T21:26:21.059 DEBUG:boto:path=/qskrps-47/obj2 2023-05-24T21:26:21.060 DEBUG:boto:auth_path=/qskrps-47/obj2 2023-05-24T21:26:21.060 DEBUG:boto:path=/qskrps-47/obj2?versionId=null 2023-05-24T21:26:21.060 DEBUG:boto:auth_path=/qskrps-47/obj2?versionId=null 2023-05-24T21:26:21.061 DEBUG:boto:Method: GET 2023-05-24T21:26:21.061 DEBUG:boto:Path: /qskrps-47/obj2?versionId=null 2023-05-24T21:26:21.061 DEBUG:boto:Data: 2023-05-24T21:26:21.062 DEBUG:boto:Headers: {} 2023-05-24T21:26:21.062 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:21.062 DEBUG:boto:Port: 8001 2023-05-24T21:26:21.063 DEBUG:boto:Params: {} 2023-05-24T21:26:21.063 DEBUG:boto:Token: None 2023-05-24T21:26:21.064 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:21 GMT /qskrps-47/obj2?versionId=null 2023-05-24T21:26:21.064 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:WyzB+dC78Pd7n84g63j9H5HbgfM= 2023-05-24T21:26:21.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WyzB+dC78Pd7n84g63j9H5HbgfM=', 'Content-Length': '0'} 2023-05-24T21:26:21.066 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000670504e1d8f47f98-00646e80fd-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:21.067 DEBUG:boto:path=/qskrps-47/obj2 2023-05-24T21:26:21.067 DEBUG:boto:auth_path=/qskrps-47/obj2 2023-05-24T21:26:21.068 DEBUG:boto:path=/qskrps-47/obj2?versionId=null 2023-05-24T21:26:21.068 DEBUG:boto:auth_path=/qskrps-47/obj2?versionId=null 2023-05-24T21:26:21.068 DEBUG:boto:Method: GET 2023-05-24T21:26:21.069 DEBUG:boto:Path: /qskrps-47/obj2?versionId=null 2023-05-24T21:26:21.069 DEBUG:boto:Data: 2023-05-24T21:26:21.069 DEBUG:boto:Headers: {} 2023-05-24T21:26:21.070 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:21.070 DEBUG:boto:Port: 8000 2023-05-24T21:26:21.070 DEBUG:boto:Params: {} 2023-05-24T21:26:21.071 DEBUG:boto:Token: None 2023-05-24T21:26:21.071 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:21 GMT /qskrps-47/obj2?versionId=null 2023-05-24T21:26:21.072 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:WyzB+dC78Pd7n84g63j9H5HbgfM= 2023-05-24T21:26:21.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WyzB+dC78Pd7n84g63j9H5HbgfM=', 'Content-Length': '0'} 2023-05-24T21:26:21.074 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000175861437952019b-00646e80fd-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:21.074 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:26:21.075 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:26:21.075 DEBUG:boto:path=/qskrps-47/?prefix=obj2 2023-05-24T21:26:21.075 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj2 2023-05-24T21:26:21.076 DEBUG:boto:Method: GET 2023-05-24T21:26:21.076 DEBUG:boto:Path: /qskrps-47/?prefix=obj2 2023-05-24T21:26:21.077 DEBUG:boto:Data: 2023-05-24T21:26:21.077 DEBUG:boto:Headers: {} 2023-05-24T21:26:21.077 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:21.078 DEBUG:boto:Port: 8001 2023-05-24T21:26:21.078 DEBUG:boto:Params: {} 2023-05-24T21:26:21.078 DEBUG:boto:Token: None 2023-05-24T21:26:21.079 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:21 GMT /qskrps-47/ 2023-05-24T21:26:21.079 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:tQhrcjy8y3G2ohCCC0wseAtKmDI= 2023-05-24T21:26:21.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:tQhrcjy8y3G2ohCCC0wseAtKmDI=', 'Content-Length': '0'} 2023-05-24T21:26:21.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fa485e5d615d5c7-00646e80fd-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:26:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:21.083 DEBUG:boto:b'qskrps-47obj21000falseobj22023-05-24T21:25:49.067Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:26:21.083 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:26:21.084 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:26:21.084 DEBUG:boto:path=/qskrps-47/?prefix=obj2 2023-05-24T21:26:21.085 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj2 2023-05-24T21:26:21.085 DEBUG:boto:Method: GET 2023-05-24T21:26:21.085 DEBUG:boto:Path: /qskrps-47/?prefix=obj2 2023-05-24T21:26:21.086 DEBUG:boto:Data: 2023-05-24T21:26:21.086 DEBUG:boto:Headers: {} 2023-05-24T21:26:21.086 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:21.087 DEBUG:boto:Port: 8000 2023-05-24T21:26:21.087 DEBUG:boto:Params: {} 2023-05-24T21:26:21.087 DEBUG:boto:Token: None 2023-05-24T21:26:21.088 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:21 GMT /qskrps-47/ 2023-05-24T21:26:21.088 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:tQhrcjy8y3G2ohCCC0wseAtKmDI= 2023-05-24T21:26:21.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:tQhrcjy8y3G2ohCCC0wseAtKmDI=', 'Content-Length': '0'} 2023-05-24T21:26:21.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5d97ab8baada0bb-00646e80fd-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:26:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:21.091 DEBUG:boto:b'qskrps-47obj21000falseobj22023-05-24T21:25:49.067Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:26:21.092 DEBUG:boto:path=/qskrps-47/obj2 2023-05-24T21:26:21.092 DEBUG:boto:auth_path=/qskrps-47/obj2 2023-05-24T21:26:21.093 DEBUG:boto:Method: HEAD 2023-05-24T21:26:21.093 DEBUG:boto:Path: /qskrps-47/obj2 2023-05-24T21:26:21.093 DEBUG:boto:Data: 2023-05-24T21:26:21.094 DEBUG:boto:Headers: {} 2023-05-24T21:26:21.094 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:21.094 DEBUG:boto:Port: 8001 2023-05-24T21:26:21.095 DEBUG:boto:Params: {} 2023-05-24T21:26:21.095 DEBUG:boto:Token: None 2023-05-24T21:26:21.096 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:26:21 GMT /qskrps-47/obj2 2023-05-24T21:26:21.096 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:gYwoKNNCnpA7sg+R69Y+2UePrfM= 2023-05-24T21:26:21.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:gYwoKNNCnpA7sg+R69Y+2UePrfM=', 'Content-Length': '0'} 2023-05-24T21:26:21.098 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000011a2bd87e007b50d-00646e80fd-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:21.098 DEBUG:boto:path=/qskrps-47/obj2 2023-05-24T21:26:21.099 DEBUG:boto:auth_path=/qskrps-47/obj2 2023-05-24T21:26:21.099 DEBUG:boto:Method: HEAD 2023-05-24T21:26:21.099 DEBUG:boto:Path: /qskrps-47/obj2 2023-05-24T21:26:21.100 DEBUG:boto:Data: 2023-05-24T21:26:21.100 DEBUG:boto:Headers: {} 2023-05-24T21:26:21.101 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:21.101 DEBUG:boto:Port: 8000 2023-05-24T21:26:21.101 DEBUG:boto:Params: {} 2023-05-24T21:26:21.102 DEBUG:boto:Token: None 2023-05-24T21:26:21.102 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:26:21 GMT /qskrps-47/obj2 2023-05-24T21:26:21.102 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:gYwoKNNCnpA7sg+R69Y+2UePrfM= 2023-05-24T21:26:21.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:gYwoKNNCnpA7sg+R69Y+2UePrfM=', 'Content-Length': '0'} 2023-05-24T21:26:21.104 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000001e6739179c12028-00646e80fd-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:21.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-24T21:26:21.105 DEBUG:boto:path=/qskrps-47/obj2 2023-05-24T21:26:21.106 DEBUG:boto:auth_path=/qskrps-47/obj2 2023-05-24T21:26:21.106 DEBUG:boto:Method: GET 2023-05-24T21:26:21.106 DEBUG:boto:Path: /qskrps-47/obj2 2023-05-24T21:26:21.107 DEBUG:boto:Data: 2023-05-24T21:26:21.107 DEBUG:boto:Headers: {} 2023-05-24T21:26:21.108 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:21.108 DEBUG:boto:Port: 8001 2023-05-24T21:26:21.108 DEBUG:boto:Params: {} 2023-05-24T21:26:21.109 DEBUG:boto:Token: None 2023-05-24T21:26:21.109 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:21 GMT /qskrps-47/obj2 2023-05-24T21:26:21.110 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:alyVW6XRHaKoyiPdLkaO2UHRLn8= 2023-05-24T21:26:21.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:alyVW6XRHaKoyiPdLkaO2UHRLn8=', 'Content-Length': '0'} 2023-05-24T21:26:21.112 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c5d63d83516b2c48-00646e80fd-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:21.112 DEBUG:boto:path=/qskrps-47/obj2 2023-05-24T21:26:21.113 DEBUG:boto:auth_path=/qskrps-47/obj2 2023-05-24T21:26:21.113 DEBUG:boto:Method: GET 2023-05-24T21:26:21.113 DEBUG:boto:Path: /qskrps-47/obj2 2023-05-24T21:26:21.114 DEBUG:boto:Data: 2023-05-24T21:26:21.114 DEBUG:boto:Headers: {} 2023-05-24T21:26:21.114 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:21.115 DEBUG:boto:Port: 8000 2023-05-24T21:26:21.115 DEBUG:boto:Params: {} 2023-05-24T21:26:21.116 DEBUG:boto:Token: None 2023-05-24T21:26:21.116 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:21 GMT /qskrps-47/obj2 2023-05-24T21:26:21.116 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:alyVW6XRHaKoyiPdLkaO2UHRLn8= 2023-05-24T21:26:21.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:alyVW6XRHaKoyiPdLkaO2UHRLn8=', 'Content-Length': '0'} 2023-05-24T21:26:21.118 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004415d131585bd798-00646e80fd-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:21.160 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:26:21.161 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:26:21.161 DEBUG:boto:path=/qskrps-47/?prefix=obj2 2023-05-24T21:26:21.161 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj2 2023-05-24T21:26:21.162 DEBUG:boto:Method: GET 2023-05-24T21:26:21.162 DEBUG:boto:Path: /qskrps-47/?prefix=obj2 2023-05-24T21:26:21.163 DEBUG:boto:Data: 2023-05-24T21:26:21.163 DEBUG:boto:Headers: {} 2023-05-24T21:26:21.163 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:21.164 DEBUG:boto:Port: 8001 2023-05-24T21:26:21.164 DEBUG:boto:Params: {} 2023-05-24T21:26:21.164 DEBUG:boto:Token: None 2023-05-24T21:26:21.165 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:21 GMT /qskrps-47/ 2023-05-24T21:26:21.165 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:tQhrcjy8y3G2ohCCC0wseAtKmDI= 2023-05-24T21:26:21.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:tQhrcjy8y3G2ohCCC0wseAtKmDI=', 'Content-Length': '0'} 2023-05-24T21:26:21.168 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7848fcc01ea521c-00646e80fd-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:26:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:21.169 DEBUG:boto:b'qskrps-47obj21000falseobj22023-05-24T21:25:49.067Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:26:21.169 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:26:21.170 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:26:21.170 DEBUG:boto:path=/qskrps-47/?prefix=obj2 2023-05-24T21:26:21.170 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj2 2023-05-24T21:26:21.171 DEBUG:boto:Method: GET 2023-05-24T21:26:21.171 DEBUG:boto:Path: /qskrps-47/?prefix=obj2 2023-05-24T21:26:21.171 DEBUG:boto:Data: 2023-05-24T21:26:21.172 DEBUG:boto:Headers: {} 2023-05-24T21:26:21.172 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:21.173 DEBUG:boto:Port: 8000 2023-05-24T21:26:21.173 DEBUG:boto:Params: {} 2023-05-24T21:26:21.173 DEBUG:boto:Token: None 2023-05-24T21:26:21.174 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:21 GMT /qskrps-47/ 2023-05-24T21:26:21.174 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:tQhrcjy8y3G2ohCCC0wseAtKmDI= 2023-05-24T21:26:21.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:tQhrcjy8y3G2ohCCC0wseAtKmDI=', 'Content-Length': '0'} 2023-05-24T21:26:21.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f5431cda3d33155-00646e80fd-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:26:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:21.177 DEBUG:boto:b'qskrps-47obj21000falseobj22023-05-24T21:25:49.067Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:26:21.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-24T21:26:21.178 DEBUG:boto:path=/qskrps-47/obj3 2023-05-24T21:26:21.178 DEBUG:boto:auth_path=/qskrps-47/obj3 2023-05-24T21:26:21.179 DEBUG:boto:path=/qskrps-47/obj3?versionId=null 2023-05-24T21:26:21.179 DEBUG:boto:auth_path=/qskrps-47/obj3?versionId=null 2023-05-24T21:26:21.180 DEBUG:boto:Method: GET 2023-05-24T21:26:21.180 DEBUG:boto:Path: /qskrps-47/obj3?versionId=null 2023-05-24T21:26:21.180 DEBUG:boto:Data: 2023-05-24T21:26:21.181 DEBUG:boto:Headers: {} 2023-05-24T21:26:21.181 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:21.181 DEBUG:boto:Port: 8001 2023-05-24T21:26:21.182 DEBUG:boto:Params: {} 2023-05-24T21:26:21.182 DEBUG:boto:Token: None 2023-05-24T21:26:21.183 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:21 GMT /qskrps-47/obj3?versionId=null 2023-05-24T21:26:21.183 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:EfDj05RIKZnZKZs3GRWL+ye+OLQ= 2023-05-24T21:26:21.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:EfDj05RIKZnZKZs3GRWL+ye+OLQ=', 'Content-Length': '0'} 2023-05-24T21:26:21.185 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003bf0ec089c6d9662-00646e80fd-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:21.228 DEBUG:boto:path=/qskrps-47/obj3 2023-05-24T21:26:21.229 DEBUG:boto:auth_path=/qskrps-47/obj3 2023-05-24T21:26:21.229 DEBUG:boto:path=/qskrps-47/obj3?versionId=null 2023-05-24T21:26:21.229 DEBUG:boto:auth_path=/qskrps-47/obj3?versionId=null 2023-05-24T21:26:21.230 DEBUG:boto:Method: GET 2023-05-24T21:26:21.230 DEBUG:boto:Path: /qskrps-47/obj3?versionId=null 2023-05-24T21:26:21.231 DEBUG:boto:Data: 2023-05-24T21:26:21.231 DEBUG:boto:Headers: {} 2023-05-24T21:26:21.231 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:21.232 DEBUG:boto:Port: 8000 2023-05-24T21:26:21.232 DEBUG:boto:Params: {} 2023-05-24T21:26:21.232 DEBUG:boto:Token: None 2023-05-24T21:26:21.233 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:21 GMT /qskrps-47/obj3?versionId=null 2023-05-24T21:26:21.233 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:EfDj05RIKZnZKZs3GRWL+ye+OLQ= 2023-05-24T21:26:21.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:EfDj05RIKZnZKZs3GRWL+ye+OLQ=', 'Content-Length': '0'} 2023-05-24T21:26:21.236 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000873bbb202807cd81-00646e80fd-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:21.236 DEBUG:boto:path=/qskrps-47/obj3 2023-05-24T21:26:21.237 DEBUG:boto:auth_path=/qskrps-47/obj3 2023-05-24T21:26:21.237 DEBUG:boto:path=/qskrps-47/obj3?versionId=null 2023-05-24T21:26:21.238 DEBUG:boto:auth_path=/qskrps-47/obj3?versionId=null 2023-05-24T21:26:21.238 DEBUG:boto:Method: HEAD 2023-05-24T21:26:21.238 DEBUG:boto:Path: /qskrps-47/obj3?versionId=null 2023-05-24T21:26:21.239 DEBUG:boto:Data: 2023-05-24T21:26:21.239 DEBUG:boto:Headers: {} 2023-05-24T21:26:21.239 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:21.240 DEBUG:boto:Port: 8001 2023-05-24T21:26:21.240 DEBUG:boto:Params: {} 2023-05-24T21:26:21.241 DEBUG:boto:Token: None 2023-05-24T21:26:21.241 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:26:21 GMT /qskrps-47/obj3?versionId=null 2023-05-24T21:26:21.241 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:2sIz38ZOe0G69TGcmfKPF9VK/SY= 2023-05-24T21:26:21.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:2sIz38ZOe0G69TGcmfKPF9VK/SY=', 'Content-Length': '0'} 2023-05-24T21:26:21.244 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f4e5dbf46a037f83-00646e80fd-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:21.244 DEBUG:boto:path=/qskrps-47/obj3 2023-05-24T21:26:21.245 DEBUG:boto:auth_path=/qskrps-47/obj3 2023-05-24T21:26:21.245 DEBUG:boto:path=/qskrps-47/obj3?versionId=null 2023-05-24T21:26:21.245 DEBUG:boto:auth_path=/qskrps-47/obj3?versionId=null 2023-05-24T21:26:21.246 DEBUG:boto:Method: HEAD 2023-05-24T21:26:21.246 DEBUG:boto:Path: /qskrps-47/obj3?versionId=null 2023-05-24T21:26:21.246 DEBUG:boto:Data: 2023-05-24T21:26:21.247 DEBUG:boto:Headers: {} 2023-05-24T21:26:21.247 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:21.247 DEBUG:boto:Port: 8000 2023-05-24T21:26:21.248 DEBUG:boto:Params: {} 2023-05-24T21:26:21.248 DEBUG:boto:Token: None 2023-05-24T21:26:21.249 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:26:21 GMT /qskrps-47/obj3?versionId=null 2023-05-24T21:26:21.249 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:2sIz38ZOe0G69TGcmfKPF9VK/SY= 2023-05-24T21:26:21.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:2sIz38ZOe0G69TGcmfKPF9VK/SY=', 'Content-Length': '0'} 2023-05-24T21:26:21.251 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d9192c14a2e32e2-00646e80fd-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:21.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-24T21:26:21.252 DEBUG:boto:path=/qskrps-47/obj3 2023-05-24T21:26:21.252 DEBUG:boto:auth_path=/qskrps-47/obj3 2023-05-24T21:26:21.253 DEBUG:boto:path=/qskrps-47/obj3?versionId=null 2023-05-24T21:26:21.253 DEBUG:boto:auth_path=/qskrps-47/obj3?versionId=null 2023-05-24T21:26:21.253 DEBUG:boto:Method: GET 2023-05-24T21:26:21.254 DEBUG:boto:Path: /qskrps-47/obj3?versionId=null 2023-05-24T21:26:21.254 DEBUG:boto:Data: 2023-05-24T21:26:21.254 DEBUG:boto:Headers: {} 2023-05-24T21:26:21.255 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:21.255 DEBUG:boto:Port: 8001 2023-05-24T21:26:21.255 DEBUG:boto:Params: {} 2023-05-24T21:26:21.256 DEBUG:boto:Token: None 2023-05-24T21:26:21.256 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:21 GMT /qskrps-47/obj3?versionId=null 2023-05-24T21:26:21.257 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:EfDj05RIKZnZKZs3GRWL+ye+OLQ= 2023-05-24T21:26:21.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:EfDj05RIKZnZKZs3GRWL+ye+OLQ=', 'Content-Length': '0'} 2023-05-24T21:26:21.259 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fc45890d813f5b4d-00646e80fd-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:21.259 DEBUG:boto:path=/qskrps-47/obj3 2023-05-24T21:26:21.260 DEBUG:boto:auth_path=/qskrps-47/obj3 2023-05-24T21:26:21.260 DEBUG:boto:path=/qskrps-47/obj3?versionId=null 2023-05-24T21:26:21.260 DEBUG:boto:auth_path=/qskrps-47/obj3?versionId=null 2023-05-24T21:26:21.261 DEBUG:boto:Method: GET 2023-05-24T21:26:21.261 DEBUG:boto:Path: /qskrps-47/obj3?versionId=null 2023-05-24T21:26:21.262 DEBUG:boto:Data: 2023-05-24T21:26:21.262 DEBUG:boto:Headers: {} 2023-05-24T21:26:21.262 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:21.263 DEBUG:boto:Port: 8000 2023-05-24T21:26:21.263 DEBUG:boto:Params: {} 2023-05-24T21:26:21.263 DEBUG:boto:Token: None 2023-05-24T21:26:21.264 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:21 GMT /qskrps-47/obj3?versionId=null 2023-05-24T21:26:21.264 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:EfDj05RIKZnZKZs3GRWL+ye+OLQ= 2023-05-24T21:26:21.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:EfDj05RIKZnZKZs3GRWL+ye+OLQ=', 'Content-Length': '0'} 2023-05-24T21:26:21.266 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d627ea825499878-00646e80fd-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:21.267 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:26:21.267 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:26:21.268 DEBUG:boto:path=/qskrps-47/?prefix=obj3 2023-05-24T21:26:21.268 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj3 2023-05-24T21:26:21.268 DEBUG:boto:Method: GET 2023-05-24T21:26:21.269 DEBUG:boto:Path: /qskrps-47/?prefix=obj3 2023-05-24T21:26:21.269 DEBUG:boto:Data: 2023-05-24T21:26:21.269 DEBUG:boto:Headers: {} 2023-05-24T21:26:21.270 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:21.270 DEBUG:boto:Port: 8001 2023-05-24T21:26:21.271 DEBUG:boto:Params: {} 2023-05-24T21:26:21.271 DEBUG:boto:Token: None 2023-05-24T21:26:21.271 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:21 GMT /qskrps-47/ 2023-05-24T21:26:21.272 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:tQhrcjy8y3G2ohCCC0wseAtKmDI= 2023-05-24T21:26:21.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:tQhrcjy8y3G2ohCCC0wseAtKmDI=', 'Content-Length': '0'} 2023-05-24T21:26:21.274 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f97aa84be1ac218-00646e80fd-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:26:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:21.275 DEBUG:boto:b'qskrps-47obj31000falseobj32023-05-24T21:25:49.084Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:26:21.276 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:26:21.276 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:26:21.276 DEBUG:boto:path=/qskrps-47/?prefix=obj3 2023-05-24T21:26:21.277 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj3 2023-05-24T21:26:21.277 DEBUG:boto:Method: GET 2023-05-24T21:26:21.277 DEBUG:boto:Path: /qskrps-47/?prefix=obj3 2023-05-24T21:26:21.278 DEBUG:boto:Data: 2023-05-24T21:26:21.278 DEBUG:boto:Headers: {} 2023-05-24T21:26:21.279 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:21.279 DEBUG:boto:Port: 8000 2023-05-24T21:26:21.279 DEBUG:boto:Params: {} 2023-05-24T21:26:21.280 DEBUG:boto:Token: None 2023-05-24T21:26:21.280 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:21 GMT /qskrps-47/ 2023-05-24T21:26:21.281 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:tQhrcjy8y3G2ohCCC0wseAtKmDI= 2023-05-24T21:26:21.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:tQhrcjy8y3G2ohCCC0wseAtKmDI=', 'Content-Length': '0'} 2023-05-24T21:26:21.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e82f9ce17b1a49c-00646e80fd-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:26:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:21.284 DEBUG:boto:b'qskrps-47obj31000falseobj32023-05-24T21:25:49.084Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:26:21.284 DEBUG:boto:path=/qskrps-47/obj3 2023-05-24T21:26:21.285 DEBUG:boto:auth_path=/qskrps-47/obj3 2023-05-24T21:26:21.285 DEBUG:boto:Method: HEAD 2023-05-24T21:26:21.285 DEBUG:boto:Path: /qskrps-47/obj3 2023-05-24T21:26:21.286 DEBUG:boto:Data: 2023-05-24T21:26:21.286 DEBUG:boto:Headers: {} 2023-05-24T21:26:21.286 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:21.287 DEBUG:boto:Port: 8001 2023-05-24T21:26:21.287 DEBUG:boto:Params: {} 2023-05-24T21:26:21.287 DEBUG:boto:Token: None 2023-05-24T21:26:21.288 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:26:21 GMT /qskrps-47/obj3 2023-05-24T21:26:21.288 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:J4ci/4LhgHKYKGGVUg7gE+xBEjY= 2023-05-24T21:26:21.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:J4ci/4LhgHKYKGGVUg7gE+xBEjY=', 'Content-Length': '0'} 2023-05-24T21:26:21.290 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008367e1c62491e790-00646e80fd-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:21.291 DEBUG:boto:path=/qskrps-47/obj3 2023-05-24T21:26:21.291 DEBUG:boto:auth_path=/qskrps-47/obj3 2023-05-24T21:26:21.291 DEBUG:boto:Method: HEAD 2023-05-24T21:26:21.292 DEBUG:boto:Path: /qskrps-47/obj3 2023-05-24T21:26:21.292 DEBUG:boto:Data: 2023-05-24T21:26:21.292 DEBUG:boto:Headers: {} 2023-05-24T21:26:21.293 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:21.293 DEBUG:boto:Port: 8000 2023-05-24T21:26:21.293 DEBUG:boto:Params: {} 2023-05-24T21:26:21.294 DEBUG:boto:Token: None 2023-05-24T21:26:21.294 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:26:21 GMT /qskrps-47/obj3 2023-05-24T21:26:21.295 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:J4ci/4LhgHKYKGGVUg7gE+xBEjY= 2023-05-24T21:26:21.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:J4ci/4LhgHKYKGGVUg7gE+xBEjY=', 'Content-Length': '0'} 2023-05-24T21:26:21.297 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003781b08ee8d6f291-00646e80fd-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:21.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-24T21:26:21.298 DEBUG:boto:path=/qskrps-47/obj3 2023-05-24T21:26:21.298 DEBUG:boto:auth_path=/qskrps-47/obj3 2023-05-24T21:26:21.298 DEBUG:boto:Method: GET 2023-05-24T21:26:21.299 DEBUG:boto:Path: /qskrps-47/obj3 2023-05-24T21:26:21.299 DEBUG:boto:Data: 2023-05-24T21:26:21.300 DEBUG:boto:Headers: {} 2023-05-24T21:26:21.300 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:21.300 DEBUG:boto:Port: 8001 2023-05-24T21:26:21.301 DEBUG:boto:Params: {} 2023-05-24T21:26:21.301 DEBUG:boto:Token: None 2023-05-24T21:26:21.302 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:21 GMT /qskrps-47/obj3 2023-05-24T21:26:21.302 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:wVCyfwsUNCqd+QXlWXKt1c7IOXU= 2023-05-24T21:26:21.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wVCyfwsUNCqd+QXlWXKt1c7IOXU=', 'Content-Length': '0'} 2023-05-24T21:26:21.304 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000078db1d330f4a963-00646e80fd-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:21.305 DEBUG:boto:path=/qskrps-47/obj3 2023-05-24T21:26:21.305 DEBUG:boto:auth_path=/qskrps-47/obj3 2023-05-24T21:26:21.305 DEBUG:boto:Method: GET 2023-05-24T21:26:21.306 DEBUG:boto:Path: /qskrps-47/obj3 2023-05-24T21:26:21.306 DEBUG:boto:Data: 2023-05-24T21:26:21.307 DEBUG:boto:Headers: {} 2023-05-24T21:26:21.307 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:21.307 DEBUG:boto:Port: 8000 2023-05-24T21:26:21.308 DEBUG:boto:Params: {} 2023-05-24T21:26:21.308 DEBUG:boto:Token: None 2023-05-24T21:26:21.308 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:21 GMT /qskrps-47/obj3 2023-05-24T21:26:21.309 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:wVCyfwsUNCqd+QXlWXKt1c7IOXU= 2023-05-24T21:26:21.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wVCyfwsUNCqd+QXlWXKt1c7IOXU=', 'Content-Length': '0'} 2023-05-24T21:26:21.311 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006c465b754d25061f-00646e80fd-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:21.312 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:26:21.312 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:26:21.313 DEBUG:boto:path=/qskrps-47/?prefix=obj3 2023-05-24T21:26:21.313 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj3 2023-05-24T21:26:21.313 DEBUG:boto:Method: GET 2023-05-24T21:26:21.314 DEBUG:boto:Path: /qskrps-47/?prefix=obj3 2023-05-24T21:26:21.314 DEBUG:boto:Data: 2023-05-24T21:26:21.314 DEBUG:boto:Headers: {} 2023-05-24T21:26:21.315 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:21.315 DEBUG:boto:Port: 8001 2023-05-24T21:26:21.315 DEBUG:boto:Params: {} 2023-05-24T21:26:21.316 DEBUG:boto:Token: None 2023-05-24T21:26:21.316 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:21 GMT /qskrps-47/ 2023-05-24T21:26:21.317 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:tQhrcjy8y3G2ohCCC0wseAtKmDI= 2023-05-24T21:26:21.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:tQhrcjy8y3G2ohCCC0wseAtKmDI=', 'Content-Length': '0'} 2023-05-24T21:26:21.319 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a78d4c1dbcf1fa8-00646e80fd-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:26:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:21.320 DEBUG:boto:b'qskrps-47obj31000falseobj32023-05-24T21:25:49.084Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:26:21.320 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:26:21.321 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:26:21.321 DEBUG:boto:path=/qskrps-47/?prefix=obj3 2023-05-24T21:26:21.321 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj3 2023-05-24T21:26:21.322 DEBUG:boto:Method: GET 2023-05-24T21:26:21.322 DEBUG:boto:Path: /qskrps-47/?prefix=obj3 2023-05-24T21:26:21.323 DEBUG:boto:Data: 2023-05-24T21:26:21.323 DEBUG:boto:Headers: {} 2023-05-24T21:26:21.323 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:21.324 DEBUG:boto:Port: 8000 2023-05-24T21:26:21.324 DEBUG:boto:Params: {} 2023-05-24T21:26:21.324 DEBUG:boto:Token: None 2023-05-24T21:26:21.325 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:21 GMT /qskrps-47/ 2023-05-24T21:26:21.325 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:tQhrcjy8y3G2ohCCC0wseAtKmDI= 2023-05-24T21:26:21.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:tQhrcjy8y3G2ohCCC0wseAtKmDI=', 'Content-Length': '0'} 2023-05-24T21:26:21.328 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a81bcf66532ddf1b-00646e80fd-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:26:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:21.328 DEBUG:boto:b'qskrps-47obj31000falseobj32023-05-24T21:25:49.084Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:26:21.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-24T21:26:21.329 DEBUG:boto:path=/qskrps-47/obj4 2023-05-24T21:26:21.330 DEBUG:boto:auth_path=/qskrps-47/obj4 2023-05-24T21:26:21.330 DEBUG:boto:path=/qskrps-47/obj4?versionId=null 2023-05-24T21:26:21.331 DEBUG:boto:auth_path=/qskrps-47/obj4?versionId=null 2023-05-24T21:26:21.331 DEBUG:boto:Method: GET 2023-05-24T21:26:21.331 DEBUG:boto:Path: /qskrps-47/obj4?versionId=null 2023-05-24T21:26:21.332 DEBUG:boto:Data: 2023-05-24T21:26:21.332 DEBUG:boto:Headers: {} 2023-05-24T21:26:21.332 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:21.333 DEBUG:boto:Port: 8001 2023-05-24T21:26:21.333 DEBUG:boto:Params: {} 2023-05-24T21:26:21.334 DEBUG:boto:Token: None 2023-05-24T21:26:21.334 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:21 GMT /qskrps-47/obj4?versionId=null 2023-05-24T21:26:21.334 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:8R9YnC5pQvl0FknfllnBBg71qUU= 2023-05-24T21:26:21.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:8R9YnC5pQvl0FknfllnBBg71qUU=', 'Content-Length': '0'} 2023-05-24T21:26:21.336 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c5a15fc7b4ac215f-00646e80fd-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:21.337 DEBUG:boto:path=/qskrps-47/obj4 2023-05-24T21:26:21.337 DEBUG:boto:auth_path=/qskrps-47/obj4 2023-05-24T21:26:21.338 DEBUG:boto:path=/qskrps-47/obj4?versionId=null 2023-05-24T21:26:21.338 DEBUG:boto:auth_path=/qskrps-47/obj4?versionId=null 2023-05-24T21:26:21.338 DEBUG:boto:Method: GET 2023-05-24T21:26:21.339 DEBUG:boto:Path: /qskrps-47/obj4?versionId=null 2023-05-24T21:26:21.339 DEBUG:boto:Data: 2023-05-24T21:26:21.340 DEBUG:boto:Headers: {} 2023-05-24T21:26:21.340 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:21.340 DEBUG:boto:Port: 8000 2023-05-24T21:26:21.341 DEBUG:boto:Params: {} 2023-05-24T21:26:21.341 DEBUG:boto:Token: None 2023-05-24T21:26:21.341 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:21 GMT /qskrps-47/obj4?versionId=null 2023-05-24T21:26:21.342 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:8R9YnC5pQvl0FknfllnBBg71qUU= 2023-05-24T21:26:21.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:8R9YnC5pQvl0FknfllnBBg71qUU=', 'Content-Length': '0'} 2023-05-24T21:26:21.344 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002e34af08d0cce64b-00646e80fd-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:21.345 DEBUG:boto:path=/qskrps-47/obj4 2023-05-24T21:26:21.345 DEBUG:boto:auth_path=/qskrps-47/obj4 2023-05-24T21:26:21.345 DEBUG:boto:path=/qskrps-47/obj4?versionId=null 2023-05-24T21:26:21.346 DEBUG:boto:auth_path=/qskrps-47/obj4?versionId=null 2023-05-24T21:26:21.346 DEBUG:boto:Method: HEAD 2023-05-24T21:26:21.346 DEBUG:boto:Path: /qskrps-47/obj4?versionId=null 2023-05-24T21:26:21.347 DEBUG:boto:Data: 2023-05-24T21:26:21.347 DEBUG:boto:Headers: {} 2023-05-24T21:26:21.348 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:21.348 DEBUG:boto:Port: 8001 2023-05-24T21:26:21.348 DEBUG:boto:Params: {} 2023-05-24T21:26:21.349 DEBUG:boto:Token: None 2023-05-24T21:26:21.349 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:26:21 GMT /qskrps-47/obj4?versionId=null 2023-05-24T21:26:21.349 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:MQmnw67ycyVrw48EaX/nplyOXQA= 2023-05-24T21:26:21.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:MQmnw67ycyVrw48EaX/nplyOXQA=', 'Content-Length': '0'} 2023-05-24T21:26:21.351 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003dda794d2d3a7ba5-00646e80fd-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:21.352 DEBUG:boto:path=/qskrps-47/obj4 2023-05-24T21:26:21.352 DEBUG:boto:auth_path=/qskrps-47/obj4 2023-05-24T21:26:21.353 DEBUG:boto:path=/qskrps-47/obj4?versionId=null 2023-05-24T21:26:21.353 DEBUG:boto:auth_path=/qskrps-47/obj4?versionId=null 2023-05-24T21:26:21.354 DEBUG:boto:Method: HEAD 2023-05-24T21:26:21.354 DEBUG:boto:Path: /qskrps-47/obj4?versionId=null 2023-05-24T21:26:21.354 DEBUG:boto:Data: 2023-05-24T21:26:21.355 DEBUG:boto:Headers: {} 2023-05-24T21:26:21.355 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:21.355 DEBUG:boto:Port: 8000 2023-05-24T21:26:21.356 DEBUG:boto:Params: {} 2023-05-24T21:26:21.356 DEBUG:boto:Token: None 2023-05-24T21:26:21.356 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:26:21 GMT /qskrps-47/obj4?versionId=null 2023-05-24T21:26:21.357 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:MQmnw67ycyVrw48EaX/nplyOXQA= 2023-05-24T21:26:21.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:MQmnw67ycyVrw48EaX/nplyOXQA=', 'Content-Length': '0'} 2023-05-24T21:26:21.359 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001cd12818dbfac16f-00646e80fd-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:21.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-24T21:26:21.360 DEBUG:boto:path=/qskrps-47/obj4 2023-05-24T21:26:21.360 DEBUG:boto:auth_path=/qskrps-47/obj4 2023-05-24T21:26:21.360 DEBUG:boto:path=/qskrps-47/obj4?versionId=null 2023-05-24T21:26:21.361 DEBUG:boto:auth_path=/qskrps-47/obj4?versionId=null 2023-05-24T21:26:21.361 DEBUG:boto:Method: GET 2023-05-24T21:26:21.362 DEBUG:boto:Path: /qskrps-47/obj4?versionId=null 2023-05-24T21:26:21.362 DEBUG:boto:Data: 2023-05-24T21:26:21.362 DEBUG:boto:Headers: {} 2023-05-24T21:26:21.363 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:21.363 DEBUG:boto:Port: 8001 2023-05-24T21:26:21.363 DEBUG:boto:Params: {} 2023-05-24T21:26:21.364 DEBUG:boto:Token: None 2023-05-24T21:26:21.364 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:21 GMT /qskrps-47/obj4?versionId=null 2023-05-24T21:26:21.365 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:8R9YnC5pQvl0FknfllnBBg71qUU= 2023-05-24T21:26:21.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:8R9YnC5pQvl0FknfllnBBg71qUU=', 'Content-Length': '0'} 2023-05-24T21:26:21.367 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b4fffca1505502c-00646e80fd-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:21.408 DEBUG:boto:path=/qskrps-47/obj4 2023-05-24T21:26:21.409 DEBUG:boto:auth_path=/qskrps-47/obj4 2023-05-24T21:26:21.409 DEBUG:boto:path=/qskrps-47/obj4?versionId=null 2023-05-24T21:26:21.409 DEBUG:boto:auth_path=/qskrps-47/obj4?versionId=null 2023-05-24T21:26:21.410 DEBUG:boto:Method: GET 2023-05-24T21:26:21.410 DEBUG:boto:Path: /qskrps-47/obj4?versionId=null 2023-05-24T21:26:21.410 DEBUG:boto:Data: 2023-05-24T21:26:21.411 DEBUG:boto:Headers: {} 2023-05-24T21:26:21.411 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:21.412 DEBUG:boto:Port: 8000 2023-05-24T21:26:21.412 DEBUG:boto:Params: {} 2023-05-24T21:26:21.412 DEBUG:boto:Token: None 2023-05-24T21:26:21.413 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:21 GMT /qskrps-47/obj4?versionId=null 2023-05-24T21:26:21.413 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:8R9YnC5pQvl0FknfllnBBg71qUU= 2023-05-24T21:26:21.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:8R9YnC5pQvl0FknfllnBBg71qUU=', 'Content-Length': '0'} 2023-05-24T21:26:21.415 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ddf4c715eaa54284-00646e80fd-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:21.416 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:26:21.416 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:26:21.417 DEBUG:boto:path=/qskrps-47/?prefix=obj4 2023-05-24T21:26:21.417 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj4 2023-05-24T21:26:21.417 DEBUG:boto:Method: GET 2023-05-24T21:26:21.418 DEBUG:boto:Path: /qskrps-47/?prefix=obj4 2023-05-24T21:26:21.418 DEBUG:boto:Data: 2023-05-24T21:26:21.419 DEBUG:boto:Headers: {} 2023-05-24T21:26:21.419 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:21.419 DEBUG:boto:Port: 8001 2023-05-24T21:26:21.420 DEBUG:boto:Params: {} 2023-05-24T21:26:21.420 DEBUG:boto:Token: None 2023-05-24T21:26:21.420 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:21 GMT /qskrps-47/ 2023-05-24T21:26:21.421 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:tQhrcjy8y3G2ohCCC0wseAtKmDI= 2023-05-24T21:26:21.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:tQhrcjy8y3G2ohCCC0wseAtKmDI=', 'Content-Length': '0'} 2023-05-24T21:26:21.423 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000183cde082736e278-00646e80fd-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:26:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:21.424 DEBUG:boto:b'qskrps-47obj41000falseobj42023-05-24T21:25:49.101Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:26:21.425 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:26:21.425 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:26:21.425 DEBUG:boto:path=/qskrps-47/?prefix=obj4 2023-05-24T21:26:21.426 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj4 2023-05-24T21:26:21.426 DEBUG:boto:Method: GET 2023-05-24T21:26:21.426 DEBUG:boto:Path: /qskrps-47/?prefix=obj4 2023-05-24T21:26:21.427 DEBUG:boto:Data: 2023-05-24T21:26:21.427 DEBUG:boto:Headers: {} 2023-05-24T21:26:21.428 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:21.428 DEBUG:boto:Port: 8000 2023-05-24T21:26:21.428 DEBUG:boto:Params: {} 2023-05-24T21:26:21.429 DEBUG:boto:Token: None 2023-05-24T21:26:21.429 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:21 GMT /qskrps-47/ 2023-05-24T21:26:21.430 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:tQhrcjy8y3G2ohCCC0wseAtKmDI= 2023-05-24T21:26:21.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:tQhrcjy8y3G2ohCCC0wseAtKmDI=', 'Content-Length': '0'} 2023-05-24T21:26:21.432 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006be68fd32c4042e9-00646e80fd-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:26:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:21.433 DEBUG:boto:b'qskrps-47obj41000falseobj42023-05-24T21:25:49.101Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:26:21.433 DEBUG:boto:path=/qskrps-47/obj4 2023-05-24T21:26:21.434 DEBUG:boto:auth_path=/qskrps-47/obj4 2023-05-24T21:26:21.434 DEBUG:boto:Method: HEAD 2023-05-24T21:26:21.435 DEBUG:boto:Path: /qskrps-47/obj4 2023-05-24T21:26:21.435 DEBUG:boto:Data: 2023-05-24T21:26:21.435 DEBUG:boto:Headers: {} 2023-05-24T21:26:21.436 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:21.436 DEBUG:boto:Port: 8001 2023-05-24T21:26:21.436 DEBUG:boto:Params: {} 2023-05-24T21:26:21.437 DEBUG:boto:Token: None 2023-05-24T21:26:21.437 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:26:21 GMT /qskrps-47/obj4 2023-05-24T21:26:21.437 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:79QtacAdMPwpsbLu0bSAeqrZxPk= 2023-05-24T21:26:21.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:79QtacAdMPwpsbLu0bSAeqrZxPk=', 'Content-Length': '0'} 2023-05-24T21:26:21.439 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008da5987f2ee96e8b-00646e80fd-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:21.440 DEBUG:boto:path=/qskrps-47/obj4 2023-05-24T21:26:21.440 DEBUG:boto:auth_path=/qskrps-47/obj4 2023-05-24T21:26:21.441 DEBUG:boto:Method: HEAD 2023-05-24T21:26:21.441 DEBUG:boto:Path: /qskrps-47/obj4 2023-05-24T21:26:21.441 DEBUG:boto:Data: 2023-05-24T21:26:21.442 DEBUG:boto:Headers: {} 2023-05-24T21:26:21.442 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:21.442 DEBUG:boto:Port: 8000 2023-05-24T21:26:21.443 DEBUG:boto:Params: {} 2023-05-24T21:26:21.443 DEBUG:boto:Token: None 2023-05-24T21:26:21.443 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:26:21 GMT /qskrps-47/obj4 2023-05-24T21:26:21.444 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:79QtacAdMPwpsbLu0bSAeqrZxPk= 2023-05-24T21:26:21.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:79QtacAdMPwpsbLu0bSAeqrZxPk=', 'Content-Length': '0'} 2023-05-24T21:26:21.446 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000006ce56548dd27a8e-00646e80fd-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:21.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-24T21:26:21.447 DEBUG:boto:path=/qskrps-47/obj4 2023-05-24T21:26:21.447 DEBUG:boto:auth_path=/qskrps-47/obj4 2023-05-24T21:26:21.448 DEBUG:boto:Method: GET 2023-05-24T21:26:21.448 DEBUG:boto:Path: /qskrps-47/obj4 2023-05-24T21:26:21.449 DEBUG:boto:Data: 2023-05-24T21:26:21.449 DEBUG:boto:Headers: {} 2023-05-24T21:26:21.449 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:21.450 DEBUG:boto:Port: 8001 2023-05-24T21:26:21.450 DEBUG:boto:Params: {} 2023-05-24T21:26:21.450 DEBUG:boto:Token: None 2023-05-24T21:26:21.451 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:21 GMT /qskrps-47/obj4 2023-05-24T21:26:21.451 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:gl9APQFWj1NQERicZIuGD0OC7mM= 2023-05-24T21:26:21.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:gl9APQFWj1NQERicZIuGD0OC7mM=', 'Content-Length': '0'} 2023-05-24T21:26:21.453 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009cc0bf2df18e39a0-00646e80fd-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:21.454 DEBUG:boto:path=/qskrps-47/obj4 2023-05-24T21:26:21.454 DEBUG:boto:auth_path=/qskrps-47/obj4 2023-05-24T21:26:21.455 DEBUG:boto:Method: GET 2023-05-24T21:26:21.455 DEBUG:boto:Path: /qskrps-47/obj4 2023-05-24T21:26:21.456 DEBUG:boto:Data: 2023-05-24T21:26:21.456 DEBUG:boto:Headers: {} 2023-05-24T21:26:21.456 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:21.457 DEBUG:boto:Port: 8000 2023-05-24T21:26:21.457 DEBUG:boto:Params: {} 2023-05-24T21:26:21.457 DEBUG:boto:Token: None 2023-05-24T21:26:21.458 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:21 GMT /qskrps-47/obj4 2023-05-24T21:26:21.458 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:gl9APQFWj1NQERicZIuGD0OC7mM= 2023-05-24T21:26:21.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:gl9APQFWj1NQERicZIuGD0OC7mM=', 'Content-Length': '0'} 2023-05-24T21:26:21.460 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008ed51debd7ecc845-00646e80fd-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:26:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:21.461 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:26:21.461 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:26:21.462 DEBUG:boto:path=/qskrps-47/?prefix=obj4 2023-05-24T21:26:21.462 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj4 2023-05-24T21:26:21.462 DEBUG:boto:Method: GET 2023-05-24T21:26:21.463 DEBUG:boto:Path: /qskrps-47/?prefix=obj4 2023-05-24T21:26:21.463 DEBUG:boto:Data: 2023-05-24T21:26:21.463 DEBUG:boto:Headers: {} 2023-05-24T21:26:21.464 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:21.464 DEBUG:boto:Port: 8001 2023-05-24T21:26:21.465 DEBUG:boto:Params: {} 2023-05-24T21:26:21.465 DEBUG:boto:Token: None 2023-05-24T21:26:21.465 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:21 GMT /qskrps-47/ 2023-05-24T21:26:21.466 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:tQhrcjy8y3G2ohCCC0wseAtKmDI= 2023-05-24T21:26:21.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:tQhrcjy8y3G2ohCCC0wseAtKmDI=', 'Content-Length': '0'} 2023-05-24T21:26:21.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fac745db765d5751-00646e80fd-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:26:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:21.469 DEBUG:boto:b'qskrps-47obj41000falseobj42023-05-24T21:25:49.101Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:26:21.469 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:26:21.470 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:26:21.470 DEBUG:boto:path=/qskrps-47/?prefix=obj4 2023-05-24T21:26:21.471 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj4 2023-05-24T21:26:21.471 DEBUG:boto:Method: GET 2023-05-24T21:26:21.471 DEBUG:boto:Path: /qskrps-47/?prefix=obj4 2023-05-24T21:26:21.472 DEBUG:boto:Data: 2023-05-24T21:26:21.472 DEBUG:boto:Headers: {} 2023-05-24T21:26:21.472 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:21.473 DEBUG:boto:Port: 8000 2023-05-24T21:26:21.473 DEBUG:boto:Params: {} 2023-05-24T21:26:21.474 DEBUG:boto:Token: None 2023-05-24T21:26:21.474 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:26:21 GMT /qskrps-47/ 2023-05-24T21:26:21.474 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:tQhrcjy8y3G2ohCCC0wseAtKmDI= 2023-05-24T21:26:21.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:tQhrcjy8y3G2ohCCC0wseAtKmDI=', 'Content-Length': '0'} 2023-05-24T21:26:21.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0cafff7fbadaed0-00646e80fd-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:26:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:21.477 DEBUG:boto:b'qskrps-47obj41000falseobj42023-05-24T21:25:49.101Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:26:21.478 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qskrps-47 zones={a2, a1} 2023-05-24T21:26:21.478 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'qskrps-46', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:26:21.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'qskrps-46', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:26:21.479 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket qskrps-46 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-24T21:26:21.529 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:26:21.529 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:26:21.624 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'qskrps-46', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:26:21.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'qskrps-46', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:26:21.625 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket qskrps-46 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-24T21:26:21.675 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:26:21.676 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:26:21.760 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'qskrps-47', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:26:21.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'qskrps-47', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:26:21.761 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket qskrps-47 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-24T21:26:21.816 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:26:21.816 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:26:21.915 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'qskrps-47', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:26:21.915 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'qskrps-47', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:26:21.916 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket qskrps-47 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-24T21:26:21.966 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:26:21.966 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:26:22.055 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:26:22.056 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:26:22.056 DEBUG:boto:Method: HEAD 2023-05-24T21:26:22.057 DEBUG:boto:Path: /qskrps-46/ 2023-05-24T21:26:22.057 DEBUG:boto:Data: 2023-05-24T21:26:22.057 DEBUG:boto:Headers: {} 2023-05-24T21:26:22.058 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:22.058 DEBUG:boto:Port: 8000 2023-05-24T21:26:22.058 DEBUG:boto:Params: {} 2023-05-24T21:26:22.059 DEBUG:boto:Token: None 2023-05-24T21:26:22.059 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:26:22 GMT /qskrps-46/ 2023-05-24T21:26:22.060 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:+tOgSSk2UtNle2dnzr6/eQMgahE= 2023-05-24T21:26:22.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:22 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:+tOgSSk2UtNle2dnzr6/eQMgahE=', 'Content-Length': '0'} 2023-05-24T21:26:22.062 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000054e5b93c9832deaa-00646e80fe-1627-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:26:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:22.063 DEBUG:boto:path=/qskrps-46/obj5 2023-05-24T21:26:22.063 DEBUG:boto:auth_path=/qskrps-46/obj5 2023-05-24T21:26:22.064 DEBUG:boto:Method: PUT 2023-05-24T21:26:22.064 DEBUG:boto:Path: /qskrps-46/obj5 2023-05-24T21:26:22.064 DEBUG:boto:Data: 2023-05-24T21:26:22.065 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-24T21:26:22.065 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:22.065 DEBUG:boto:Port: 8000 2023-05-24T21:26:22.066 DEBUG:boto:Params: {} 2023-05-24T21:26:22.066 DEBUG:boto:Token: None 2023-05-24T21:26:22.066 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 24 May 2023 21:26:22 GMT /qskrps-46/obj5 2023-05-24T21:26:22.067 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:YnFdJEUKZscyxobU4eDwE60jLvg= 2023-05-24T21:26:22.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:26:22 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:YnFdJEUKZscyxobU4eDwE60jLvg='} 2023-05-24T21:26:22.073 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b3d79cedd1a3ec75-00646e80fe-1627-a1'), ('Date', 'Wed, 24 May 2023 21:26:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:22.073 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:26:22.073 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:26:22.074 DEBUG:boto:Method: HEAD 2023-05-24T21:26:22.074 DEBUG:boto:Path: /qskrps-46/ 2023-05-24T21:26:22.074 DEBUG:boto:Data: 2023-05-24T21:26:22.075 DEBUG:boto:Headers: {} 2023-05-24T21:26:22.075 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:22.075 DEBUG:boto:Port: 8000 2023-05-24T21:26:22.076 DEBUG:boto:Params: {} 2023-05-24T21:26:22.076 DEBUG:boto:Token: None 2023-05-24T21:26:22.077 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:26:22 GMT /qskrps-46/ 2023-05-24T21:26:22.077 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:+tOgSSk2UtNle2dnzr6/eQMgahE= 2023-05-24T21:26:22.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:22 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:+tOgSSk2UtNle2dnzr6/eQMgahE=', 'Content-Length': '0'} 2023-05-24T21:26:22.079 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd95290da122d7b3-00646e80fe-1627-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:26:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:22.080 DEBUG:boto:path=/qskrps-46/obj6 2023-05-24T21:26:22.080 DEBUG:boto:auth_path=/qskrps-46/obj6 2023-05-24T21:26:22.081 DEBUG:boto:Method: PUT 2023-05-24T21:26:22.081 DEBUG:boto:Path: /qskrps-46/obj6 2023-05-24T21:26:22.081 DEBUG:boto:Data: 2023-05-24T21:26:22.082 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-24T21:26:22.082 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:22.082 DEBUG:boto:Port: 8000 2023-05-24T21:26:22.083 DEBUG:boto:Params: {} 2023-05-24T21:26:22.083 DEBUG:boto:Token: None 2023-05-24T21:26:22.083 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 24 May 2023 21:26:22 GMT /qskrps-46/obj6 2023-05-24T21:26:22.084 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:cDA663xu4WMxedrE7+edjexEo8M= 2023-05-24T21:26:22.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:26:22 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:cDA663xu4WMxedrE7+edjexEo8M='} 2023-05-24T21:26:22.128 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f83d8043ba6db06c-00646e80fe-1627-a1'), ('Date', 'Wed, 24 May 2023 21:26:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:22.129 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:26:22.129 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:26:22.129 DEBUG:boto:Method: HEAD 2023-05-24T21:26:22.130 DEBUG:boto:Path: /qskrps-46/ 2023-05-24T21:26:22.130 DEBUG:boto:Data: 2023-05-24T21:26:22.131 DEBUG:boto:Headers: {} 2023-05-24T21:26:22.131 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:22.131 DEBUG:boto:Port: 8000 2023-05-24T21:26:22.132 DEBUG:boto:Params: {} 2023-05-24T21:26:22.132 DEBUG:boto:Token: None 2023-05-24T21:26:22.132 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:26:22 GMT /qskrps-46/ 2023-05-24T21:26:22.133 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:+tOgSSk2UtNle2dnzr6/eQMgahE= 2023-05-24T21:26:22.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:22 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:+tOgSSk2UtNle2dnzr6/eQMgahE=', 'Content-Length': '0'} 2023-05-24T21:26:22.135 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f01938f0cf719114-00646e80fe-1627-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:26:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:22.136 DEBUG:boto:path=/qskrps-46/obj7 2023-05-24T21:26:22.136 DEBUG:boto:auth_path=/qskrps-46/obj7 2023-05-24T21:26:22.136 DEBUG:boto:Method: PUT 2023-05-24T21:26:22.137 DEBUG:boto:Path: /qskrps-46/obj7 2023-05-24T21:26:22.137 DEBUG:boto:Data: 2023-05-24T21:26:22.137 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-24T21:26:22.138 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:22.138 DEBUG:boto:Port: 8000 2023-05-24T21:26:22.138 DEBUG:boto:Params: {} 2023-05-24T21:26:22.139 DEBUG:boto:Token: None 2023-05-24T21:26:22.139 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 24 May 2023 21:26:22 GMT /qskrps-46/obj7 2023-05-24T21:26:22.140 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:iSttaFycsqZX0oAWVeiJo++zC50= 2023-05-24T21:26:22.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:26:22 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:iSttaFycsqZX0oAWVeiJo++zC50='} 2023-05-24T21:26:22.145 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000435f209b27300981-00646e80fe-1627-a1'), ('Date', 'Wed, 24 May 2023 21:26:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:22.146 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:26:22.146 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:26:22.146 DEBUG:boto:Method: HEAD 2023-05-24T21:26:22.147 DEBUG:boto:Path: /qskrps-46/ 2023-05-24T21:26:22.147 DEBUG:boto:Data: 2023-05-24T21:26:22.147 DEBUG:boto:Headers: {} 2023-05-24T21:26:22.148 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:22.148 DEBUG:boto:Port: 8000 2023-05-24T21:26:22.148 DEBUG:boto:Params: {} 2023-05-24T21:26:22.149 DEBUG:boto:Token: None 2023-05-24T21:26:22.149 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:26:22 GMT /qskrps-46/ 2023-05-24T21:26:22.149 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:+tOgSSk2UtNle2dnzr6/eQMgahE= 2023-05-24T21:26:22.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:22 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:+tOgSSk2UtNle2dnzr6/eQMgahE=', 'Content-Length': '0'} 2023-05-24T21:26:22.152 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000988100ee18c676c4-00646e80fe-1627-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:26:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:22.152 DEBUG:boto:path=/qskrps-46/obj8 2023-05-24T21:26:22.153 DEBUG:boto:auth_path=/qskrps-46/obj8 2023-05-24T21:26:22.153 DEBUG:boto:Method: PUT 2023-05-24T21:26:22.153 DEBUG:boto:Path: /qskrps-46/obj8 2023-05-24T21:26:22.154 DEBUG:boto:Data: 2023-05-24T21:26:22.154 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-24T21:26:22.154 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:26:22.155 DEBUG:boto:Port: 8000 2023-05-24T21:26:22.155 DEBUG:boto:Params: {} 2023-05-24T21:26:22.155 DEBUG:boto:Token: None 2023-05-24T21:26:22.156 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 24 May 2023 21:26:22 GMT /qskrps-46/obj8 2023-05-24T21:26:22.156 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:CivRvY5ztyJWoi9l78Tym7izjww= 2023-05-24T21:26:22.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:26:22 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:CivRvY5ztyJWoi9l78Tym7izjww='} 2023-05-24T21:26:22.162 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000770e5348bec43077-00646e80fe-1627-a1'), ('Date', 'Wed, 24 May 2023 21:26:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:22.162 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:26:22.163 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:26:22.163 DEBUG:boto:Method: HEAD 2023-05-24T21:26:22.163 DEBUG:boto:Path: /qskrps-47/ 2023-05-24T21:26:22.164 DEBUG:boto:Data: 2023-05-24T21:26:22.164 DEBUG:boto:Headers: {} 2023-05-24T21:26:22.164 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:22.165 DEBUG:boto:Port: 8001 2023-05-24T21:26:22.165 DEBUG:boto:Params: {} 2023-05-24T21:26:22.165 DEBUG:boto:Token: None 2023-05-24T21:26:22.166 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:26:22 GMT /qskrps-47/ 2023-05-24T21:26:22.166 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:vdjvR/k39Qec7W9d07E62CkebGM= 2023-05-24T21:26:22.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:22 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:vdjvR/k39Qec7W9d07E62CkebGM=', 'Content-Length': '0'} 2023-05-24T21:26:22.168 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000df703f8c06fab9cb-00646e80fe-1625-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:26:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:22.169 DEBUG:boto:path=/qskrps-47/obj5 2023-05-24T21:26:22.169 DEBUG:boto:auth_path=/qskrps-47/obj5 2023-05-24T21:26:22.170 DEBUG:boto:Method: PUT 2023-05-24T21:26:22.170 DEBUG:boto:Path: /qskrps-47/obj5 2023-05-24T21:26:22.170 DEBUG:boto:Data: 2023-05-24T21:26:22.171 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-24T21:26:22.171 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:22.171 DEBUG:boto:Port: 8001 2023-05-24T21:26:22.172 DEBUG:boto:Params: {} 2023-05-24T21:26:22.172 DEBUG:boto:Token: None 2023-05-24T21:26:22.172 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 24 May 2023 21:26:22 GMT /qskrps-47/obj5 2023-05-24T21:26:22.173 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:kMefhdN18lu+kiwsVYS/vj0w0Jc= 2023-05-24T21:26:22.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:26:22 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:kMefhdN18lu+kiwsVYS/vj0w0Jc='} 2023-05-24T21:26:22.179 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000050bf874d599b1e53-00646e80fe-1625-a2'), ('Date', 'Wed, 24 May 2023 21:26:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:22.180 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:26:22.180 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:26:22.180 DEBUG:boto:Method: HEAD 2023-05-24T21:26:22.181 DEBUG:boto:Path: /qskrps-47/ 2023-05-24T21:26:22.181 DEBUG:boto:Data: 2023-05-24T21:26:22.181 DEBUG:boto:Headers: {} 2023-05-24T21:26:22.182 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:22.182 DEBUG:boto:Port: 8001 2023-05-24T21:26:22.182 DEBUG:boto:Params: {} 2023-05-24T21:26:22.183 DEBUG:boto:Token: None 2023-05-24T21:26:22.183 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:26:22 GMT /qskrps-47/ 2023-05-24T21:26:22.183 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:vdjvR/k39Qec7W9d07E62CkebGM= 2023-05-24T21:26:22.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:22 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:vdjvR/k39Qec7W9d07E62CkebGM=', 'Content-Length': '0'} 2023-05-24T21:26:22.186 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000040ab354a4dc59b8-00646e80fe-1625-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:26:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:22.186 DEBUG:boto:path=/qskrps-47/obj6 2023-05-24T21:26:22.187 DEBUG:boto:auth_path=/qskrps-47/obj6 2023-05-24T21:26:22.187 DEBUG:boto:Method: PUT 2023-05-24T21:26:22.187 DEBUG:boto:Path: /qskrps-47/obj6 2023-05-24T21:26:22.188 DEBUG:boto:Data: 2023-05-24T21:26:22.188 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-24T21:26:22.188 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:22.189 DEBUG:boto:Port: 8001 2023-05-24T21:26:22.189 DEBUG:boto:Params: {} 2023-05-24T21:26:22.189 DEBUG:boto:Token: None 2023-05-24T21:26:22.190 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 24 May 2023 21:26:22 GMT /qskrps-47/obj6 2023-05-24T21:26:22.190 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:twxFcO6CPX0Nz2ebCUw0lHlWKCo= 2023-05-24T21:26:22.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:26:22 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:twxFcO6CPX0Nz2ebCUw0lHlWKCo='} 2023-05-24T21:26:22.232 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d301ed82fb1c19cb-00646e80fe-1625-a2'), ('Date', 'Wed, 24 May 2023 21:26:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:22.233 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:26:22.233 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:26:22.233 DEBUG:boto:Method: HEAD 2023-05-24T21:26:22.234 DEBUG:boto:Path: /qskrps-47/ 2023-05-24T21:26:22.234 DEBUG:boto:Data: 2023-05-24T21:26:22.234 DEBUG:boto:Headers: {} 2023-05-24T21:26:22.235 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:22.235 DEBUG:boto:Port: 8001 2023-05-24T21:26:22.235 DEBUG:boto:Params: {} 2023-05-24T21:26:22.236 DEBUG:boto:Token: None 2023-05-24T21:26:22.236 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:26:22 GMT /qskrps-47/ 2023-05-24T21:26:22.237 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:vdjvR/k39Qec7W9d07E62CkebGM= 2023-05-24T21:26:22.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:22 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:vdjvR/k39Qec7W9d07E62CkebGM=', 'Content-Length': '0'} 2023-05-24T21:26:22.239 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000479c66debbdae7dd-00646e80fe-1625-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:26:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:22.239 DEBUG:boto:path=/qskrps-47/obj7 2023-05-24T21:26:22.240 DEBUG:boto:auth_path=/qskrps-47/obj7 2023-05-24T21:26:22.240 DEBUG:boto:Method: PUT 2023-05-24T21:26:22.241 DEBUG:boto:Path: /qskrps-47/obj7 2023-05-24T21:26:22.241 DEBUG:boto:Data: 2023-05-24T21:26:22.241 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-24T21:26:22.242 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:22.242 DEBUG:boto:Port: 8001 2023-05-24T21:26:22.242 DEBUG:boto:Params: {} 2023-05-24T21:26:22.243 DEBUG:boto:Token: None 2023-05-24T21:26:22.243 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 24 May 2023 21:26:22 GMT /qskrps-47/obj7 2023-05-24T21:26:22.243 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:XR8QnfkEETAEWXemyBSvfM7iWXo= 2023-05-24T21:26:22.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:26:22 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:XR8QnfkEETAEWXemyBSvfM7iWXo='} 2023-05-24T21:26:22.251 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c99f9e78f63f1947-00646e80fe-1625-a2'), ('Date', 'Wed, 24 May 2023 21:26:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:22.251 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:26:22.251 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:26:22.252 DEBUG:boto:Method: HEAD 2023-05-24T21:26:22.252 DEBUG:boto:Path: /qskrps-47/ 2023-05-24T21:26:22.253 DEBUG:boto:Data: 2023-05-24T21:26:22.253 DEBUG:boto:Headers: {} 2023-05-24T21:26:22.253 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:22.253 DEBUG:boto:Port: 8001 2023-05-24T21:26:22.254 DEBUG:boto:Params: {} 2023-05-24T21:26:22.254 DEBUG:boto:Token: None 2023-05-24T21:26:22.255 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:26:22 GMT /qskrps-47/ 2023-05-24T21:26:22.255 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:vdjvR/k39Qec7W9d07E62CkebGM= 2023-05-24T21:26:22.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:26:22 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:vdjvR/k39Qec7W9d07E62CkebGM=', 'Content-Length': '0'} 2023-05-24T21:26:22.257 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003a7b5fe889be71fd-00646e80fe-1625-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:26:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:22.258 DEBUG:boto:path=/qskrps-47/obj8 2023-05-24T21:26:22.258 DEBUG:boto:auth_path=/qskrps-47/obj8 2023-05-24T21:26:22.259 DEBUG:boto:Method: PUT 2023-05-24T21:26:22.259 DEBUG:boto:Path: /qskrps-47/obj8 2023-05-24T21:26:22.259 DEBUG:boto:Data: 2023-05-24T21:26:22.260 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-24T21:26:22.260 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:26:22.260 DEBUG:boto:Port: 8001 2023-05-24T21:26:22.261 DEBUG:boto:Params: {} 2023-05-24T21:26:22.261 DEBUG:boto:Token: None 2023-05-24T21:26:22.261 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 24 May 2023 21:26:22 GMT /qskrps-47/obj8 2023-05-24T21:26:22.262 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:f1L4FJR+ARndkqW1/7neLR28yB4= 2023-05-24T21:26:22.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:26:22 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:f1L4FJR+ARndkqW1/7neLR28yB4='} 2023-05-24T21:26:22.304 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bd9af34f994f14cd-00646e80fe-1625-a2'), ('Date', 'Wed, 24 May 2023 21:26:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:26:22.305 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=qskrps-46 2023-05-24T21:26:22.305 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-46', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:26:22.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-46', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:26:22.306 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-46 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:26:22.363 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:26:22.363 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:26:22.421 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:26:22.416+0000 7f8f9ceb4780 1 waiting for incremental sync to catch up: 2023-05-24T21:26:22.422 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [00000000001.23.6, , 00000000001.101.6, , 00000000001.14.6, , , 00000000001.19.6, , , ] 2023-05-24T21:26:22.422 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [00000000004.27.6, 00000000002.14.1, 00000000004.109.6, 00000000002.16.1, 00000000004.22.6, 00000000002.17.1, 00000000002.26.1, 00000000004.23.6, 00000000002.47.1, 00000000002.18.1, 00000000002.15.1] 2023-05-24T21:26:27.423 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:26:27.416+0000 7f8f9ceb4780 1 waiting for incremental sync to catch up: 2023-05-24T21:26:27.423 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [00000000001.23.6, , 00000000001.101.6, , 00000000001.14.6, , , 00000000001.19.6, , , ] 2023-05-24T21:26:27.424 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [00000000004.27.6, 00000000002.14.1, 00000000004.109.6, 00000000002.16.1, 00000000004.22.6, 00000000002.17.1, 00000000002.26.1, 00000000004.23.6, 00000000002.47.1, 00000000002.18.1, 00000000002.15.1] 2023-05-24T21:26:32.424 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:26:32.417+0000 7f8f9ceb4780 1 bucket sync caught up with source: 2023-05-24T21:26:32.425 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [00000000004.27.6, 00000000002.14.1, 00000000004.109.6, 00000000002.16.1, 00000000004.22.6, 00000000002.17.1, 00000000002.26.1, 00000000004.23.6, 00000000002.47.1, 00000000002.18.1, 00000000002.15.1] 2023-05-24T21:26:32.425 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [00000000004.27.6, 00000000002.14.1, 00000000004.109.6, 00000000002.16.1, 00000000004.22.6, 00000000002.17.1, 00000000002.26.1, 00000000004.23.6, 00000000002.47.1, 00000000002.18.1, 00000000002.15.1] 2023-05-24T21:26:32.425 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:26:32.417+0000 7f8f9ceb4780 0 bucket checkpoint complete 2023-05-24T21:26:32.440 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=qskrps-46 2023-05-24T21:26:32.441 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-46', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:26:32.442 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-46', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:26:32.442 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-46 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:26:32.502 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:26:32.502 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:26:32.540 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:26:32.534+0000 7fb2ecffd780 1 waiting for incremental sync to catch up: 2023-05-24T21:26:32.540 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [00000000001.386.6, , 00000000001.1699.6, , 00000000001.508.6, , , 00000000001.664.6, , , ] 2023-05-24T21:26:32.541 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [00000000002.388.6, , 00000000002.1701.6, , 00000000002.512.6, , , 00000000002.666.6, , , ] 2023-05-24T21:26:37.542 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:26:37.535+0000 7fb2ecffd780 1 waiting for incremental sync to catch up: 2023-05-24T21:26:37.542 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [00000000001.386.6, , 00000000001.1699.6, , 00000000001.508.6, , , 00000000001.664.6, , , ] 2023-05-24T21:26:37.542 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [00000000002.388.6, , 00000000002.1701.6, , 00000000002.512.6, , , 00000000002.666.6, , , ] 2023-05-24T21:26:42.543 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:26:42.536+0000 7fb2ecffd780 1 waiting for incremental sync to catch up: 2023-05-24T21:26:42.543 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [00000000001.386.6, , 00000000002.1701.6, , 00000000001.508.6, , , 00000000001.664.6, , , ] 2023-05-24T21:26:42.543 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [00000000002.388.6, , 00000000002.1701.6, , 00000000002.512.6, , , 00000000002.666.6, , , ] 2023-05-24T21:26:47.544 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:26:47.537+0000 7fb2ecffd780 1 waiting for incremental sync to catch up: 2023-05-24T21:26:47.544 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [00000000001.386.6, , 00000000002.1701.6, , 00000000001.508.6, , , 00000000001.664.6, , , ] 2023-05-24T21:26:47.544 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [00000000002.388.6, , 00000000002.1701.6, , 00000000002.512.6, , , 00000000002.666.6, , , ] 2023-05-24T21:26:52.545 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:26:52.540+0000 7fb2ecffd780 1 waiting for incremental sync to catch up: 2023-05-24T21:26:52.546 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [00000000002.388.6, , 00000000002.1701.6, , 00000000002.512.6, , , 00000000001.664.6, , , ] 2023-05-24T21:26:52.546 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [00000000002.388.6, , 00000000002.1701.6, , 00000000002.512.6, , , 00000000002.666.6, , , ] 2023-05-24T21:26:57.547 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:26:57.541+0000 7fb2ecffd780 1 waiting for incremental sync to catch up: 2023-05-24T21:26:57.547 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [00000000002.388.6, , 00000000002.1701.6, , 00000000002.512.6, , , 00000000001.664.6, , , ] 2023-05-24T21:26:57.547 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [00000000002.388.6, , 00000000002.1701.6, , 00000000002.512.6, , , 00000000002.666.6, , , ] 2023-05-24T21:27:02.548 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:27:02.542+0000 7fb2ecffd780 1 bucket sync caught up with source: 2023-05-24T21:27:02.548 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [00000000002.388.6, , 00000000002.1701.6, , 00000000002.512.6, , , 00000000002.666.6, , , ] 2023-05-24T21:27:02.548 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [00000000002.388.6, , 00000000002.1701.6, , 00000000002.512.6, , , 00000000002.666.6, , , ] 2023-05-24T21:27:02.548 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:27:02.542+0000 7fb2ecffd780 0 bucket checkpoint complete 2023-05-24T21:27:02.572 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qskrps-46 zones={a2, a1} 2023-05-24T21:27:02.573 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:27:02.573 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:27:02.574 DEBUG:boto:Method: HEAD 2023-05-24T21:27:02.574 DEBUG:boto:Path: /qskrps-46/ 2023-05-24T21:27:02.574 DEBUG:boto:Data: 2023-05-24T21:27:02.575 DEBUG:boto:Headers: {} 2023-05-24T21:27:02.575 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:02.575 DEBUG:boto:Port: 8001 2023-05-24T21:27:02.576 DEBUG:boto:Params: {} 2023-05-24T21:27:02.576 DEBUG:boto:Token: None 2023-05-24T21:27:02.577 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:02 GMT /qskrps-46/ 2023-05-24T21:27:02.577 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Ti2Ga+tNb9P3eKs+bs6k1t1ie2o= 2023-05-24T21:27:02.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Ti2Ga+tNb9P3eKs+bs6k1t1ie2o=', 'Content-Length': '0'} 2023-05-24T21:27:02.580 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bca4e6a4e9b3e962-00646e8126-1625-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:27:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:02.580 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:27:02.581 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:27:02.581 DEBUG:boto:Method: HEAD 2023-05-24T21:27:02.581 DEBUG:boto:Path: /qskrps-46/ 2023-05-24T21:27:02.582 DEBUG:boto:Data: 2023-05-24T21:27:02.582 DEBUG:boto:Headers: {} 2023-05-24T21:27:02.582 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:02.583 DEBUG:boto:Port: 8000 2023-05-24T21:27:02.583 DEBUG:boto:Params: {} 2023-05-24T21:27:02.583 DEBUG:boto:Token: None 2023-05-24T21:27:02.584 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:02 GMT /qskrps-46/ 2023-05-24T21:27:02.584 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Ti2Ga+tNb9P3eKs+bs6k1t1ie2o= 2023-05-24T21:27:02.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Ti2Ga+tNb9P3eKs+bs6k1t1ie2o=', 'Content-Length': '0'} 2023-05-24T21:27:02.587 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f598f44c4361787-00646e8126-1627-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:27:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:02.588 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-24T21:27:02.588 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:27:02.588 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:27:02.589 DEBUG:boto:path=/qskrps-46/?versions&max-keys=999 2023-05-24T21:27:02.589 DEBUG:boto:auth_path=/qskrps-46/?versions&max-keys=999 2023-05-24T21:27:02.589 DEBUG:boto:Method: GET 2023-05-24T21:27:02.590 DEBUG:boto:Path: /qskrps-46/?versions&max-keys=999 2023-05-24T21:27:02.590 DEBUG:boto:Data: 2023-05-24T21:27:02.590 DEBUG:boto:Headers: {} 2023-05-24T21:27:02.591 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:02.591 DEBUG:boto:Port: 8001 2023-05-24T21:27:02.591 DEBUG:boto:Params: {} 2023-05-24T21:27:02.592 DEBUG:boto:Token: None 2023-05-24T21:27:02.592 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:02 GMT /qskrps-46/?versions 2023-05-24T21:27:02.593 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ngeMqeQVmbLeMcAT+JaFBOYAqbM= 2023-05-24T21:27:02.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ngeMqeQVmbLeMcAT+JaFBOYAqbM=', 'Content-Length': '0'} 2023-05-24T21:27:02.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061fe5ec07e0591e7-00646e8126-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:02.596 DEBUG:boto:b'qskrps-46999falseobj1nulltrue2023-05-24T21:25:48.924Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-24T21:25:48.942Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-24T21:25:48.968Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-24T21:25:48.995Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-05-24T21:26:22.064Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-05-24T21:26:22.081Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-05-24T21:26:22.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-05-24T21:26:22.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:02.597 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-05-24T21:27:02.597 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-05-24T21:27:02.597 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-05-24T21:27:02.598 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-05-24T21:27:02.598 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-05-24T21:27:02.598 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-05-24T21:27:02.599 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-05-24T21:27:02.599 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-05-24T21:27:02.599 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-24T21:27:02.600 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:27:02.600 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:27:02.600 DEBUG:boto:path=/qskrps-46/?versions&max-keys=999 2023-05-24T21:27:02.601 DEBUG:boto:auth_path=/qskrps-46/?versions&max-keys=999 2023-05-24T21:27:02.601 DEBUG:boto:Method: GET 2023-05-24T21:27:02.601 DEBUG:boto:Path: /qskrps-46/?versions&max-keys=999 2023-05-24T21:27:02.602 DEBUG:boto:Data: 2023-05-24T21:27:02.602 DEBUG:boto:Headers: {} 2023-05-24T21:27:02.602 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:02.603 DEBUG:boto:Port: 8000 2023-05-24T21:27:02.603 DEBUG:boto:Params: {} 2023-05-24T21:27:02.603 DEBUG:boto:Token: None 2023-05-24T21:27:02.604 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:02 GMT /qskrps-46/?versions 2023-05-24T21:27:02.604 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ngeMqeQVmbLeMcAT+JaFBOYAqbM= 2023-05-24T21:27:02.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ngeMqeQVmbLeMcAT+JaFBOYAqbM=', 'Content-Length': '0'} 2023-05-24T21:27:02.607 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d979439beabfdc6-00646e8126-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:02.607 DEBUG:boto:b'qskrps-46999falseobj1nulltrue2023-05-24T21:25:48.924Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-24T21:25:48.942Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-24T21:25:48.968Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-24T21:25:48.995Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-05-24T21:26:22.064Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-05-24T21:26:22.081Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-05-24T21:26:22.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-05-24T21:26:22.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:02.608 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-05-24T21:27:02.609 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-05-24T21:27:02.609 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-05-24T21:27:02.609 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-05-24T21:27:02.610 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-05-24T21:27:02.610 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-05-24T21:27:02.610 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-05-24T21:27:02.610 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-05-24T21:27:02.611 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:27:02.611 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:27:02.612 DEBUG:boto:path=/qskrps-46/?versions&max-keys=999 2023-05-24T21:27:02.612 DEBUG:boto:auth_path=/qskrps-46/?versions&max-keys=999 2023-05-24T21:27:02.612 DEBUG:boto:Method: GET 2023-05-24T21:27:02.613 DEBUG:boto:Path: /qskrps-46/?versions&max-keys=999 2023-05-24T21:27:02.613 DEBUG:boto:Data: 2023-05-24T21:27:02.613 DEBUG:boto:Headers: {} 2023-05-24T21:27:02.614 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:02.614 DEBUG:boto:Port: 8001 2023-05-24T21:27:02.614 DEBUG:boto:Params: {} 2023-05-24T21:27:02.615 DEBUG:boto:Token: None 2023-05-24T21:27:02.615 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:02 GMT /qskrps-46/?versions 2023-05-24T21:27:02.616 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ngeMqeQVmbLeMcAT+JaFBOYAqbM= 2023-05-24T21:27:02.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ngeMqeQVmbLeMcAT+JaFBOYAqbM=', 'Content-Length': '0'} 2023-05-24T21:27:02.618 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004f8e9c0cf9159e4-00646e8126-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:02.619 DEBUG:boto:b'qskrps-46999falseobj1nulltrue2023-05-24T21:25:48.924Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-24T21:25:48.942Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-24T21:25:48.968Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-24T21:25:48.995Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-05-24T21:26:22.064Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-05-24T21:26:22.081Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-05-24T21:26:22.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-05-24T21:26:22.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:02.620 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:27:02.620 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:27:02.620 DEBUG:boto:path=/qskrps-46/?versions&max-keys=999 2023-05-24T21:27:02.621 DEBUG:boto:auth_path=/qskrps-46/?versions&max-keys=999 2023-05-24T21:27:02.621 DEBUG:boto:Method: GET 2023-05-24T21:27:02.621 DEBUG:boto:Path: /qskrps-46/?versions&max-keys=999 2023-05-24T21:27:02.622 DEBUG:boto:Data: 2023-05-24T21:27:02.622 DEBUG:boto:Headers: {} 2023-05-24T21:27:02.622 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:02.623 DEBUG:boto:Port: 8000 2023-05-24T21:27:02.623 DEBUG:boto:Params: {} 2023-05-24T21:27:02.623 DEBUG:boto:Token: None 2023-05-24T21:27:02.624 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:02 GMT /qskrps-46/?versions 2023-05-24T21:27:02.624 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ngeMqeQVmbLeMcAT+JaFBOYAqbM= 2023-05-24T21:27:02.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ngeMqeQVmbLeMcAT+JaFBOYAqbM=', 'Content-Length': '0'} 2023-05-24T21:27:02.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d72e24ead03e5a2-00646e8126-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:02.628 DEBUG:boto:b'qskrps-46999falseobj1nulltrue2023-05-24T21:25:48.924Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-24T21:25:48.942Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-24T21:25:48.968Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-24T21:25:48.995Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-05-24T21:26:22.064Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-05-24T21:26:22.081Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-05-24T21:26:22.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-05-24T21:26:22.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:02.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-24T21:27:02.629 DEBUG:boto:path=/qskrps-46/obj1 2023-05-24T21:27:02.629 DEBUG:boto:auth_path=/qskrps-46/obj1 2023-05-24T21:27:02.630 DEBUG:boto:path=/qskrps-46/obj1?versionId=null 2023-05-24T21:27:02.630 DEBUG:boto:auth_path=/qskrps-46/obj1?versionId=null 2023-05-24T21:27:02.631 DEBUG:boto:Method: GET 2023-05-24T21:27:02.631 DEBUG:boto:Path: /qskrps-46/obj1?versionId=null 2023-05-24T21:27:02.631 DEBUG:boto:Data: 2023-05-24T21:27:02.632 DEBUG:boto:Headers: {} 2023-05-24T21:27:02.632 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:02.632 DEBUG:boto:Port: 8001 2023-05-24T21:27:02.633 DEBUG:boto:Params: {} 2023-05-24T21:27:02.633 DEBUG:boto:Token: None 2023-05-24T21:27:02.633 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:02 GMT /qskrps-46/obj1?versionId=null 2023-05-24T21:27:02.634 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:1dhlX6xChCEsRoPD8n4DCxsla+E= 2023-05-24T21:27:02.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:1dhlX6xChCEsRoPD8n4DCxsla+E=', 'Content-Length': '0'} 2023-05-24T21:27:02.636 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000de5987ef16744e4b-00646e8126-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:02.637 DEBUG:boto:path=/qskrps-46/obj1 2023-05-24T21:27:02.637 DEBUG:boto:auth_path=/qskrps-46/obj1 2023-05-24T21:27:02.637 DEBUG:boto:path=/qskrps-46/obj1?versionId=null 2023-05-24T21:27:02.638 DEBUG:boto:auth_path=/qskrps-46/obj1?versionId=null 2023-05-24T21:27:02.638 DEBUG:boto:Method: GET 2023-05-24T21:27:02.639 DEBUG:boto:Path: /qskrps-46/obj1?versionId=null 2023-05-24T21:27:02.639 DEBUG:boto:Data: 2023-05-24T21:27:02.639 DEBUG:boto:Headers: {} 2023-05-24T21:27:02.640 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:02.640 DEBUG:boto:Port: 8000 2023-05-24T21:27:02.640 DEBUG:boto:Params: {} 2023-05-24T21:27:02.641 DEBUG:boto:Token: None 2023-05-24T21:27:02.641 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:02 GMT /qskrps-46/obj1?versionId=null 2023-05-24T21:27:02.642 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:1dhlX6xChCEsRoPD8n4DCxsla+E= 2023-05-24T21:27:02.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:1dhlX6xChCEsRoPD8n4DCxsla+E=', 'Content-Length': '0'} 2023-05-24T21:27:02.644 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e30ac492a3b784de-00646e8126-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:02.644 DEBUG:boto:path=/qskrps-46/obj1 2023-05-24T21:27:02.645 DEBUG:boto:auth_path=/qskrps-46/obj1 2023-05-24T21:27:02.645 DEBUG:boto:path=/qskrps-46/obj1?versionId=null 2023-05-24T21:27:02.645 DEBUG:boto:auth_path=/qskrps-46/obj1?versionId=null 2023-05-24T21:27:02.646 DEBUG:boto:Method: HEAD 2023-05-24T21:27:02.646 DEBUG:boto:Path: /qskrps-46/obj1?versionId=null 2023-05-24T21:27:02.646 DEBUG:boto:Data: 2023-05-24T21:27:02.647 DEBUG:boto:Headers: {} 2023-05-24T21:27:02.647 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:02.647 DEBUG:boto:Port: 8001 2023-05-24T21:27:02.648 DEBUG:boto:Params: {} 2023-05-24T21:27:02.648 DEBUG:boto:Token: None 2023-05-24T21:27:02.649 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:02 GMT /qskrps-46/obj1?versionId=null 2023-05-24T21:27:02.649 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:FyxTGumnnHzOW3TaRue2H82j8p8= 2023-05-24T21:27:02.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:FyxTGumnnHzOW3TaRue2H82j8p8=', 'Content-Length': '0'} 2023-05-24T21:27:02.651 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008c06899e52523e7d-00646e8126-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:02.652 DEBUG:boto:path=/qskrps-46/obj1 2023-05-24T21:27:02.652 DEBUG:boto:auth_path=/qskrps-46/obj1 2023-05-24T21:27:02.652 DEBUG:boto:path=/qskrps-46/obj1?versionId=null 2023-05-24T21:27:02.653 DEBUG:boto:auth_path=/qskrps-46/obj1?versionId=null 2023-05-24T21:27:02.653 DEBUG:boto:Method: HEAD 2023-05-24T21:27:02.653 DEBUG:boto:Path: /qskrps-46/obj1?versionId=null 2023-05-24T21:27:02.654 DEBUG:boto:Data: 2023-05-24T21:27:02.654 DEBUG:boto:Headers: {} 2023-05-24T21:27:02.654 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:02.655 DEBUG:boto:Port: 8000 2023-05-24T21:27:02.655 DEBUG:boto:Params: {} 2023-05-24T21:27:02.655 DEBUG:boto:Token: None 2023-05-24T21:27:02.656 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:02 GMT /qskrps-46/obj1?versionId=null 2023-05-24T21:27:02.656 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:FyxTGumnnHzOW3TaRue2H82j8p8= 2023-05-24T21:27:02.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:FyxTGumnnHzOW3TaRue2H82j8p8=', 'Content-Length': '0'} 2023-05-24T21:27:02.658 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fad0fbe4715290f5-00646e8126-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:02.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-24T21:27:02.659 DEBUG:boto:path=/qskrps-46/obj1 2023-05-24T21:27:02.659 DEBUG:boto:auth_path=/qskrps-46/obj1 2023-05-24T21:27:02.660 DEBUG:boto:path=/qskrps-46/obj1?versionId=null 2023-05-24T21:27:02.660 DEBUG:boto:auth_path=/qskrps-46/obj1?versionId=null 2023-05-24T21:27:02.661 DEBUG:boto:Method: GET 2023-05-24T21:27:02.661 DEBUG:boto:Path: /qskrps-46/obj1?versionId=null 2023-05-24T21:27:02.661 DEBUG:boto:Data: 2023-05-24T21:27:02.662 DEBUG:boto:Headers: {} 2023-05-24T21:27:02.662 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:02.662 DEBUG:boto:Port: 8001 2023-05-24T21:27:02.663 DEBUG:boto:Params: {} 2023-05-24T21:27:02.663 DEBUG:boto:Token: None 2023-05-24T21:27:02.663 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:02 GMT /qskrps-46/obj1?versionId=null 2023-05-24T21:27:02.664 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:1dhlX6xChCEsRoPD8n4DCxsla+E= 2023-05-24T21:27:02.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:1dhlX6xChCEsRoPD8n4DCxsla+E=', 'Content-Length': '0'} 2023-05-24T21:27:02.666 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb2378dbbf1c50bd-00646e8126-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:02.667 DEBUG:boto:path=/qskrps-46/obj1 2023-05-24T21:27:02.667 DEBUG:boto:auth_path=/qskrps-46/obj1 2023-05-24T21:27:02.667 DEBUG:boto:path=/qskrps-46/obj1?versionId=null 2023-05-24T21:27:02.668 DEBUG:boto:auth_path=/qskrps-46/obj1?versionId=null 2023-05-24T21:27:02.668 DEBUG:boto:Method: GET 2023-05-24T21:27:02.668 DEBUG:boto:Path: /qskrps-46/obj1?versionId=null 2023-05-24T21:27:02.669 DEBUG:boto:Data: 2023-05-24T21:27:02.669 DEBUG:boto:Headers: {} 2023-05-24T21:27:02.670 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:02.670 DEBUG:boto:Port: 8000 2023-05-24T21:27:02.670 DEBUG:boto:Params: {} 2023-05-24T21:27:02.671 DEBUG:boto:Token: None 2023-05-24T21:27:02.671 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:02 GMT /qskrps-46/obj1?versionId=null 2023-05-24T21:27:02.671 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:1dhlX6xChCEsRoPD8n4DCxsla+E= 2023-05-24T21:27:02.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:1dhlX6xChCEsRoPD8n4DCxsla+E=', 'Content-Length': '0'} 2023-05-24T21:27:02.674 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000010178c31b38c2cf6-00646e8126-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:02.674 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:27:02.675 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:27:02.675 DEBUG:boto:path=/qskrps-46/?prefix=obj1 2023-05-24T21:27:02.676 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj1 2023-05-24T21:27:02.676 DEBUG:boto:Method: GET 2023-05-24T21:27:02.676 DEBUG:boto:Path: /qskrps-46/?prefix=obj1 2023-05-24T21:27:02.677 DEBUG:boto:Data: 2023-05-24T21:27:02.677 DEBUG:boto:Headers: {} 2023-05-24T21:27:02.677 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:02.678 DEBUG:boto:Port: 8001 2023-05-24T21:27:02.678 DEBUG:boto:Params: {} 2023-05-24T21:27:02.678 DEBUG:boto:Token: None 2023-05-24T21:27:02.679 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:02 GMT /qskrps-46/ 2023-05-24T21:27:02.679 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:RDukf52BH4OYVtaB7UuqFq/ZJQs= 2023-05-24T21:27:02.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:RDukf52BH4OYVtaB7UuqFq/ZJQs=', 'Content-Length': '0'} 2023-05-24T21:27:02.682 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000129d91d5032c7b50-00646e8126-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:02.682 DEBUG:boto:b'qskrps-46obj11000falseobj12023-05-24T21:25:48.924Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:02.683 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:27:02.683 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:27:02.684 DEBUG:boto:path=/qskrps-46/?prefix=obj1 2023-05-24T21:27:02.684 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj1 2023-05-24T21:27:02.684 DEBUG:boto:Method: GET 2023-05-24T21:27:02.685 DEBUG:boto:Path: /qskrps-46/?prefix=obj1 2023-05-24T21:27:02.685 DEBUG:boto:Data: 2023-05-24T21:27:02.685 DEBUG:boto:Headers: {} 2023-05-24T21:27:02.686 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:02.686 DEBUG:boto:Port: 8000 2023-05-24T21:27:02.686 DEBUG:boto:Params: {} 2023-05-24T21:27:02.687 DEBUG:boto:Token: None 2023-05-24T21:27:02.687 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:02 GMT /qskrps-46/ 2023-05-24T21:27:02.688 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:RDukf52BH4OYVtaB7UuqFq/ZJQs= 2023-05-24T21:27:02.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:RDukf52BH4OYVtaB7UuqFq/ZJQs=', 'Content-Length': '0'} 2023-05-24T21:27:02.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007c663b4445e37a7-00646e8126-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:02.691 DEBUG:boto:b'qskrps-46obj11000falseobj12023-05-24T21:25:48.924Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:02.691 DEBUG:boto:path=/qskrps-46/obj1 2023-05-24T21:27:02.692 DEBUG:boto:auth_path=/qskrps-46/obj1 2023-05-24T21:27:02.692 DEBUG:boto:Method: HEAD 2023-05-24T21:27:02.692 DEBUG:boto:Path: /qskrps-46/obj1 2023-05-24T21:27:02.693 DEBUG:boto:Data: 2023-05-24T21:27:02.693 DEBUG:boto:Headers: {} 2023-05-24T21:27:02.693 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:02.694 DEBUG:boto:Port: 8001 2023-05-24T21:27:02.694 DEBUG:boto:Params: {} 2023-05-24T21:27:02.694 DEBUG:boto:Token: None 2023-05-24T21:27:02.695 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:02 GMT /qskrps-46/obj1 2023-05-24T21:27:02.695 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:v/o2WT5gfFrELcyLTdDgugkuh6k= 2023-05-24T21:27:02.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:v/o2WT5gfFrELcyLTdDgugkuh6k=', 'Content-Length': '0'} 2023-05-24T21:27:02.697 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002627e3d0f11c77d9-00646e8126-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:02.698 DEBUG:boto:path=/qskrps-46/obj1 2023-05-24T21:27:02.698 DEBUG:boto:auth_path=/qskrps-46/obj1 2023-05-24T21:27:02.698 DEBUG:boto:Method: HEAD 2023-05-24T21:27:02.699 DEBUG:boto:Path: /qskrps-46/obj1 2023-05-24T21:27:02.699 DEBUG:boto:Data: 2023-05-24T21:27:02.699 DEBUG:boto:Headers: {} 2023-05-24T21:27:02.700 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:02.700 DEBUG:boto:Port: 8000 2023-05-24T21:27:02.700 DEBUG:boto:Params: {} 2023-05-24T21:27:02.701 DEBUG:boto:Token: None 2023-05-24T21:27:02.701 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:02 GMT /qskrps-46/obj1 2023-05-24T21:27:02.701 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:v/o2WT5gfFrELcyLTdDgugkuh6k= 2023-05-24T21:27:02.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:v/o2WT5gfFrELcyLTdDgugkuh6k=', 'Content-Length': '0'} 2023-05-24T21:27:02.703 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ec8e60da3cbb51cb-00646e8126-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:02.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-24T21:27:02.704 DEBUG:boto:path=/qskrps-46/obj1 2023-05-24T21:27:02.705 DEBUG:boto:auth_path=/qskrps-46/obj1 2023-05-24T21:27:02.705 DEBUG:boto:Method: GET 2023-05-24T21:27:02.705 DEBUG:boto:Path: /qskrps-46/obj1 2023-05-24T21:27:02.706 DEBUG:boto:Data: 2023-05-24T21:27:02.706 DEBUG:boto:Headers: {} 2023-05-24T21:27:02.707 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:02.707 DEBUG:boto:Port: 8001 2023-05-24T21:27:02.707 DEBUG:boto:Params: {} 2023-05-24T21:27:02.708 DEBUG:boto:Token: None 2023-05-24T21:27:02.708 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:02 GMT /qskrps-46/obj1 2023-05-24T21:27:02.708 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:URV7Vw/v28hq+BSWSfgY2ijVcsk= 2023-05-24T21:27:02.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:URV7Vw/v28hq+BSWSfgY2ijVcsk=', 'Content-Length': '0'} 2023-05-24T21:27:02.711 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009d932c09dc164610-00646e8126-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:02.711 DEBUG:boto:path=/qskrps-46/obj1 2023-05-24T21:27:02.712 DEBUG:boto:auth_path=/qskrps-46/obj1 2023-05-24T21:27:02.712 DEBUG:boto:Method: GET 2023-05-24T21:27:02.712 DEBUG:boto:Path: /qskrps-46/obj1 2023-05-24T21:27:02.713 DEBUG:boto:Data: 2023-05-24T21:27:02.713 DEBUG:boto:Headers: {} 2023-05-24T21:27:02.713 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:02.714 DEBUG:boto:Port: 8000 2023-05-24T21:27:02.714 DEBUG:boto:Params: {} 2023-05-24T21:27:02.715 DEBUG:boto:Token: None 2023-05-24T21:27:02.715 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:02 GMT /qskrps-46/obj1 2023-05-24T21:27:02.715 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:URV7Vw/v28hq+BSWSfgY2ijVcsk= 2023-05-24T21:27:02.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:URV7Vw/v28hq+BSWSfgY2ijVcsk=', 'Content-Length': '0'} 2023-05-24T21:27:02.718 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004611dc98650cda67-00646e8126-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:02.718 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:27:02.719 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:27:02.719 DEBUG:boto:path=/qskrps-46/?prefix=obj1 2023-05-24T21:27:02.719 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj1 2023-05-24T21:27:02.720 DEBUG:boto:Method: GET 2023-05-24T21:27:02.720 DEBUG:boto:Path: /qskrps-46/?prefix=obj1 2023-05-24T21:27:02.720 DEBUG:boto:Data: 2023-05-24T21:27:02.721 DEBUG:boto:Headers: {} 2023-05-24T21:27:02.721 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:02.721 DEBUG:boto:Port: 8001 2023-05-24T21:27:02.722 DEBUG:boto:Params: {} 2023-05-24T21:27:02.722 DEBUG:boto:Token: None 2023-05-24T21:27:02.723 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:02 GMT /qskrps-46/ 2023-05-24T21:27:02.723 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:RDukf52BH4OYVtaB7UuqFq/ZJQs= 2023-05-24T21:27:02.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:RDukf52BH4OYVtaB7UuqFq/ZJQs=', 'Content-Length': '0'} 2023-05-24T21:27:02.726 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec2b78ea03747606-00646e8126-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:02.726 DEBUG:boto:b'qskrps-46obj11000falseobj12023-05-24T21:25:48.924Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:02.727 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:27:02.727 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:27:02.728 DEBUG:boto:path=/qskrps-46/?prefix=obj1 2023-05-24T21:27:02.728 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj1 2023-05-24T21:27:02.728 DEBUG:boto:Method: GET 2023-05-24T21:27:02.729 DEBUG:boto:Path: /qskrps-46/?prefix=obj1 2023-05-24T21:27:02.729 DEBUG:boto:Data: 2023-05-24T21:27:02.729 DEBUG:boto:Headers: {} 2023-05-24T21:27:02.730 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:02.730 DEBUG:boto:Port: 8000 2023-05-24T21:27:02.731 DEBUG:boto:Params: {} 2023-05-24T21:27:02.731 DEBUG:boto:Token: None 2023-05-24T21:27:02.731 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:02 GMT /qskrps-46/ 2023-05-24T21:27:02.732 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:RDukf52BH4OYVtaB7UuqFq/ZJQs= 2023-05-24T21:27:02.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:RDukf52BH4OYVtaB7UuqFq/ZJQs=', 'Content-Length': '0'} 2023-05-24T21:27:02.734 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c922935cd574d09-00646e8126-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:02.735 DEBUG:boto:b'qskrps-46obj11000falseobj12023-05-24T21:25:48.924Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:02.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-24T21:27:02.736 DEBUG:boto:path=/qskrps-46/obj2 2023-05-24T21:27:02.736 DEBUG:boto:auth_path=/qskrps-46/obj2 2023-05-24T21:27:02.736 DEBUG:boto:path=/qskrps-46/obj2?versionId=null 2023-05-24T21:27:02.737 DEBUG:boto:auth_path=/qskrps-46/obj2?versionId=null 2023-05-24T21:27:02.737 DEBUG:boto:Method: GET 2023-05-24T21:27:02.738 DEBUG:boto:Path: /qskrps-46/obj2?versionId=null 2023-05-24T21:27:02.738 DEBUG:boto:Data: 2023-05-24T21:27:02.738 DEBUG:boto:Headers: {} 2023-05-24T21:27:02.739 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:02.739 DEBUG:boto:Port: 8001 2023-05-24T21:27:02.739 DEBUG:boto:Params: {} 2023-05-24T21:27:02.740 DEBUG:boto:Token: None 2023-05-24T21:27:02.740 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:02 GMT /qskrps-46/obj2?versionId=null 2023-05-24T21:27:02.741 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:kZ0jPCHU7vwcjZnQn3qa6aGdqZ4= 2023-05-24T21:27:02.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:kZ0jPCHU7vwcjZnQn3qa6aGdqZ4=', 'Content-Length': '0'} 2023-05-24T21:27:02.743 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d62d149a7a967056-00646e8126-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:02.743 DEBUG:boto:path=/qskrps-46/obj2 2023-05-24T21:27:02.744 DEBUG:boto:auth_path=/qskrps-46/obj2 2023-05-24T21:27:02.744 DEBUG:boto:path=/qskrps-46/obj2?versionId=null 2023-05-24T21:27:02.744 DEBUG:boto:auth_path=/qskrps-46/obj2?versionId=null 2023-05-24T21:27:02.745 DEBUG:boto:Method: GET 2023-05-24T21:27:02.745 DEBUG:boto:Path: /qskrps-46/obj2?versionId=null 2023-05-24T21:27:02.745 DEBUG:boto:Data: 2023-05-24T21:27:02.746 DEBUG:boto:Headers: {} 2023-05-24T21:27:02.746 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:02.746 DEBUG:boto:Port: 8000 2023-05-24T21:27:02.747 DEBUG:boto:Params: {} 2023-05-24T21:27:02.747 DEBUG:boto:Token: None 2023-05-24T21:27:02.748 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:02 GMT /qskrps-46/obj2?versionId=null 2023-05-24T21:27:02.748 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:kZ0jPCHU7vwcjZnQn3qa6aGdqZ4= 2023-05-24T21:27:02.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:kZ0jPCHU7vwcjZnQn3qa6aGdqZ4=', 'Content-Length': '0'} 2023-05-24T21:27:02.750 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002184e75e32d0a0d9-00646e8126-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:02.751 DEBUG:boto:path=/qskrps-46/obj2 2023-05-24T21:27:02.751 DEBUG:boto:auth_path=/qskrps-46/obj2 2023-05-24T21:27:02.752 DEBUG:boto:path=/qskrps-46/obj2?versionId=null 2023-05-24T21:27:02.752 DEBUG:boto:auth_path=/qskrps-46/obj2?versionId=null 2023-05-24T21:27:02.752 DEBUG:boto:Method: HEAD 2023-05-24T21:27:02.753 DEBUG:boto:Path: /qskrps-46/obj2?versionId=null 2023-05-24T21:27:02.753 DEBUG:boto:Data: 2023-05-24T21:27:02.753 DEBUG:boto:Headers: {} 2023-05-24T21:27:02.754 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:02.754 DEBUG:boto:Port: 8001 2023-05-24T21:27:02.754 DEBUG:boto:Params: {} 2023-05-24T21:27:02.755 DEBUG:boto:Token: None 2023-05-24T21:27:02.755 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:02 GMT /qskrps-46/obj2?versionId=null 2023-05-24T21:27:02.755 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:fJ29EYCE1t54sJOjmnjysxENFFA= 2023-05-24T21:27:02.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:fJ29EYCE1t54sJOjmnjysxENFFA=', 'Content-Length': '0'} 2023-05-24T21:27:02.757 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007e7b70beba621e31-00646e8126-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:02.758 DEBUG:boto:path=/qskrps-46/obj2 2023-05-24T21:27:02.758 DEBUG:boto:auth_path=/qskrps-46/obj2 2023-05-24T21:27:02.759 DEBUG:boto:path=/qskrps-46/obj2?versionId=null 2023-05-24T21:27:02.759 DEBUG:boto:auth_path=/qskrps-46/obj2?versionId=null 2023-05-24T21:27:02.759 DEBUG:boto:Method: HEAD 2023-05-24T21:27:02.760 DEBUG:boto:Path: /qskrps-46/obj2?versionId=null 2023-05-24T21:27:02.760 DEBUG:boto:Data: 2023-05-24T21:27:02.760 DEBUG:boto:Headers: {} 2023-05-24T21:27:02.761 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:02.761 DEBUG:boto:Port: 8000 2023-05-24T21:27:02.761 DEBUG:boto:Params: {} 2023-05-24T21:27:02.762 DEBUG:boto:Token: None 2023-05-24T21:27:02.762 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:02 GMT /qskrps-46/obj2?versionId=null 2023-05-24T21:27:02.762 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:fJ29EYCE1t54sJOjmnjysxENFFA= 2023-05-24T21:27:02.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:fJ29EYCE1t54sJOjmnjysxENFFA=', 'Content-Length': '0'} 2023-05-24T21:27:02.764 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b55309ec6817ab0b-00646e8126-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:02.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-24T21:27:02.765 DEBUG:boto:path=/qskrps-46/obj2 2023-05-24T21:27:02.766 DEBUG:boto:auth_path=/qskrps-46/obj2 2023-05-24T21:27:02.766 DEBUG:boto:path=/qskrps-46/obj2?versionId=null 2023-05-24T21:27:02.767 DEBUG:boto:auth_path=/qskrps-46/obj2?versionId=null 2023-05-24T21:27:02.767 DEBUG:boto:Method: GET 2023-05-24T21:27:02.767 DEBUG:boto:Path: /qskrps-46/obj2?versionId=null 2023-05-24T21:27:02.768 DEBUG:boto:Data: 2023-05-24T21:27:02.768 DEBUG:boto:Headers: {} 2023-05-24T21:27:02.768 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:02.769 DEBUG:boto:Port: 8001 2023-05-24T21:27:02.769 DEBUG:boto:Params: {} 2023-05-24T21:27:02.769 DEBUG:boto:Token: None 2023-05-24T21:27:02.770 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:02 GMT /qskrps-46/obj2?versionId=null 2023-05-24T21:27:02.770 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:kZ0jPCHU7vwcjZnQn3qa6aGdqZ4= 2023-05-24T21:27:02.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:kZ0jPCHU7vwcjZnQn3qa6aGdqZ4=', 'Content-Length': '0'} 2023-05-24T21:27:02.772 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000282fae7c8412d201-00646e8126-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:02.773 DEBUG:boto:path=/qskrps-46/obj2 2023-05-24T21:27:02.773 DEBUG:boto:auth_path=/qskrps-46/obj2 2023-05-24T21:27:02.774 DEBUG:boto:path=/qskrps-46/obj2?versionId=null 2023-05-24T21:27:02.774 DEBUG:boto:auth_path=/qskrps-46/obj2?versionId=null 2023-05-24T21:27:02.775 DEBUG:boto:Method: GET 2023-05-24T21:27:02.775 DEBUG:boto:Path: /qskrps-46/obj2?versionId=null 2023-05-24T21:27:02.775 DEBUG:boto:Data: 2023-05-24T21:27:02.776 DEBUG:boto:Headers: {} 2023-05-24T21:27:02.776 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:02.776 DEBUG:boto:Port: 8000 2023-05-24T21:27:02.777 DEBUG:boto:Params: {} 2023-05-24T21:27:02.777 DEBUG:boto:Token: None 2023-05-24T21:27:02.777 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:02 GMT /qskrps-46/obj2?versionId=null 2023-05-24T21:27:02.778 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:kZ0jPCHU7vwcjZnQn3qa6aGdqZ4= 2023-05-24T21:27:02.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:kZ0jPCHU7vwcjZnQn3qa6aGdqZ4=', 'Content-Length': '0'} 2023-05-24T21:27:02.780 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000eab0de4c58e83e4-00646e8126-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:02.824 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:27:02.825 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:27:02.825 DEBUG:boto:path=/qskrps-46/?prefix=obj2 2023-05-24T21:27:02.825 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj2 2023-05-24T21:27:02.826 DEBUG:boto:Method: GET 2023-05-24T21:27:02.826 DEBUG:boto:Path: /qskrps-46/?prefix=obj2 2023-05-24T21:27:02.827 DEBUG:boto:Data: 2023-05-24T21:27:02.827 DEBUG:boto:Headers: {} 2023-05-24T21:27:02.827 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:02.828 DEBUG:boto:Port: 8001 2023-05-24T21:27:02.828 DEBUG:boto:Params: {} 2023-05-24T21:27:02.828 DEBUG:boto:Token: None 2023-05-24T21:27:02.829 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:02 GMT /qskrps-46/ 2023-05-24T21:27:02.829 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:RDukf52BH4OYVtaB7UuqFq/ZJQs= 2023-05-24T21:27:02.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:RDukf52BH4OYVtaB7UuqFq/ZJQs=', 'Content-Length': '0'} 2023-05-24T21:27:02.832 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000144bd70e2cf8aa83-00646e8126-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:02.833 DEBUG:boto:b'qskrps-46obj21000falseobj22023-05-24T21:25:48.942Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:02.833 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:27:02.834 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:27:02.834 DEBUG:boto:path=/qskrps-46/?prefix=obj2 2023-05-24T21:27:02.834 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj2 2023-05-24T21:27:02.835 DEBUG:boto:Method: GET 2023-05-24T21:27:02.835 DEBUG:boto:Path: /qskrps-46/?prefix=obj2 2023-05-24T21:27:02.836 DEBUG:boto:Data: 2023-05-24T21:27:02.836 DEBUG:boto:Headers: {} 2023-05-24T21:27:02.836 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:02.837 DEBUG:boto:Port: 8000 2023-05-24T21:27:02.837 DEBUG:boto:Params: {} 2023-05-24T21:27:02.837 DEBUG:boto:Token: None 2023-05-24T21:27:02.838 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:02 GMT /qskrps-46/ 2023-05-24T21:27:02.838 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:RDukf52BH4OYVtaB7UuqFq/ZJQs= 2023-05-24T21:27:02.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:RDukf52BH4OYVtaB7UuqFq/ZJQs=', 'Content-Length': '0'} 2023-05-24T21:27:02.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bebac48eb4aef74-00646e8126-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:02.841 DEBUG:boto:b'qskrps-46obj21000falseobj22023-05-24T21:25:48.942Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:02.842 DEBUG:boto:path=/qskrps-46/obj2 2023-05-24T21:27:02.843 DEBUG:boto:auth_path=/qskrps-46/obj2 2023-05-24T21:27:02.843 DEBUG:boto:Method: HEAD 2023-05-24T21:27:02.843 DEBUG:boto:Path: /qskrps-46/obj2 2023-05-24T21:27:02.844 DEBUG:boto:Data: 2023-05-24T21:27:02.844 DEBUG:boto:Headers: {} 2023-05-24T21:27:02.844 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:02.845 DEBUG:boto:Port: 8001 2023-05-24T21:27:02.845 DEBUG:boto:Params: {} 2023-05-24T21:27:02.845 DEBUG:boto:Token: None 2023-05-24T21:27:02.846 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:02 GMT /qskrps-46/obj2 2023-05-24T21:27:02.846 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:wWFKPR5O1vL7aqlaUvCxh8vV1d8= 2023-05-24T21:27:02.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wWFKPR5O1vL7aqlaUvCxh8vV1d8=', 'Content-Length': '0'} 2023-05-24T21:27:02.848 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000034db319a71bf1fac-00646e8126-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:02.849 DEBUG:boto:path=/qskrps-46/obj2 2023-05-24T21:27:02.849 DEBUG:boto:auth_path=/qskrps-46/obj2 2023-05-24T21:27:02.849 DEBUG:boto:Method: HEAD 2023-05-24T21:27:02.850 DEBUG:boto:Path: /qskrps-46/obj2 2023-05-24T21:27:02.850 DEBUG:boto:Data: 2023-05-24T21:27:02.850 DEBUG:boto:Headers: {} 2023-05-24T21:27:02.851 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:02.851 DEBUG:boto:Port: 8000 2023-05-24T21:27:02.851 DEBUG:boto:Params: {} 2023-05-24T21:27:02.852 DEBUG:boto:Token: None 2023-05-24T21:27:02.852 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:02 GMT /qskrps-46/obj2 2023-05-24T21:27:02.853 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:wWFKPR5O1vL7aqlaUvCxh8vV1d8= 2023-05-24T21:27:02.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wWFKPR5O1vL7aqlaUvCxh8vV1d8=', 'Content-Length': '0'} 2023-05-24T21:27:02.855 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000649114b6cf8b0fa6-00646e8126-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:02.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-24T21:27:02.856 DEBUG:boto:path=/qskrps-46/obj2 2023-05-24T21:27:02.856 DEBUG:boto:auth_path=/qskrps-46/obj2 2023-05-24T21:27:02.857 DEBUG:boto:Method: GET 2023-05-24T21:27:02.857 DEBUG:boto:Path: /qskrps-46/obj2 2023-05-24T21:27:02.857 DEBUG:boto:Data: 2023-05-24T21:27:02.858 DEBUG:boto:Headers: {} 2023-05-24T21:27:02.858 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:02.858 DEBUG:boto:Port: 8001 2023-05-24T21:27:02.859 DEBUG:boto:Params: {} 2023-05-24T21:27:02.859 DEBUG:boto:Token: None 2023-05-24T21:27:02.859 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:02 GMT /qskrps-46/obj2 2023-05-24T21:27:02.860 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:qVQBCUuxRpr38NTsFPoR36crv0w= 2023-05-24T21:27:02.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qVQBCUuxRpr38NTsFPoR36crv0w=', 'Content-Length': '0'} 2023-05-24T21:27:02.862 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000158bf96b5b187353-00646e8126-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:02.904 DEBUG:boto:path=/qskrps-46/obj2 2023-05-24T21:27:02.905 DEBUG:boto:auth_path=/qskrps-46/obj2 2023-05-24T21:27:02.905 DEBUG:boto:Method: GET 2023-05-24T21:27:02.906 DEBUG:boto:Path: /qskrps-46/obj2 2023-05-24T21:27:02.906 DEBUG:boto:Data: 2023-05-24T21:27:02.906 DEBUG:boto:Headers: {} 2023-05-24T21:27:02.907 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:02.907 DEBUG:boto:Port: 8000 2023-05-24T21:27:02.907 DEBUG:boto:Params: {} 2023-05-24T21:27:02.908 DEBUG:boto:Token: None 2023-05-24T21:27:02.908 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:02 GMT /qskrps-46/obj2 2023-05-24T21:27:02.909 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:qVQBCUuxRpr38NTsFPoR36crv0w= 2023-05-24T21:27:02.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qVQBCUuxRpr38NTsFPoR36crv0w=', 'Content-Length': '0'} 2023-05-24T21:27:02.911 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000002430968f53175ac-00646e8126-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:02.911 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:27:02.912 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:27:02.912 DEBUG:boto:path=/qskrps-46/?prefix=obj2 2023-05-24T21:27:02.912 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj2 2023-05-24T21:27:02.913 DEBUG:boto:Method: GET 2023-05-24T21:27:02.913 DEBUG:boto:Path: /qskrps-46/?prefix=obj2 2023-05-24T21:27:02.914 DEBUG:boto:Data: 2023-05-24T21:27:02.914 DEBUG:boto:Headers: {} 2023-05-24T21:27:02.914 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:02.915 DEBUG:boto:Port: 8001 2023-05-24T21:27:02.915 DEBUG:boto:Params: {} 2023-05-24T21:27:02.915 DEBUG:boto:Token: None 2023-05-24T21:27:02.916 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:02 GMT /qskrps-46/ 2023-05-24T21:27:02.916 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:RDukf52BH4OYVtaB7UuqFq/ZJQs= 2023-05-24T21:27:02.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:RDukf52BH4OYVtaB7UuqFq/ZJQs=', 'Content-Length': '0'} 2023-05-24T21:27:02.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8f71b6a6a783733-00646e8126-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:02.919 DEBUG:boto:b'qskrps-46obj21000falseobj22023-05-24T21:25:48.942Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:02.920 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:27:02.920 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:27:02.921 DEBUG:boto:path=/qskrps-46/?prefix=obj2 2023-05-24T21:27:02.921 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj2 2023-05-24T21:27:02.921 DEBUG:boto:Method: GET 2023-05-24T21:27:02.922 DEBUG:boto:Path: /qskrps-46/?prefix=obj2 2023-05-24T21:27:02.922 DEBUG:boto:Data: 2023-05-24T21:27:02.923 DEBUG:boto:Headers: {} 2023-05-24T21:27:02.923 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:02.923 DEBUG:boto:Port: 8000 2023-05-24T21:27:02.924 DEBUG:boto:Params: {} 2023-05-24T21:27:02.924 DEBUG:boto:Token: None 2023-05-24T21:27:02.924 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:02 GMT /qskrps-46/ 2023-05-24T21:27:02.925 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:RDukf52BH4OYVtaB7UuqFq/ZJQs= 2023-05-24T21:27:02.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:RDukf52BH4OYVtaB7UuqFq/ZJQs=', 'Content-Length': '0'} 2023-05-24T21:27:02.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009dd098f75c97dbf-00646e8126-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:02.928 DEBUG:boto:b'qskrps-46obj21000falseobj22023-05-24T21:25:48.942Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:02.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-24T21:27:02.929 DEBUG:boto:path=/qskrps-46/obj3 2023-05-24T21:27:02.929 DEBUG:boto:auth_path=/qskrps-46/obj3 2023-05-24T21:27:02.930 DEBUG:boto:path=/qskrps-46/obj3?versionId=null 2023-05-24T21:27:02.930 DEBUG:boto:auth_path=/qskrps-46/obj3?versionId=null 2023-05-24T21:27:02.931 DEBUG:boto:Method: GET 2023-05-24T21:27:02.931 DEBUG:boto:Path: /qskrps-46/obj3?versionId=null 2023-05-24T21:27:02.931 DEBUG:boto:Data: 2023-05-24T21:27:02.932 DEBUG:boto:Headers: {} 2023-05-24T21:27:02.932 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:02.932 DEBUG:boto:Port: 8001 2023-05-24T21:27:02.933 DEBUG:boto:Params: {} 2023-05-24T21:27:02.933 DEBUG:boto:Token: None 2023-05-24T21:27:02.933 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:02 GMT /qskrps-46/obj3?versionId=null 2023-05-24T21:27:02.934 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:3ykA+tX40+X8T4cjFotqswLd9ek= 2023-05-24T21:27:02.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:3ykA+tX40+X8T4cjFotqswLd9ek=', 'Content-Length': '0'} 2023-05-24T21:27:02.936 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007890622dbbfbaf35-00646e8126-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:02.937 DEBUG:boto:path=/qskrps-46/obj3 2023-05-24T21:27:02.937 DEBUG:boto:auth_path=/qskrps-46/obj3 2023-05-24T21:27:02.938 DEBUG:boto:path=/qskrps-46/obj3?versionId=null 2023-05-24T21:27:02.938 DEBUG:boto:auth_path=/qskrps-46/obj3?versionId=null 2023-05-24T21:27:02.938 DEBUG:boto:Method: GET 2023-05-24T21:27:02.939 DEBUG:boto:Path: /qskrps-46/obj3?versionId=null 2023-05-24T21:27:02.939 DEBUG:boto:Data: 2023-05-24T21:27:02.939 DEBUG:boto:Headers: {} 2023-05-24T21:27:02.940 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:02.940 DEBUG:boto:Port: 8000 2023-05-24T21:27:02.941 DEBUG:boto:Params: {} 2023-05-24T21:27:02.941 DEBUG:boto:Token: None 2023-05-24T21:27:02.941 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:02 GMT /qskrps-46/obj3?versionId=null 2023-05-24T21:27:02.942 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:3ykA+tX40+X8T4cjFotqswLd9ek= 2023-05-24T21:27:02.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:3ykA+tX40+X8T4cjFotqswLd9ek=', 'Content-Length': '0'} 2023-05-24T21:27:02.944 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d86f5c5f8c9b9f2-00646e8126-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:02.945 DEBUG:boto:path=/qskrps-46/obj3 2023-05-24T21:27:02.945 DEBUG:boto:auth_path=/qskrps-46/obj3 2023-05-24T21:27:02.945 DEBUG:boto:path=/qskrps-46/obj3?versionId=null 2023-05-24T21:27:02.946 DEBUG:boto:auth_path=/qskrps-46/obj3?versionId=null 2023-05-24T21:27:02.946 DEBUG:boto:Method: HEAD 2023-05-24T21:27:02.946 DEBUG:boto:Path: /qskrps-46/obj3?versionId=null 2023-05-24T21:27:02.947 DEBUG:boto:Data: 2023-05-24T21:27:02.947 DEBUG:boto:Headers: {} 2023-05-24T21:27:02.947 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:02.948 DEBUG:boto:Port: 8001 2023-05-24T21:27:02.948 DEBUG:boto:Params: {} 2023-05-24T21:27:02.948 DEBUG:boto:Token: None 2023-05-24T21:27:02.949 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:02 GMT /qskrps-46/obj3?versionId=null 2023-05-24T21:27:02.949 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:oAlpau0V3h5VGREKuuPhALu4qmY= 2023-05-24T21:27:02.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:oAlpau0V3h5VGREKuuPhALu4qmY=', 'Content-Length': '0'} 2023-05-24T21:27:02.951 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b0ad09b11dfd2293-00646e8126-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:02.952 DEBUG:boto:path=/qskrps-46/obj3 2023-05-24T21:27:02.952 DEBUG:boto:auth_path=/qskrps-46/obj3 2023-05-24T21:27:02.953 DEBUG:boto:path=/qskrps-46/obj3?versionId=null 2023-05-24T21:27:02.953 DEBUG:boto:auth_path=/qskrps-46/obj3?versionId=null 2023-05-24T21:27:02.953 DEBUG:boto:Method: HEAD 2023-05-24T21:27:02.954 DEBUG:boto:Path: /qskrps-46/obj3?versionId=null 2023-05-24T21:27:02.954 DEBUG:boto:Data: 2023-05-24T21:27:02.954 DEBUG:boto:Headers: {} 2023-05-24T21:27:02.955 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:02.955 DEBUG:boto:Port: 8000 2023-05-24T21:27:02.955 DEBUG:boto:Params: {} 2023-05-24T21:27:02.956 DEBUG:boto:Token: None 2023-05-24T21:27:02.956 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:02 GMT /qskrps-46/obj3?versionId=null 2023-05-24T21:27:02.957 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:oAlpau0V3h5VGREKuuPhALu4qmY= 2023-05-24T21:27:02.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:oAlpau0V3h5VGREKuuPhALu4qmY=', 'Content-Length': '0'} 2023-05-24T21:27:02.958 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004a6018f92c8ff5dc-00646e8126-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:02.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-24T21:27:02.959 DEBUG:boto:path=/qskrps-46/obj3 2023-05-24T21:27:02.960 DEBUG:boto:auth_path=/qskrps-46/obj3 2023-05-24T21:27:02.960 DEBUG:boto:path=/qskrps-46/obj3?versionId=null 2023-05-24T21:27:02.960 DEBUG:boto:auth_path=/qskrps-46/obj3?versionId=null 2023-05-24T21:27:02.961 DEBUG:boto:Method: GET 2023-05-24T21:27:02.961 DEBUG:boto:Path: /qskrps-46/obj3?versionId=null 2023-05-24T21:27:02.962 DEBUG:boto:Data: 2023-05-24T21:27:02.962 DEBUG:boto:Headers: {} 2023-05-24T21:27:02.962 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:02.963 DEBUG:boto:Port: 8001 2023-05-24T21:27:02.963 DEBUG:boto:Params: {} 2023-05-24T21:27:02.963 DEBUG:boto:Token: None 2023-05-24T21:27:02.964 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:02 GMT /qskrps-46/obj3?versionId=null 2023-05-24T21:27:02.964 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:3ykA+tX40+X8T4cjFotqswLd9ek= 2023-05-24T21:27:02.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:3ykA+tX40+X8T4cjFotqswLd9ek=', 'Content-Length': '0'} 2023-05-24T21:27:02.966 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f67eee4334a138fd-00646e8126-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:02.967 DEBUG:boto:path=/qskrps-46/obj3 2023-05-24T21:27:02.967 DEBUG:boto:auth_path=/qskrps-46/obj3 2023-05-24T21:27:02.968 DEBUG:boto:path=/qskrps-46/obj3?versionId=null 2023-05-24T21:27:02.968 DEBUG:boto:auth_path=/qskrps-46/obj3?versionId=null 2023-05-24T21:27:02.968 DEBUG:boto:Method: GET 2023-05-24T21:27:02.969 DEBUG:boto:Path: /qskrps-46/obj3?versionId=null 2023-05-24T21:27:02.969 DEBUG:boto:Data: 2023-05-24T21:27:02.969 DEBUG:boto:Headers: {} 2023-05-24T21:27:02.970 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:02.970 DEBUG:boto:Port: 8000 2023-05-24T21:27:02.970 DEBUG:boto:Params: {} 2023-05-24T21:27:02.971 DEBUG:boto:Token: None 2023-05-24T21:27:02.971 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:02 GMT /qskrps-46/obj3?versionId=null 2023-05-24T21:27:02.972 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:3ykA+tX40+X8T4cjFotqswLd9ek= 2023-05-24T21:27:02.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:3ykA+tX40+X8T4cjFotqswLd9ek=', 'Content-Length': '0'} 2023-05-24T21:27:02.974 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000012e5f53ba2f6aabb-00646e8126-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.016 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:27:03.017 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:27:03.017 DEBUG:boto:path=/qskrps-46/?prefix=obj3 2023-05-24T21:27:03.017 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj3 2023-05-24T21:27:03.018 DEBUG:boto:Method: GET 2023-05-24T21:27:03.018 DEBUG:boto:Path: /qskrps-46/?prefix=obj3 2023-05-24T21:27:03.018 DEBUG:boto:Data: 2023-05-24T21:27:03.019 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.019 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:03.020 DEBUG:boto:Port: 8001 2023-05-24T21:27:03.020 DEBUG:boto:Params: {} 2023-05-24T21:27:03.020 DEBUG:boto:Token: None 2023-05-24T21:27:03.021 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/ 2023-05-24T21:27:03.021 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I= 2023-05-24T21:27:03.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I=', 'Content-Length': '0'} 2023-05-24T21:27:03.024 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f99f3a62a2f662a-00646e8127-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.025 DEBUG:boto:b'qskrps-46obj31000falseobj32023-05-24T21:25:48.968Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:03.025 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:27:03.026 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:27:03.026 DEBUG:boto:path=/qskrps-46/?prefix=obj3 2023-05-24T21:27:03.026 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj3 2023-05-24T21:27:03.027 DEBUG:boto:Method: GET 2023-05-24T21:27:03.027 DEBUG:boto:Path: /qskrps-46/?prefix=obj3 2023-05-24T21:27:03.028 DEBUG:boto:Data: 2023-05-24T21:27:03.028 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.028 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:03.029 DEBUG:boto:Port: 8000 2023-05-24T21:27:03.029 DEBUG:boto:Params: {} 2023-05-24T21:27:03.029 DEBUG:boto:Token: None 2023-05-24T21:27:03.030 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/ 2023-05-24T21:27:03.030 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I= 2023-05-24T21:27:03.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I=', 'Content-Length': '0'} 2023-05-24T21:27:03.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e70e4723988b79e6-00646e8127-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.034 DEBUG:boto:b'qskrps-46obj31000falseobj32023-05-24T21:25:48.968Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:03.034 DEBUG:boto:path=/qskrps-46/obj3 2023-05-24T21:27:03.035 DEBUG:boto:auth_path=/qskrps-46/obj3 2023-05-24T21:27:03.035 DEBUG:boto:Method: HEAD 2023-05-24T21:27:03.035 DEBUG:boto:Path: /qskrps-46/obj3 2023-05-24T21:27:03.036 DEBUG:boto:Data: 2023-05-24T21:27:03.036 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.036 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:03.037 DEBUG:boto:Port: 8001 2023-05-24T21:27:03.037 DEBUG:boto:Params: {} 2023-05-24T21:27:03.037 DEBUG:boto:Token: None 2023-05-24T21:27:03.038 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj3 2023-05-24T21:27:03.038 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:vs1P+vA+5RH8FMSRbOHWGbdftOg= 2023-05-24T21:27:03.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:vs1P+vA+5RH8FMSRbOHWGbdftOg=', 'Content-Length': '0'} 2023-05-24T21:27:03.040 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001c9d1682a7ede9de-00646e8127-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.041 DEBUG:boto:path=/qskrps-46/obj3 2023-05-24T21:27:03.041 DEBUG:boto:auth_path=/qskrps-46/obj3 2023-05-24T21:27:03.041 DEBUG:boto:Method: HEAD 2023-05-24T21:27:03.042 DEBUG:boto:Path: /qskrps-46/obj3 2023-05-24T21:27:03.042 DEBUG:boto:Data: 2023-05-24T21:27:03.042 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.043 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:03.043 DEBUG:boto:Port: 8000 2023-05-24T21:27:03.043 DEBUG:boto:Params: {} 2023-05-24T21:27:03.044 DEBUG:boto:Token: None 2023-05-24T21:27:03.044 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj3 2023-05-24T21:27:03.044 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:vs1P+vA+5RH8FMSRbOHWGbdftOg= 2023-05-24T21:27:03.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:vs1P+vA+5RH8FMSRbOHWGbdftOg=', 'Content-Length': '0'} 2023-05-24T21:27:03.046 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004c5b50864c9b2da6-00646e8127-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-24T21:27:03.047 DEBUG:boto:path=/qskrps-46/obj3 2023-05-24T21:27:03.048 DEBUG:boto:auth_path=/qskrps-46/obj3 2023-05-24T21:27:03.048 DEBUG:boto:Method: GET 2023-05-24T21:27:03.049 DEBUG:boto:Path: /qskrps-46/obj3 2023-05-24T21:27:03.049 DEBUG:boto:Data: 2023-05-24T21:27:03.049 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.050 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:03.050 DEBUG:boto:Port: 8001 2023-05-24T21:27:03.050 DEBUG:boto:Params: {} 2023-05-24T21:27:03.051 DEBUG:boto:Token: None 2023-05-24T21:27:03.051 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj3 2023-05-24T21:27:03.052 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:WCxVO+mEBz4gnY/LUeFECBHXlGk= 2023-05-24T21:27:03.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WCxVO+mEBz4gnY/LUeFECBHXlGk=', 'Content-Length': '0'} 2023-05-24T21:27:03.054 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f426a4d8d98a7e4c-00646e8127-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.096 DEBUG:boto:path=/qskrps-46/obj3 2023-05-24T21:27:03.097 DEBUG:boto:auth_path=/qskrps-46/obj3 2023-05-24T21:27:03.097 DEBUG:boto:Method: GET 2023-05-24T21:27:03.097 DEBUG:boto:Path: /qskrps-46/obj3 2023-05-24T21:27:03.098 DEBUG:boto:Data: 2023-05-24T21:27:03.098 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.098 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:03.099 DEBUG:boto:Port: 8000 2023-05-24T21:27:03.099 DEBUG:boto:Params: {} 2023-05-24T21:27:03.100 DEBUG:boto:Token: None 2023-05-24T21:27:03.100 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj3 2023-05-24T21:27:03.100 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:WCxVO+mEBz4gnY/LUeFECBHXlGk= 2023-05-24T21:27:03.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WCxVO+mEBz4gnY/LUeFECBHXlGk=', 'Content-Length': '0'} 2023-05-24T21:27:03.103 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008a6813da4440a486-00646e8127-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.103 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:27:03.104 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:27:03.104 DEBUG:boto:path=/qskrps-46/?prefix=obj3 2023-05-24T21:27:03.104 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj3 2023-05-24T21:27:03.105 DEBUG:boto:Method: GET 2023-05-24T21:27:03.105 DEBUG:boto:Path: /qskrps-46/?prefix=obj3 2023-05-24T21:27:03.106 DEBUG:boto:Data: 2023-05-24T21:27:03.106 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.106 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:03.107 DEBUG:boto:Port: 8001 2023-05-24T21:27:03.107 DEBUG:boto:Params: {} 2023-05-24T21:27:03.107 DEBUG:boto:Token: None 2023-05-24T21:27:03.108 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/ 2023-05-24T21:27:03.108 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I= 2023-05-24T21:27:03.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I=', 'Content-Length': '0'} 2023-05-24T21:27:03.111 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000541c058826ef8e2e-00646e8127-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.111 DEBUG:boto:b'qskrps-46obj31000falseobj32023-05-24T21:25:48.968Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:03.112 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:27:03.112 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:27:03.113 DEBUG:boto:path=/qskrps-46/?prefix=obj3 2023-05-24T21:27:03.113 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj3 2023-05-24T21:27:03.113 DEBUG:boto:Method: GET 2023-05-24T21:27:03.114 DEBUG:boto:Path: /qskrps-46/?prefix=obj3 2023-05-24T21:27:03.114 DEBUG:boto:Data: 2023-05-24T21:27:03.115 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.115 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:03.115 DEBUG:boto:Port: 8000 2023-05-24T21:27:03.116 DEBUG:boto:Params: {} 2023-05-24T21:27:03.116 DEBUG:boto:Token: None 2023-05-24T21:27:03.116 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/ 2023-05-24T21:27:03.117 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I= 2023-05-24T21:27:03.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I=', 'Content-Length': '0'} 2023-05-24T21:27:03.120 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d651641e99c5f8a3-00646e8127-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.120 DEBUG:boto:b'qskrps-46obj31000falseobj32023-05-24T21:25:48.968Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:03.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-24T21:27:03.121 DEBUG:boto:path=/qskrps-46/obj4 2023-05-24T21:27:03.122 DEBUG:boto:auth_path=/qskrps-46/obj4 2023-05-24T21:27:03.122 DEBUG:boto:path=/qskrps-46/obj4?versionId=null 2023-05-24T21:27:03.123 DEBUG:boto:auth_path=/qskrps-46/obj4?versionId=null 2023-05-24T21:27:03.123 DEBUG:boto:Method: GET 2023-05-24T21:27:03.123 DEBUG:boto:Path: /qskrps-46/obj4?versionId=null 2023-05-24T21:27:03.124 DEBUG:boto:Data: 2023-05-24T21:27:03.124 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.124 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:03.125 DEBUG:boto:Port: 8001 2023-05-24T21:27:03.125 DEBUG:boto:Params: {} 2023-05-24T21:27:03.125 DEBUG:boto:Token: None 2023-05-24T21:27:03.126 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj4?versionId=null 2023-05-24T21:27:03.126 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ErHDDvyn6LmbNRie5lJY7bBpqhw= 2023-05-24T21:27:03.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ErHDDvyn6LmbNRie5lJY7bBpqhw=', 'Content-Length': '0'} 2023-05-24T21:27:03.129 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001594e317ed3f44b9-00646e8127-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.129 DEBUG:boto:path=/qskrps-46/obj4 2023-05-24T21:27:03.130 DEBUG:boto:auth_path=/qskrps-46/obj4 2023-05-24T21:27:03.130 DEBUG:boto:path=/qskrps-46/obj4?versionId=null 2023-05-24T21:27:03.130 DEBUG:boto:auth_path=/qskrps-46/obj4?versionId=null 2023-05-24T21:27:03.131 DEBUG:boto:Method: GET 2023-05-24T21:27:03.131 DEBUG:boto:Path: /qskrps-46/obj4?versionId=null 2023-05-24T21:27:03.132 DEBUG:boto:Data: 2023-05-24T21:27:03.132 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.132 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:03.133 DEBUG:boto:Port: 8000 2023-05-24T21:27:03.133 DEBUG:boto:Params: {} 2023-05-24T21:27:03.133 DEBUG:boto:Token: None 2023-05-24T21:27:03.134 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj4?versionId=null 2023-05-24T21:27:03.134 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ErHDDvyn6LmbNRie5lJY7bBpqhw= 2023-05-24T21:27:03.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ErHDDvyn6LmbNRie5lJY7bBpqhw=', 'Content-Length': '0'} 2023-05-24T21:27:03.136 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000caeadfaa585c476a-00646e8127-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.180 DEBUG:boto:path=/qskrps-46/obj4 2023-05-24T21:27:03.181 DEBUG:boto:auth_path=/qskrps-46/obj4 2023-05-24T21:27:03.181 DEBUG:boto:path=/qskrps-46/obj4?versionId=null 2023-05-24T21:27:03.181 DEBUG:boto:auth_path=/qskrps-46/obj4?versionId=null 2023-05-24T21:27:03.182 DEBUG:boto:Method: HEAD 2023-05-24T21:27:03.182 DEBUG:boto:Path: /qskrps-46/obj4?versionId=null 2023-05-24T21:27:03.182 DEBUG:boto:Data: 2023-05-24T21:27:03.183 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.183 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:03.183 DEBUG:boto:Port: 8001 2023-05-24T21:27:03.184 DEBUG:boto:Params: {} 2023-05-24T21:27:03.184 DEBUG:boto:Token: None 2023-05-24T21:27:03.184 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj4?versionId=null 2023-05-24T21:27:03.185 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:+HTXF2l5Lvnnxn7LQI1d4cvfO1s= 2023-05-24T21:27:03.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:+HTXF2l5Lvnnxn7LQI1d4cvfO1s=', 'Content-Length': '0'} 2023-05-24T21:27:03.187 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a7504fcf7610f94a-00646e8127-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.188 DEBUG:boto:path=/qskrps-46/obj4 2023-05-24T21:27:03.188 DEBUG:boto:auth_path=/qskrps-46/obj4 2023-05-24T21:27:03.188 DEBUG:boto:path=/qskrps-46/obj4?versionId=null 2023-05-24T21:27:03.189 DEBUG:boto:auth_path=/qskrps-46/obj4?versionId=null 2023-05-24T21:27:03.189 DEBUG:boto:Method: HEAD 2023-05-24T21:27:03.189 DEBUG:boto:Path: /qskrps-46/obj4?versionId=null 2023-05-24T21:27:03.190 DEBUG:boto:Data: 2023-05-24T21:27:03.190 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.190 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:03.191 DEBUG:boto:Port: 8000 2023-05-24T21:27:03.191 DEBUG:boto:Params: {} 2023-05-24T21:27:03.191 DEBUG:boto:Token: None 2023-05-24T21:27:03.192 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj4?versionId=null 2023-05-24T21:27:03.192 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:+HTXF2l5Lvnnxn7LQI1d4cvfO1s= 2023-05-24T21:27:03.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:+HTXF2l5Lvnnxn7LQI1d4cvfO1s=', 'Content-Length': '0'} 2023-05-24T21:27:03.194 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000026cd8219e0fb52f7-00646e8127-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-24T21:27:03.195 DEBUG:boto:path=/qskrps-46/obj4 2023-05-24T21:27:03.196 DEBUG:boto:auth_path=/qskrps-46/obj4 2023-05-24T21:27:03.196 DEBUG:boto:path=/qskrps-46/obj4?versionId=null 2023-05-24T21:27:03.197 DEBUG:boto:auth_path=/qskrps-46/obj4?versionId=null 2023-05-24T21:27:03.197 DEBUG:boto:Method: GET 2023-05-24T21:27:03.197 DEBUG:boto:Path: /qskrps-46/obj4?versionId=null 2023-05-24T21:27:03.198 DEBUG:boto:Data: 2023-05-24T21:27:03.198 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.198 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:03.199 DEBUG:boto:Port: 8001 2023-05-24T21:27:03.199 DEBUG:boto:Params: {} 2023-05-24T21:27:03.199 DEBUG:boto:Token: None 2023-05-24T21:27:03.200 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj4?versionId=null 2023-05-24T21:27:03.200 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ErHDDvyn6LmbNRie5lJY7bBpqhw= 2023-05-24T21:27:03.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ErHDDvyn6LmbNRie5lJY7bBpqhw=', 'Content-Length': '0'} 2023-05-24T21:27:03.202 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000096a4ba5256f87969-00646e8127-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.203 DEBUG:boto:path=/qskrps-46/obj4 2023-05-24T21:27:03.203 DEBUG:boto:auth_path=/qskrps-46/obj4 2023-05-24T21:27:03.204 DEBUG:boto:path=/qskrps-46/obj4?versionId=null 2023-05-24T21:27:03.204 DEBUG:boto:auth_path=/qskrps-46/obj4?versionId=null 2023-05-24T21:27:03.205 DEBUG:boto:Method: GET 2023-05-24T21:27:03.205 DEBUG:boto:Path: /qskrps-46/obj4?versionId=null 2023-05-24T21:27:03.205 DEBUG:boto:Data: 2023-05-24T21:27:03.206 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.206 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:03.206 DEBUG:boto:Port: 8000 2023-05-24T21:27:03.207 DEBUG:boto:Params: {} 2023-05-24T21:27:03.207 DEBUG:boto:Token: None 2023-05-24T21:27:03.208 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj4?versionId=null 2023-05-24T21:27:03.208 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ErHDDvyn6LmbNRie5lJY7bBpqhw= 2023-05-24T21:27:03.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ErHDDvyn6LmbNRie5lJY7bBpqhw=', 'Content-Length': '0'} 2023-05-24T21:27:03.210 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ecd01aa0f1cbda95-00646e8127-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.211 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:27:03.211 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:27:03.212 DEBUG:boto:path=/qskrps-46/?prefix=obj4 2023-05-24T21:27:03.212 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj4 2023-05-24T21:27:03.212 DEBUG:boto:Method: GET 2023-05-24T21:27:03.213 DEBUG:boto:Path: /qskrps-46/?prefix=obj4 2023-05-24T21:27:03.213 DEBUG:boto:Data: 2023-05-24T21:27:03.213 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.214 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:03.214 DEBUG:boto:Port: 8001 2023-05-24T21:27:03.214 DEBUG:boto:Params: {} 2023-05-24T21:27:03.215 DEBUG:boto:Token: None 2023-05-24T21:27:03.215 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/ 2023-05-24T21:27:03.216 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I= 2023-05-24T21:27:03.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I=', 'Content-Length': '0'} 2023-05-24T21:27:03.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065d05c328d6a2ec6-00646e8127-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.219 DEBUG:boto:b'qskrps-46obj41000falseobj42023-05-24T21:25:48.995Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:03.219 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:27:03.220 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:27:03.220 DEBUG:boto:path=/qskrps-46/?prefix=obj4 2023-05-24T21:27:03.221 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj4 2023-05-24T21:27:03.221 DEBUG:boto:Method: GET 2023-05-24T21:27:03.221 DEBUG:boto:Path: /qskrps-46/?prefix=obj4 2023-05-24T21:27:03.222 DEBUG:boto:Data: 2023-05-24T21:27:03.222 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.222 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:03.223 DEBUG:boto:Port: 8000 2023-05-24T21:27:03.223 DEBUG:boto:Params: {} 2023-05-24T21:27:03.223 DEBUG:boto:Token: None 2023-05-24T21:27:03.224 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/ 2023-05-24T21:27:03.224 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I= 2023-05-24T21:27:03.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I=', 'Content-Length': '0'} 2023-05-24T21:27:03.227 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce8db99f3af76951-00646e8127-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.227 DEBUG:boto:b'qskrps-46obj41000falseobj42023-05-24T21:25:48.995Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:03.228 DEBUG:boto:path=/qskrps-46/obj4 2023-05-24T21:27:03.228 DEBUG:boto:auth_path=/qskrps-46/obj4 2023-05-24T21:27:03.229 DEBUG:boto:Method: HEAD 2023-05-24T21:27:03.229 DEBUG:boto:Path: /qskrps-46/obj4 2023-05-24T21:27:03.229 DEBUG:boto:Data: 2023-05-24T21:27:03.230 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.230 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:03.230 DEBUG:boto:Port: 8001 2023-05-24T21:27:03.231 DEBUG:boto:Params: {} 2023-05-24T21:27:03.231 DEBUG:boto:Token: None 2023-05-24T21:27:03.231 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj4 2023-05-24T21:27:03.232 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:hGTd1gdHg7NuAZit9YZTJOCYNfo= 2023-05-24T21:27:03.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:hGTd1gdHg7NuAZit9YZTJOCYNfo=', 'Content-Length': '0'} 2023-05-24T21:27:03.234 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000045b775a0883a7000-00646e8127-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.235 DEBUG:boto:path=/qskrps-46/obj4 2023-05-24T21:27:03.235 DEBUG:boto:auth_path=/qskrps-46/obj4 2023-05-24T21:27:03.235 DEBUG:boto:Method: HEAD 2023-05-24T21:27:03.236 DEBUG:boto:Path: /qskrps-46/obj4 2023-05-24T21:27:03.236 DEBUG:boto:Data: 2023-05-24T21:27:03.236 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.237 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:03.237 DEBUG:boto:Port: 8000 2023-05-24T21:27:03.237 DEBUG:boto:Params: {} 2023-05-24T21:27:03.238 DEBUG:boto:Token: None 2023-05-24T21:27:03.238 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj4 2023-05-24T21:27:03.238 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:hGTd1gdHg7NuAZit9YZTJOCYNfo= 2023-05-24T21:27:03.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:hGTd1gdHg7NuAZit9YZTJOCYNfo=', 'Content-Length': '0'} 2023-05-24T21:27:03.241 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000602ccc61b98aa393-00646e8127-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-24T21:27:03.241 DEBUG:boto:path=/qskrps-46/obj4 2023-05-24T21:27:03.242 DEBUG:boto:auth_path=/qskrps-46/obj4 2023-05-24T21:27:03.242 DEBUG:boto:Method: GET 2023-05-24T21:27:03.243 DEBUG:boto:Path: /qskrps-46/obj4 2023-05-24T21:27:03.243 DEBUG:boto:Data: 2023-05-24T21:27:03.243 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.244 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:03.244 DEBUG:boto:Port: 8001 2023-05-24T21:27:03.244 DEBUG:boto:Params: {} 2023-05-24T21:27:03.245 DEBUG:boto:Token: None 2023-05-24T21:27:03.245 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj4 2023-05-24T21:27:03.246 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Dzdtpx1VmUezUBRSXzDV4n1QG5k= 2023-05-24T21:27:03.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Dzdtpx1VmUezUBRSXzDV4n1QG5k=', 'Content-Length': '0'} 2023-05-24T21:27:03.248 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000001fc25e31a46604d-00646e8127-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.248 DEBUG:boto:path=/qskrps-46/obj4 2023-05-24T21:27:03.249 DEBUG:boto:auth_path=/qskrps-46/obj4 2023-05-24T21:27:03.249 DEBUG:boto:Method: GET 2023-05-24T21:27:03.249 DEBUG:boto:Path: /qskrps-46/obj4 2023-05-24T21:27:03.250 DEBUG:boto:Data: 2023-05-24T21:27:03.250 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.250 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:03.251 DEBUG:boto:Port: 8000 2023-05-24T21:27:03.251 DEBUG:boto:Params: {} 2023-05-24T21:27:03.252 DEBUG:boto:Token: None 2023-05-24T21:27:03.252 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj4 2023-05-24T21:27:03.252 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Dzdtpx1VmUezUBRSXzDV4n1QG5k= 2023-05-24T21:27:03.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Dzdtpx1VmUezUBRSXzDV4n1QG5k=', 'Content-Length': '0'} 2023-05-24T21:27:03.254 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb924eb79695b6c7-00646e8127-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.255 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:27:03.256 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:27:03.256 DEBUG:boto:path=/qskrps-46/?prefix=obj4 2023-05-24T21:27:03.256 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj4 2023-05-24T21:27:03.257 DEBUG:boto:Method: GET 2023-05-24T21:27:03.257 DEBUG:boto:Path: /qskrps-46/?prefix=obj4 2023-05-24T21:27:03.257 DEBUG:boto:Data: 2023-05-24T21:27:03.258 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.258 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:03.258 DEBUG:boto:Port: 8001 2023-05-24T21:27:03.259 DEBUG:boto:Params: {} 2023-05-24T21:27:03.259 DEBUG:boto:Token: None 2023-05-24T21:27:03.260 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/ 2023-05-24T21:27:03.260 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I= 2023-05-24T21:27:03.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I=', 'Content-Length': '0'} 2023-05-24T21:27:03.263 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a442f14abcd594b6-00646e8127-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.263 DEBUG:boto:b'qskrps-46obj41000falseobj42023-05-24T21:25:48.995Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:03.264 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:27:03.264 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:27:03.265 DEBUG:boto:path=/qskrps-46/?prefix=obj4 2023-05-24T21:27:03.265 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj4 2023-05-24T21:27:03.265 DEBUG:boto:Method: GET 2023-05-24T21:27:03.266 DEBUG:boto:Path: /qskrps-46/?prefix=obj4 2023-05-24T21:27:03.266 DEBUG:boto:Data: 2023-05-24T21:27:03.266 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.267 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:03.267 DEBUG:boto:Port: 8000 2023-05-24T21:27:03.267 DEBUG:boto:Params: {} 2023-05-24T21:27:03.268 DEBUG:boto:Token: None 2023-05-24T21:27:03.268 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/ 2023-05-24T21:27:03.269 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I= 2023-05-24T21:27:03.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I=', 'Content-Length': '0'} 2023-05-24T21:27:03.271 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d110d2560a8bf46-00646e8127-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.272 DEBUG:boto:b'qskrps-46obj41000falseobj42023-05-24T21:25:48.995Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:03.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-05-24T21:27:03.273 DEBUG:boto:path=/qskrps-46/obj5 2023-05-24T21:27:03.273 DEBUG:boto:auth_path=/qskrps-46/obj5 2023-05-24T21:27:03.273 DEBUG:boto:path=/qskrps-46/obj5?versionId=null 2023-05-24T21:27:03.274 DEBUG:boto:auth_path=/qskrps-46/obj5?versionId=null 2023-05-24T21:27:03.274 DEBUG:boto:Method: GET 2023-05-24T21:27:03.274 DEBUG:boto:Path: /qskrps-46/obj5?versionId=null 2023-05-24T21:27:03.275 DEBUG:boto:Data: 2023-05-24T21:27:03.275 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.276 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:03.276 DEBUG:boto:Port: 8001 2023-05-24T21:27:03.276 DEBUG:boto:Params: {} 2023-05-24T21:27:03.277 DEBUG:boto:Token: None 2023-05-24T21:27:03.277 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj5?versionId=null 2023-05-24T21:27:03.277 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:vjSWJ+WvnolVhsts0bZiI3R4bbg= 2023-05-24T21:27:03.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:vjSWJ+WvnolVhsts0bZiI3R4bbg=', 'Content-Length': '0'} 2023-05-24T21:27:03.280 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ebfd00d8ae94cc47-00646e8127-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.281 DEBUG:boto:path=/qskrps-46/obj5 2023-05-24T21:27:03.281 DEBUG:boto:auth_path=/qskrps-46/obj5 2023-05-24T21:27:03.281 DEBUG:boto:path=/qskrps-46/obj5?versionId=null 2023-05-24T21:27:03.282 DEBUG:boto:auth_path=/qskrps-46/obj5?versionId=null 2023-05-24T21:27:03.282 DEBUG:boto:Method: GET 2023-05-24T21:27:03.283 DEBUG:boto:Path: /qskrps-46/obj5?versionId=null 2023-05-24T21:27:03.283 DEBUG:boto:Data: 2023-05-24T21:27:03.283 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.284 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:03.284 DEBUG:boto:Port: 8000 2023-05-24T21:27:03.284 DEBUG:boto:Params: {} 2023-05-24T21:27:03.285 DEBUG:boto:Token: None 2023-05-24T21:27:03.285 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj5?versionId=null 2023-05-24T21:27:03.286 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:vjSWJ+WvnolVhsts0bZiI3R4bbg= 2023-05-24T21:27:03.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:vjSWJ+WvnolVhsts0bZiI3R4bbg=', 'Content-Length': '0'} 2023-05-24T21:27:03.288 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000664b32480c7911fb-00646e8127-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.289 DEBUG:boto:path=/qskrps-46/obj5 2023-05-24T21:27:03.289 DEBUG:boto:auth_path=/qskrps-46/obj5 2023-05-24T21:27:03.289 DEBUG:boto:path=/qskrps-46/obj5?versionId=null 2023-05-24T21:27:03.290 DEBUG:boto:auth_path=/qskrps-46/obj5?versionId=null 2023-05-24T21:27:03.290 DEBUG:boto:Method: HEAD 2023-05-24T21:27:03.290 DEBUG:boto:Path: /qskrps-46/obj5?versionId=null 2023-05-24T21:27:03.291 DEBUG:boto:Data: 2023-05-24T21:27:03.291 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.291 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:03.292 DEBUG:boto:Port: 8001 2023-05-24T21:27:03.292 DEBUG:boto:Params: {} 2023-05-24T21:27:03.292 DEBUG:boto:Token: None 2023-05-24T21:27:03.293 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj5?versionId=null 2023-05-24T21:27:03.293 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5yHPtN/R8dYG1t2/OwXp2Ky9oVo= 2023-05-24T21:27:03.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5yHPtN/R8dYG1t2/OwXp2Ky9oVo=', 'Content-Length': '0'} 2023-05-24T21:27:03.295 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e75fa91b55337244-00646e8127-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.296 DEBUG:boto:path=/qskrps-46/obj5 2023-05-24T21:27:03.296 DEBUG:boto:auth_path=/qskrps-46/obj5 2023-05-24T21:27:03.297 DEBUG:boto:path=/qskrps-46/obj5?versionId=null 2023-05-24T21:27:03.297 DEBUG:boto:auth_path=/qskrps-46/obj5?versionId=null 2023-05-24T21:27:03.297 DEBUG:boto:Method: HEAD 2023-05-24T21:27:03.298 DEBUG:boto:Path: /qskrps-46/obj5?versionId=null 2023-05-24T21:27:03.298 DEBUG:boto:Data: 2023-05-24T21:27:03.298 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.299 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:03.299 DEBUG:boto:Port: 8000 2023-05-24T21:27:03.299 DEBUG:boto:Params: {} 2023-05-24T21:27:03.300 DEBUG:boto:Token: None 2023-05-24T21:27:03.300 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj5?versionId=null 2023-05-24T21:27:03.301 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5yHPtN/R8dYG1t2/OwXp2Ky9oVo= 2023-05-24T21:27:03.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5yHPtN/R8dYG1t2/OwXp2Ky9oVo=', 'Content-Length': '0'} 2023-05-24T21:27:03.303 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000061f08bb067051d93-00646e8127-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-05-24T21:27:03.304 DEBUG:boto:path=/qskrps-46/obj5 2023-05-24T21:27:03.304 DEBUG:boto:auth_path=/qskrps-46/obj5 2023-05-24T21:27:03.304 DEBUG:boto:path=/qskrps-46/obj5?versionId=null 2023-05-24T21:27:03.305 DEBUG:boto:auth_path=/qskrps-46/obj5?versionId=null 2023-05-24T21:27:03.305 DEBUG:boto:Method: GET 2023-05-24T21:27:03.306 DEBUG:boto:Path: /qskrps-46/obj5?versionId=null 2023-05-24T21:27:03.306 DEBUG:boto:Data: 2023-05-24T21:27:03.306 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.307 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:03.307 DEBUG:boto:Port: 8001 2023-05-24T21:27:03.307 DEBUG:boto:Params: {} 2023-05-24T21:27:03.308 DEBUG:boto:Token: None 2023-05-24T21:27:03.308 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj5?versionId=null 2023-05-24T21:27:03.309 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:vjSWJ+WvnolVhsts0bZiI3R4bbg= 2023-05-24T21:27:03.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:vjSWJ+WvnolVhsts0bZiI3R4bbg=', 'Content-Length': '0'} 2023-05-24T21:27:03.311 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a592fd0293f8213f-00646e8127-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.312 DEBUG:boto:path=/qskrps-46/obj5 2023-05-24T21:27:03.312 DEBUG:boto:auth_path=/qskrps-46/obj5 2023-05-24T21:27:03.312 DEBUG:boto:path=/qskrps-46/obj5?versionId=null 2023-05-24T21:27:03.313 DEBUG:boto:auth_path=/qskrps-46/obj5?versionId=null 2023-05-24T21:27:03.313 DEBUG:boto:Method: GET 2023-05-24T21:27:03.313 DEBUG:boto:Path: /qskrps-46/obj5?versionId=null 2023-05-24T21:27:03.314 DEBUG:boto:Data: 2023-05-24T21:27:03.314 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.314 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:03.315 DEBUG:boto:Port: 8000 2023-05-24T21:27:03.315 DEBUG:boto:Params: {} 2023-05-24T21:27:03.316 DEBUG:boto:Token: None 2023-05-24T21:27:03.316 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj5?versionId=null 2023-05-24T21:27:03.316 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:vjSWJ+WvnolVhsts0bZiI3R4bbg= 2023-05-24T21:27:03.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:vjSWJ+WvnolVhsts0bZiI3R4bbg=', 'Content-Length': '0'} 2023-05-24T21:27:03.319 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000327b10f3113ff651-00646e8127-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.319 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:27:03.320 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:27:03.320 DEBUG:boto:path=/qskrps-46/?prefix=obj5 2023-05-24T21:27:03.320 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj5 2023-05-24T21:27:03.321 DEBUG:boto:Method: GET 2023-05-24T21:27:03.321 DEBUG:boto:Path: /qskrps-46/?prefix=obj5 2023-05-24T21:27:03.322 DEBUG:boto:Data: 2023-05-24T21:27:03.322 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.322 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:03.323 DEBUG:boto:Port: 8001 2023-05-24T21:27:03.323 DEBUG:boto:Params: {} 2023-05-24T21:27:03.323 DEBUG:boto:Token: None 2023-05-24T21:27:03.324 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/ 2023-05-24T21:27:03.324 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I= 2023-05-24T21:27:03.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I=', 'Content-Length': '0'} 2023-05-24T21:27:03.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f16a7e414767cd33-00646e8127-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.327 DEBUG:boto:b'qskrps-46obj51000falseobj52023-05-24T21:26:22.064Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:03.328 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:27:03.329 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:27:03.329 DEBUG:boto:path=/qskrps-46/?prefix=obj5 2023-05-24T21:27:03.329 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj5 2023-05-24T21:27:03.330 DEBUG:boto:Method: GET 2023-05-24T21:27:03.330 DEBUG:boto:Path: /qskrps-46/?prefix=obj5 2023-05-24T21:27:03.330 DEBUG:boto:Data: 2023-05-24T21:27:03.331 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.331 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:03.331 DEBUG:boto:Port: 8000 2023-05-24T21:27:03.332 DEBUG:boto:Params: {} 2023-05-24T21:27:03.332 DEBUG:boto:Token: None 2023-05-24T21:27:03.333 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/ 2023-05-24T21:27:03.333 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I= 2023-05-24T21:27:03.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I=', 'Content-Length': '0'} 2023-05-24T21:27:03.336 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c1058859a069795-00646e8127-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.336 DEBUG:boto:b'qskrps-46obj51000falseobj52023-05-24T21:26:22.064Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:03.337 DEBUG:boto:path=/qskrps-46/obj5 2023-05-24T21:27:03.337 DEBUG:boto:auth_path=/qskrps-46/obj5 2023-05-24T21:27:03.338 DEBUG:boto:Method: HEAD 2023-05-24T21:27:03.338 DEBUG:boto:Path: /qskrps-46/obj5 2023-05-24T21:27:03.338 DEBUG:boto:Data: 2023-05-24T21:27:03.339 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.339 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:03.339 DEBUG:boto:Port: 8001 2023-05-24T21:27:03.340 DEBUG:boto:Params: {} 2023-05-24T21:27:03.340 DEBUG:boto:Token: None 2023-05-24T21:27:03.340 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj5 2023-05-24T21:27:03.341 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:kq4dHtcu3dMd2dew+FD0teKgwZU= 2023-05-24T21:27:03.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:kq4dHtcu3dMd2dew+FD0teKgwZU=', 'Content-Length': '0'} 2023-05-24T21:27:03.343 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000011f7c42f7b03584f-00646e8127-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.344 DEBUG:boto:path=/qskrps-46/obj5 2023-05-24T21:27:03.344 DEBUG:boto:auth_path=/qskrps-46/obj5 2023-05-24T21:27:03.344 DEBUG:boto:Method: HEAD 2023-05-24T21:27:03.345 DEBUG:boto:Path: /qskrps-46/obj5 2023-05-24T21:27:03.345 DEBUG:boto:Data: 2023-05-24T21:27:03.345 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.346 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:03.346 DEBUG:boto:Port: 8000 2023-05-24T21:27:03.346 DEBUG:boto:Params: {} 2023-05-24T21:27:03.347 DEBUG:boto:Token: None 2023-05-24T21:27:03.347 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj5 2023-05-24T21:27:03.348 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:kq4dHtcu3dMd2dew+FD0teKgwZU= 2023-05-24T21:27:03.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:kq4dHtcu3dMd2dew+FD0teKgwZU=', 'Content-Length': '0'} 2023-05-24T21:27:03.350 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b87a0e42c89732c8-00646e8127-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-05-24T21:27:03.351 DEBUG:boto:path=/qskrps-46/obj5 2023-05-24T21:27:03.352 DEBUG:boto:auth_path=/qskrps-46/obj5 2023-05-24T21:27:03.352 DEBUG:boto:Method: GET 2023-05-24T21:27:03.352 DEBUG:boto:Path: /qskrps-46/obj5 2023-05-24T21:27:03.353 DEBUG:boto:Data: 2023-05-24T21:27:03.353 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.353 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:03.354 DEBUG:boto:Port: 8001 2023-05-24T21:27:03.354 DEBUG:boto:Params: {} 2023-05-24T21:27:03.355 DEBUG:boto:Token: None 2023-05-24T21:27:03.355 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj5 2023-05-24T21:27:03.355 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:SvmN5Xdqa9nY4kWxG1W9YeTpjt0= 2023-05-24T21:27:03.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:SvmN5Xdqa9nY4kWxG1W9YeTpjt0=', 'Content-Length': '0'} 2023-05-24T21:27:03.357 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f5ded38618b4020-00646e8127-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.358 DEBUG:boto:path=/qskrps-46/obj5 2023-05-24T21:27:03.358 DEBUG:boto:auth_path=/qskrps-46/obj5 2023-05-24T21:27:03.359 DEBUG:boto:Method: GET 2023-05-24T21:27:03.359 DEBUG:boto:Path: /qskrps-46/obj5 2023-05-24T21:27:03.360 DEBUG:boto:Data: 2023-05-24T21:27:03.360 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.360 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:03.361 DEBUG:boto:Port: 8000 2023-05-24T21:27:03.361 DEBUG:boto:Params: {} 2023-05-24T21:27:03.361 DEBUG:boto:Token: None 2023-05-24T21:27:03.362 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj5 2023-05-24T21:27:03.362 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:SvmN5Xdqa9nY4kWxG1W9YeTpjt0= 2023-05-24T21:27:03.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:SvmN5Xdqa9nY4kWxG1W9YeTpjt0=', 'Content-Length': '0'} 2023-05-24T21:27:03.364 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000161e79d07d25401b-00646e8127-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.365 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:27:03.365 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:27:03.366 DEBUG:boto:path=/qskrps-46/?prefix=obj5 2023-05-24T21:27:03.366 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj5 2023-05-24T21:27:03.366 DEBUG:boto:Method: GET 2023-05-24T21:27:03.367 DEBUG:boto:Path: /qskrps-46/?prefix=obj5 2023-05-24T21:27:03.367 DEBUG:boto:Data: 2023-05-24T21:27:03.367 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.368 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:03.368 DEBUG:boto:Port: 8001 2023-05-24T21:27:03.368 DEBUG:boto:Params: {} 2023-05-24T21:27:03.369 DEBUG:boto:Token: None 2023-05-24T21:27:03.369 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/ 2023-05-24T21:27:03.370 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I= 2023-05-24T21:27:03.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I=', 'Content-Length': '0'} 2023-05-24T21:27:03.372 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a59219261f1c6e01-00646e8127-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.373 DEBUG:boto:b'qskrps-46obj51000falseobj52023-05-24T21:26:22.064Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:03.373 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:27:03.374 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:27:03.374 DEBUG:boto:path=/qskrps-46/?prefix=obj5 2023-05-24T21:27:03.375 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj5 2023-05-24T21:27:03.375 DEBUG:boto:Method: GET 2023-05-24T21:27:03.375 DEBUG:boto:Path: /qskrps-46/?prefix=obj5 2023-05-24T21:27:03.376 DEBUG:boto:Data: 2023-05-24T21:27:03.376 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.376 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:03.377 DEBUG:boto:Port: 8000 2023-05-24T21:27:03.377 DEBUG:boto:Params: {} 2023-05-24T21:27:03.377 DEBUG:boto:Token: None 2023-05-24T21:27:03.378 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/ 2023-05-24T21:27:03.378 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I= 2023-05-24T21:27:03.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I=', 'Content-Length': '0'} 2023-05-24T21:27:03.381 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047115f186eb05613-00646e8127-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.424 DEBUG:boto:b'qskrps-46obj51000falseobj52023-05-24T21:26:22.064Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:03.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-05-24T21:27:03.425 DEBUG:boto:path=/qskrps-46/obj6 2023-05-24T21:27:03.426 DEBUG:boto:auth_path=/qskrps-46/obj6 2023-05-24T21:27:03.426 DEBUG:boto:path=/qskrps-46/obj6?versionId=null 2023-05-24T21:27:03.427 DEBUG:boto:auth_path=/qskrps-46/obj6?versionId=null 2023-05-24T21:27:03.427 DEBUG:boto:Method: GET 2023-05-24T21:27:03.427 DEBUG:boto:Path: /qskrps-46/obj6?versionId=null 2023-05-24T21:27:03.428 DEBUG:boto:Data: 2023-05-24T21:27:03.428 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.428 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:03.429 DEBUG:boto:Port: 8001 2023-05-24T21:27:03.429 DEBUG:boto:Params: {} 2023-05-24T21:27:03.429 DEBUG:boto:Token: None 2023-05-24T21:27:03.430 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj6?versionId=null 2023-05-24T21:27:03.430 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:OTDN3E4CuwUZJbOK9FCksYBr/8U= 2023-05-24T21:27:03.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:OTDN3E4CuwUZJbOK9FCksYBr/8U=', 'Content-Length': '0'} 2023-05-24T21:27:03.433 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e5d68e433aeb75a-00646e8127-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.434 DEBUG:boto:path=/qskrps-46/obj6 2023-05-24T21:27:03.434 DEBUG:boto:auth_path=/qskrps-46/obj6 2023-05-24T21:27:03.434 DEBUG:boto:path=/qskrps-46/obj6?versionId=null 2023-05-24T21:27:03.435 DEBUG:boto:auth_path=/qskrps-46/obj6?versionId=null 2023-05-24T21:27:03.435 DEBUG:boto:Method: GET 2023-05-24T21:27:03.435 DEBUG:boto:Path: /qskrps-46/obj6?versionId=null 2023-05-24T21:27:03.436 DEBUG:boto:Data: 2023-05-24T21:27:03.436 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.437 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:03.437 DEBUG:boto:Port: 8000 2023-05-24T21:27:03.437 DEBUG:boto:Params: {} 2023-05-24T21:27:03.438 DEBUG:boto:Token: None 2023-05-24T21:27:03.438 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj6?versionId=null 2023-05-24T21:27:03.438 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:OTDN3E4CuwUZJbOK9FCksYBr/8U= 2023-05-24T21:27:03.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:OTDN3E4CuwUZJbOK9FCksYBr/8U=', 'Content-Length': '0'} 2023-05-24T21:27:03.441 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa799e40a326d4d9-00646e8127-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.441 DEBUG:boto:path=/qskrps-46/obj6 2023-05-24T21:27:03.442 DEBUG:boto:auth_path=/qskrps-46/obj6 2023-05-24T21:27:03.442 DEBUG:boto:path=/qskrps-46/obj6?versionId=null 2023-05-24T21:27:03.442 DEBUG:boto:auth_path=/qskrps-46/obj6?versionId=null 2023-05-24T21:27:03.443 DEBUG:boto:Method: HEAD 2023-05-24T21:27:03.443 DEBUG:boto:Path: /qskrps-46/obj6?versionId=null 2023-05-24T21:27:03.443 DEBUG:boto:Data: 2023-05-24T21:27:03.444 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.444 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:03.444 DEBUG:boto:Port: 8001 2023-05-24T21:27:03.445 DEBUG:boto:Params: {} 2023-05-24T21:27:03.445 DEBUG:boto:Token: None 2023-05-24T21:27:03.445 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj6?versionId=null 2023-05-24T21:27:03.446 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:h9ieroXPc6zQNAsV1J9LmgrIJwM= 2023-05-24T21:27:03.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:h9ieroXPc6zQNAsV1J9LmgrIJwM=', 'Content-Length': '0'} 2023-05-24T21:27:03.448 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000694f3bfd7c9bc40a-00646e8127-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.449 DEBUG:boto:path=/qskrps-46/obj6 2023-05-24T21:27:03.449 DEBUG:boto:auth_path=/qskrps-46/obj6 2023-05-24T21:27:03.449 DEBUG:boto:path=/qskrps-46/obj6?versionId=null 2023-05-24T21:27:03.450 DEBUG:boto:auth_path=/qskrps-46/obj6?versionId=null 2023-05-24T21:27:03.450 DEBUG:boto:Method: HEAD 2023-05-24T21:27:03.450 DEBUG:boto:Path: /qskrps-46/obj6?versionId=null 2023-05-24T21:27:03.451 DEBUG:boto:Data: 2023-05-24T21:27:03.451 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.451 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:03.452 DEBUG:boto:Port: 8000 2023-05-24T21:27:03.452 DEBUG:boto:Params: {} 2023-05-24T21:27:03.453 DEBUG:boto:Token: None 2023-05-24T21:27:03.453 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj6?versionId=null 2023-05-24T21:27:03.453 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:h9ieroXPc6zQNAsV1J9LmgrIJwM= 2023-05-24T21:27:03.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:h9ieroXPc6zQNAsV1J9LmgrIJwM=', 'Content-Length': '0'} 2023-05-24T21:27:03.456 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000383f30a118272e87-00646e8127-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-05-24T21:27:03.457 DEBUG:boto:path=/qskrps-46/obj6 2023-05-24T21:27:03.457 DEBUG:boto:auth_path=/qskrps-46/obj6 2023-05-24T21:27:03.457 DEBUG:boto:path=/qskrps-46/obj6?versionId=null 2023-05-24T21:27:03.458 DEBUG:boto:auth_path=/qskrps-46/obj6?versionId=null 2023-05-24T21:27:03.458 DEBUG:boto:Method: GET 2023-05-24T21:27:03.458 DEBUG:boto:Path: /qskrps-46/obj6?versionId=null 2023-05-24T21:27:03.459 DEBUG:boto:Data: 2023-05-24T21:27:03.459 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.459 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:03.460 DEBUG:boto:Port: 8001 2023-05-24T21:27:03.460 DEBUG:boto:Params: {} 2023-05-24T21:27:03.461 DEBUG:boto:Token: None 2023-05-24T21:27:03.461 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj6?versionId=null 2023-05-24T21:27:03.461 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:OTDN3E4CuwUZJbOK9FCksYBr/8U= 2023-05-24T21:27:03.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:OTDN3E4CuwUZJbOK9FCksYBr/8U=', 'Content-Length': '0'} 2023-05-24T21:27:03.464 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e3fac1453765e8ed-00646e8127-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.504 DEBUG:boto:path=/qskrps-46/obj6 2023-05-24T21:27:03.505 DEBUG:boto:auth_path=/qskrps-46/obj6 2023-05-24T21:27:03.505 DEBUG:boto:path=/qskrps-46/obj6?versionId=null 2023-05-24T21:27:03.506 DEBUG:boto:auth_path=/qskrps-46/obj6?versionId=null 2023-05-24T21:27:03.506 DEBUG:boto:Method: GET 2023-05-24T21:27:03.506 DEBUG:boto:Path: /qskrps-46/obj6?versionId=null 2023-05-24T21:27:03.507 DEBUG:boto:Data: 2023-05-24T21:27:03.507 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.507 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:03.508 DEBUG:boto:Port: 8000 2023-05-24T21:27:03.508 DEBUG:boto:Params: {} 2023-05-24T21:27:03.509 DEBUG:boto:Token: None 2023-05-24T21:27:03.509 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj6?versionId=null 2023-05-24T21:27:03.509 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:OTDN3E4CuwUZJbOK9FCksYBr/8U= 2023-05-24T21:27:03.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:OTDN3E4CuwUZJbOK9FCksYBr/8U=', 'Content-Length': '0'} 2023-05-24T21:27:03.512 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000033c10e97404676ad-00646e8127-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.513 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:27:03.513 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:27:03.513 DEBUG:boto:path=/qskrps-46/?prefix=obj6 2023-05-24T21:27:03.514 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj6 2023-05-24T21:27:03.514 DEBUG:boto:Method: GET 2023-05-24T21:27:03.514 DEBUG:boto:Path: /qskrps-46/?prefix=obj6 2023-05-24T21:27:03.515 DEBUG:boto:Data: 2023-05-24T21:27:03.515 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.515 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:03.516 DEBUG:boto:Port: 8001 2023-05-24T21:27:03.516 DEBUG:boto:Params: {} 2023-05-24T21:27:03.517 DEBUG:boto:Token: None 2023-05-24T21:27:03.517 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/ 2023-05-24T21:27:03.518 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I= 2023-05-24T21:27:03.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I=', 'Content-Length': '0'} 2023-05-24T21:27:03.520 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d320b52cc42dcf0f-00646e8127-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.521 DEBUG:boto:b'qskrps-46obj61000falseobj62023-05-24T21:26:22.081Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:03.521 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:27:03.522 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:27:03.522 DEBUG:boto:path=/qskrps-46/?prefix=obj6 2023-05-24T21:27:03.523 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj6 2023-05-24T21:27:03.523 DEBUG:boto:Method: GET 2023-05-24T21:27:03.523 DEBUG:boto:Path: /qskrps-46/?prefix=obj6 2023-05-24T21:27:03.524 DEBUG:boto:Data: 2023-05-24T21:27:03.524 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.524 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:03.525 DEBUG:boto:Port: 8000 2023-05-24T21:27:03.525 DEBUG:boto:Params: {} 2023-05-24T21:27:03.525 DEBUG:boto:Token: None 2023-05-24T21:27:03.526 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/ 2023-05-24T21:27:03.526 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I= 2023-05-24T21:27:03.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I=', 'Content-Length': '0'} 2023-05-24T21:27:03.529 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033e3dab698c954c6-00646e8127-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.530 DEBUG:boto:b'qskrps-46obj61000falseobj62023-05-24T21:26:22.081Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:03.530 DEBUG:boto:path=/qskrps-46/obj6 2023-05-24T21:27:03.531 DEBUG:boto:auth_path=/qskrps-46/obj6 2023-05-24T21:27:03.531 DEBUG:boto:Method: HEAD 2023-05-24T21:27:03.531 DEBUG:boto:Path: /qskrps-46/obj6 2023-05-24T21:27:03.532 DEBUG:boto:Data: 2023-05-24T21:27:03.532 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.533 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:03.533 DEBUG:boto:Port: 8001 2023-05-24T21:27:03.533 DEBUG:boto:Params: {} 2023-05-24T21:27:03.534 DEBUG:boto:Token: None 2023-05-24T21:27:03.534 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj6 2023-05-24T21:27:03.534 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:+eAlruIwoQ6JKtRufaF/PggmpGo= 2023-05-24T21:27:03.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:+eAlruIwoQ6JKtRufaF/PggmpGo=', 'Content-Length': '0'} 2023-05-24T21:27:03.537 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f040798970d846d1-00646e8127-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.537 DEBUG:boto:path=/qskrps-46/obj6 2023-05-24T21:27:03.537 DEBUG:boto:auth_path=/qskrps-46/obj6 2023-05-24T21:27:03.538 DEBUG:boto:Method: HEAD 2023-05-24T21:27:03.538 DEBUG:boto:Path: /qskrps-46/obj6 2023-05-24T21:27:03.539 DEBUG:boto:Data: 2023-05-24T21:27:03.539 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.539 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:03.540 DEBUG:boto:Port: 8000 2023-05-24T21:27:03.540 DEBUG:boto:Params: {} 2023-05-24T21:27:03.540 DEBUG:boto:Token: None 2023-05-24T21:27:03.541 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj6 2023-05-24T21:27:03.541 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:+eAlruIwoQ6JKtRufaF/PggmpGo= 2023-05-24T21:27:03.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:+eAlruIwoQ6JKtRufaF/PggmpGo=', 'Content-Length': '0'} 2023-05-24T21:27:03.543 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000022fca070e3209933-00646e8127-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-05-24T21:27:03.544 DEBUG:boto:path=/qskrps-46/obj6 2023-05-24T21:27:03.545 DEBUG:boto:auth_path=/qskrps-46/obj6 2023-05-24T21:27:03.545 DEBUG:boto:Method: GET 2023-05-24T21:27:03.545 DEBUG:boto:Path: /qskrps-46/obj6 2023-05-24T21:27:03.546 DEBUG:boto:Data: 2023-05-24T21:27:03.546 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.546 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:03.547 DEBUG:boto:Port: 8001 2023-05-24T21:27:03.547 DEBUG:boto:Params: {} 2023-05-24T21:27:03.548 DEBUG:boto:Token: None 2023-05-24T21:27:03.548 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj6 2023-05-24T21:27:03.548 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:z5HJVTQ25twUxku2B06ea9u9vdY= 2023-05-24T21:27:03.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:z5HJVTQ25twUxku2B06ea9u9vdY=', 'Content-Length': '0'} 2023-05-24T21:27:03.551 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fed237f33b699e67-00646e8127-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.592 DEBUG:boto:path=/qskrps-46/obj6 2023-05-24T21:27:03.593 DEBUG:boto:auth_path=/qskrps-46/obj6 2023-05-24T21:27:03.593 DEBUG:boto:Method: GET 2023-05-24T21:27:03.594 DEBUG:boto:Path: /qskrps-46/obj6 2023-05-24T21:27:03.594 DEBUG:boto:Data: 2023-05-24T21:27:03.594 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.595 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:03.595 DEBUG:boto:Port: 8000 2023-05-24T21:27:03.595 DEBUG:boto:Params: {} 2023-05-24T21:27:03.596 DEBUG:boto:Token: None 2023-05-24T21:27:03.596 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj6 2023-05-24T21:27:03.597 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:z5HJVTQ25twUxku2B06ea9u9vdY= 2023-05-24T21:27:03.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:z5HJVTQ25twUxku2B06ea9u9vdY=', 'Content-Length': '0'} 2023-05-24T21:27:03.599 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006dc8d88402dbdb02-00646e8127-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.600 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:27:03.600 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:27:03.601 DEBUG:boto:path=/qskrps-46/?prefix=obj6 2023-05-24T21:27:03.601 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj6 2023-05-24T21:27:03.601 DEBUG:boto:Method: GET 2023-05-24T21:27:03.602 DEBUG:boto:Path: /qskrps-46/?prefix=obj6 2023-05-24T21:27:03.602 DEBUG:boto:Data: 2023-05-24T21:27:03.603 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.603 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:03.603 DEBUG:boto:Port: 8001 2023-05-24T21:27:03.604 DEBUG:boto:Params: {} 2023-05-24T21:27:03.604 DEBUG:boto:Token: None 2023-05-24T21:27:03.604 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/ 2023-05-24T21:27:03.605 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I= 2023-05-24T21:27:03.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I=', 'Content-Length': '0'} 2023-05-24T21:27:03.608 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020d7f0aa7e9375c5-00646e8127-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.608 DEBUG:boto:b'qskrps-46obj61000falseobj62023-05-24T21:26:22.081Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:03.609 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:27:03.609 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:27:03.610 DEBUG:boto:path=/qskrps-46/?prefix=obj6 2023-05-24T21:27:03.610 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj6 2023-05-24T21:27:03.610 DEBUG:boto:Method: GET 2023-05-24T21:27:03.611 DEBUG:boto:Path: /qskrps-46/?prefix=obj6 2023-05-24T21:27:03.611 DEBUG:boto:Data: 2023-05-24T21:27:03.611 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.612 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:03.612 DEBUG:boto:Port: 8000 2023-05-24T21:27:03.613 DEBUG:boto:Params: {} 2023-05-24T21:27:03.613 DEBUG:boto:Token: None 2023-05-24T21:27:03.613 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/ 2023-05-24T21:27:03.614 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I= 2023-05-24T21:27:03.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I=', 'Content-Length': '0'} 2023-05-24T21:27:03.616 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b058bc0661e06f74-00646e8127-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.617 DEBUG:boto:b'qskrps-46obj61000falseobj62023-05-24T21:26:22.081Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:03.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-05-24T21:27:03.618 DEBUG:boto:path=/qskrps-46/obj7 2023-05-24T21:27:03.618 DEBUG:boto:auth_path=/qskrps-46/obj7 2023-05-24T21:27:03.619 DEBUG:boto:path=/qskrps-46/obj7?versionId=null 2023-05-24T21:27:03.619 DEBUG:boto:auth_path=/qskrps-46/obj7?versionId=null 2023-05-24T21:27:03.619 DEBUG:boto:Method: GET 2023-05-24T21:27:03.620 DEBUG:boto:Path: /qskrps-46/obj7?versionId=null 2023-05-24T21:27:03.620 DEBUG:boto:Data: 2023-05-24T21:27:03.620 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.621 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:03.621 DEBUG:boto:Port: 8001 2023-05-24T21:27:03.621 DEBUG:boto:Params: {} 2023-05-24T21:27:03.622 DEBUG:boto:Token: None 2023-05-24T21:27:03.622 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj7?versionId=null 2023-05-24T21:27:03.623 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:RimMsa+6vohKuCJWKmco3WlAHBA= 2023-05-24T21:27:03.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:RimMsa+6vohKuCJWKmco3WlAHBA=', 'Content-Length': '0'} 2023-05-24T21:27:03.625 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b6d773431a512ee-00646e8127-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.626 DEBUG:boto:path=/qskrps-46/obj7 2023-05-24T21:27:03.626 DEBUG:boto:auth_path=/qskrps-46/obj7 2023-05-24T21:27:03.627 DEBUG:boto:path=/qskrps-46/obj7?versionId=null 2023-05-24T21:27:03.627 DEBUG:boto:auth_path=/qskrps-46/obj7?versionId=null 2023-05-24T21:27:03.627 DEBUG:boto:Method: GET 2023-05-24T21:27:03.628 DEBUG:boto:Path: /qskrps-46/obj7?versionId=null 2023-05-24T21:27:03.628 DEBUG:boto:Data: 2023-05-24T21:27:03.628 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.629 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:03.629 DEBUG:boto:Port: 8000 2023-05-24T21:27:03.629 DEBUG:boto:Params: {} 2023-05-24T21:27:03.630 DEBUG:boto:Token: None 2023-05-24T21:27:03.630 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj7?versionId=null 2023-05-24T21:27:03.631 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:RimMsa+6vohKuCJWKmco3WlAHBA= 2023-05-24T21:27:03.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:RimMsa+6vohKuCJWKmco3WlAHBA=', 'Content-Length': '0'} 2023-05-24T21:27:03.633 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ced1377257dd033a-00646e8127-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.634 DEBUG:boto:path=/qskrps-46/obj7 2023-05-24T21:27:03.634 DEBUG:boto:auth_path=/qskrps-46/obj7 2023-05-24T21:27:03.634 DEBUG:boto:path=/qskrps-46/obj7?versionId=null 2023-05-24T21:27:03.635 DEBUG:boto:auth_path=/qskrps-46/obj7?versionId=null 2023-05-24T21:27:03.635 DEBUG:boto:Method: HEAD 2023-05-24T21:27:03.635 DEBUG:boto:Path: /qskrps-46/obj7?versionId=null 2023-05-24T21:27:03.636 DEBUG:boto:Data: 2023-05-24T21:27:03.636 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.636 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:03.637 DEBUG:boto:Port: 8001 2023-05-24T21:27:03.637 DEBUG:boto:Params: {} 2023-05-24T21:27:03.637 DEBUG:boto:Token: None 2023-05-24T21:27:03.638 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj7?versionId=null 2023-05-24T21:27:03.638 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:+fjApcvVGFcXDWBVkQRSzkjXOKc= 2023-05-24T21:27:03.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:+fjApcvVGFcXDWBVkQRSzkjXOKc=', 'Content-Length': '0'} 2023-05-24T21:27:03.640 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000022a3cf09d64c5c12-00646e8127-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.641 DEBUG:boto:path=/qskrps-46/obj7 2023-05-24T21:27:03.641 DEBUG:boto:auth_path=/qskrps-46/obj7 2023-05-24T21:27:03.642 DEBUG:boto:path=/qskrps-46/obj7?versionId=null 2023-05-24T21:27:03.642 DEBUG:boto:auth_path=/qskrps-46/obj7?versionId=null 2023-05-24T21:27:03.642 DEBUG:boto:Method: HEAD 2023-05-24T21:27:03.643 DEBUG:boto:Path: /qskrps-46/obj7?versionId=null 2023-05-24T21:27:03.643 DEBUG:boto:Data: 2023-05-24T21:27:03.643 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.644 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:03.644 DEBUG:boto:Port: 8000 2023-05-24T21:27:03.644 DEBUG:boto:Params: {} 2023-05-24T21:27:03.645 DEBUG:boto:Token: None 2023-05-24T21:27:03.645 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj7?versionId=null 2023-05-24T21:27:03.646 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:+fjApcvVGFcXDWBVkQRSzkjXOKc= 2023-05-24T21:27:03.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:+fjApcvVGFcXDWBVkQRSzkjXOKc=', 'Content-Length': '0'} 2023-05-24T21:27:03.648 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000617f458dcf632fa6-00646e8127-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-05-24T21:27:03.649 DEBUG:boto:path=/qskrps-46/obj7 2023-05-24T21:27:03.649 DEBUG:boto:auth_path=/qskrps-46/obj7 2023-05-24T21:27:03.650 DEBUG:boto:path=/qskrps-46/obj7?versionId=null 2023-05-24T21:27:03.650 DEBUG:boto:auth_path=/qskrps-46/obj7?versionId=null 2023-05-24T21:27:03.650 DEBUG:boto:Method: GET 2023-05-24T21:27:03.651 DEBUG:boto:Path: /qskrps-46/obj7?versionId=null 2023-05-24T21:27:03.651 DEBUG:boto:Data: 2023-05-24T21:27:03.651 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.652 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:03.652 DEBUG:boto:Port: 8001 2023-05-24T21:27:03.652 DEBUG:boto:Params: {} 2023-05-24T21:27:03.653 DEBUG:boto:Token: None 2023-05-24T21:27:03.653 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj7?versionId=null 2023-05-24T21:27:03.654 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:RimMsa+6vohKuCJWKmco3WlAHBA= 2023-05-24T21:27:03.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:RimMsa+6vohKuCJWKmco3WlAHBA=', 'Content-Length': '0'} 2023-05-24T21:27:03.656 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000010063c6ed2ba972c-00646e8127-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.696 DEBUG:boto:path=/qskrps-46/obj7 2023-05-24T21:27:03.697 DEBUG:boto:auth_path=/qskrps-46/obj7 2023-05-24T21:27:03.697 DEBUG:boto:path=/qskrps-46/obj7?versionId=null 2023-05-24T21:27:03.698 DEBUG:boto:auth_path=/qskrps-46/obj7?versionId=null 2023-05-24T21:27:03.698 DEBUG:boto:Method: GET 2023-05-24T21:27:03.698 DEBUG:boto:Path: /qskrps-46/obj7?versionId=null 2023-05-24T21:27:03.699 DEBUG:boto:Data: 2023-05-24T21:27:03.699 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.699 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:03.700 DEBUG:boto:Port: 8000 2023-05-24T21:27:03.700 DEBUG:boto:Params: {} 2023-05-24T21:27:03.700 DEBUG:boto:Token: None 2023-05-24T21:27:03.701 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj7?versionId=null 2023-05-24T21:27:03.701 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:RimMsa+6vohKuCJWKmco3WlAHBA= 2023-05-24T21:27:03.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:RimMsa+6vohKuCJWKmco3WlAHBA=', 'Content-Length': '0'} 2023-05-24T21:27:03.704 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000060548e821d5f62a1-00646e8127-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.704 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:27:03.705 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:27:03.705 DEBUG:boto:path=/qskrps-46/?prefix=obj7 2023-05-24T21:27:03.705 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj7 2023-05-24T21:27:03.706 DEBUG:boto:Method: GET 2023-05-24T21:27:03.706 DEBUG:boto:Path: /qskrps-46/?prefix=obj7 2023-05-24T21:27:03.707 DEBUG:boto:Data: 2023-05-24T21:27:03.707 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.707 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:03.708 DEBUG:boto:Port: 8001 2023-05-24T21:27:03.708 DEBUG:boto:Params: {} 2023-05-24T21:27:03.708 DEBUG:boto:Token: None 2023-05-24T21:27:03.709 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/ 2023-05-24T21:27:03.709 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I= 2023-05-24T21:27:03.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I=', 'Content-Length': '0'} 2023-05-24T21:27:03.712 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003aade316f1792922-00646e8127-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.712 DEBUG:boto:b'qskrps-46obj71000falseobj72023-05-24T21:26:22.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:03.713 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:27:03.714 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:27:03.714 DEBUG:boto:path=/qskrps-46/?prefix=obj7 2023-05-24T21:27:03.714 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj7 2023-05-24T21:27:03.715 DEBUG:boto:Method: GET 2023-05-24T21:27:03.715 DEBUG:boto:Path: /qskrps-46/?prefix=obj7 2023-05-24T21:27:03.715 DEBUG:boto:Data: 2023-05-24T21:27:03.716 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.716 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:03.716 DEBUG:boto:Port: 8000 2023-05-24T21:27:03.717 DEBUG:boto:Params: {} 2023-05-24T21:27:03.717 DEBUG:boto:Token: None 2023-05-24T21:27:03.718 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/ 2023-05-24T21:27:03.718 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I= 2023-05-24T21:27:03.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I=', 'Content-Length': '0'} 2023-05-24T21:27:03.721 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bfbe6b31ddfffbf-00646e8127-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.721 DEBUG:boto:b'qskrps-46obj71000falseobj72023-05-24T21:26:22.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:03.722 DEBUG:boto:path=/qskrps-46/obj7 2023-05-24T21:27:03.722 DEBUG:boto:auth_path=/qskrps-46/obj7 2023-05-24T21:27:03.722 DEBUG:boto:Method: HEAD 2023-05-24T21:27:03.723 DEBUG:boto:Path: /qskrps-46/obj7 2023-05-24T21:27:03.723 DEBUG:boto:Data: 2023-05-24T21:27:03.724 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.724 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:03.724 DEBUG:boto:Port: 8001 2023-05-24T21:27:03.725 DEBUG:boto:Params: {} 2023-05-24T21:27:03.725 DEBUG:boto:Token: None 2023-05-24T21:27:03.725 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj7 2023-05-24T21:27:03.726 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:6Mbx7tcaWXBewKQCu1JxpNYe1ag= 2023-05-24T21:27:03.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:6Mbx7tcaWXBewKQCu1JxpNYe1ag=', 'Content-Length': '0'} 2023-05-24T21:27:03.728 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000629f57f80c150317-00646e8127-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.728 DEBUG:boto:path=/qskrps-46/obj7 2023-05-24T21:27:03.729 DEBUG:boto:auth_path=/qskrps-46/obj7 2023-05-24T21:27:03.729 DEBUG:boto:Method: HEAD 2023-05-24T21:27:03.729 DEBUG:boto:Path: /qskrps-46/obj7 2023-05-24T21:27:03.731 DEBUG:boto:Data: 2023-05-24T21:27:03.732 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.732 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:03.732 DEBUG:boto:Port: 8000 2023-05-24T21:27:03.733 DEBUG:boto:Params: {} 2023-05-24T21:27:03.733 DEBUG:boto:Token: None 2023-05-24T21:27:03.733 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj7 2023-05-24T21:27:03.734 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:6Mbx7tcaWXBewKQCu1JxpNYe1ag= 2023-05-24T21:27:03.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:6Mbx7tcaWXBewKQCu1JxpNYe1ag=', 'Content-Length': '0'} 2023-05-24T21:27:03.736 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d5572749c44fa4b6-00646e8127-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-05-24T21:27:03.737 DEBUG:boto:path=/qskrps-46/obj7 2023-05-24T21:27:03.737 DEBUG:boto:auth_path=/qskrps-46/obj7 2023-05-24T21:27:03.737 DEBUG:boto:Method: GET 2023-05-24T21:27:03.738 DEBUG:boto:Path: /qskrps-46/obj7 2023-05-24T21:27:03.738 DEBUG:boto:Data: 2023-05-24T21:27:03.739 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.739 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:03.739 DEBUG:boto:Port: 8001 2023-05-24T21:27:03.740 DEBUG:boto:Params: {} 2023-05-24T21:27:03.740 DEBUG:boto:Token: None 2023-05-24T21:27:03.740 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj7 2023-05-24T21:27:03.741 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:prWistlkbz1u1ZTjzT8Zt8fhk1s= 2023-05-24T21:27:03.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:prWistlkbz1u1ZTjzT8Zt8fhk1s=', 'Content-Length': '0'} 2023-05-24T21:27:03.743 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c20b4dfbe02f94b9-00646e8127-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.743 DEBUG:boto:path=/qskrps-46/obj7 2023-05-24T21:27:03.744 DEBUG:boto:auth_path=/qskrps-46/obj7 2023-05-24T21:27:03.744 DEBUG:boto:Method: GET 2023-05-24T21:27:03.745 DEBUG:boto:Path: /qskrps-46/obj7 2023-05-24T21:27:03.745 DEBUG:boto:Data: 2023-05-24T21:27:03.745 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.746 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:03.746 DEBUG:boto:Port: 8000 2023-05-24T21:27:03.746 DEBUG:boto:Params: {} 2023-05-24T21:27:03.747 DEBUG:boto:Token: None 2023-05-24T21:27:03.747 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj7 2023-05-24T21:27:03.748 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:prWistlkbz1u1ZTjzT8Zt8fhk1s= 2023-05-24T21:27:03.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:prWistlkbz1u1ZTjzT8Zt8fhk1s=', 'Content-Length': '0'} 2023-05-24T21:27:03.750 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000997d1ddde475a527-00646e8127-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.792 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:27:03.793 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:27:03.793 DEBUG:boto:path=/qskrps-46/?prefix=obj7 2023-05-24T21:27:03.793 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj7 2023-05-24T21:27:03.794 DEBUG:boto:Method: GET 2023-05-24T21:27:03.794 DEBUG:boto:Path: /qskrps-46/?prefix=obj7 2023-05-24T21:27:03.794 DEBUG:boto:Data: 2023-05-24T21:27:03.795 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.795 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:03.795 DEBUG:boto:Port: 8001 2023-05-24T21:27:03.796 DEBUG:boto:Params: {} 2023-05-24T21:27:03.796 DEBUG:boto:Token: None 2023-05-24T21:27:03.797 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/ 2023-05-24T21:27:03.797 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I= 2023-05-24T21:27:03.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I=', 'Content-Length': '0'} 2023-05-24T21:27:03.800 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b40b5965da2db62-00646e8127-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.800 DEBUG:boto:b'qskrps-46obj71000falseobj72023-05-24T21:26:22.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:03.801 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:27:03.801 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:27:03.802 DEBUG:boto:path=/qskrps-46/?prefix=obj7 2023-05-24T21:27:03.802 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj7 2023-05-24T21:27:03.803 DEBUG:boto:Method: GET 2023-05-24T21:27:03.803 DEBUG:boto:Path: /qskrps-46/?prefix=obj7 2023-05-24T21:27:03.803 DEBUG:boto:Data: 2023-05-24T21:27:03.804 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.804 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:03.804 DEBUG:boto:Port: 8000 2023-05-24T21:27:03.805 DEBUG:boto:Params: {} 2023-05-24T21:27:03.805 DEBUG:boto:Token: None 2023-05-24T21:27:03.805 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/ 2023-05-24T21:27:03.806 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I= 2023-05-24T21:27:03.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I=', 'Content-Length': '0'} 2023-05-24T21:27:03.809 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003937ae4088441e21-00646e8127-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.809 DEBUG:boto:b'qskrps-46obj71000falseobj72023-05-24T21:26:22.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:03.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-05-24T21:27:03.810 DEBUG:boto:path=/qskrps-46/obj8 2023-05-24T21:27:03.811 DEBUG:boto:auth_path=/qskrps-46/obj8 2023-05-24T21:27:03.811 DEBUG:boto:path=/qskrps-46/obj8?versionId=null 2023-05-24T21:27:03.811 DEBUG:boto:auth_path=/qskrps-46/obj8?versionId=null 2023-05-24T21:27:03.812 DEBUG:boto:Method: GET 2023-05-24T21:27:03.812 DEBUG:boto:Path: /qskrps-46/obj8?versionId=null 2023-05-24T21:27:03.812 DEBUG:boto:Data: 2023-05-24T21:27:03.813 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.813 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:03.813 DEBUG:boto:Port: 8001 2023-05-24T21:27:03.814 DEBUG:boto:Params: {} 2023-05-24T21:27:03.814 DEBUG:boto:Token: None 2023-05-24T21:27:03.815 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj8?versionId=null 2023-05-24T21:27:03.815 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:vay6iU1myehR4nFG+2End3txAUQ= 2023-05-24T21:27:03.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:vay6iU1myehR4nFG+2End3txAUQ=', 'Content-Length': '0'} 2023-05-24T21:27:03.817 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f847728cd4491490-00646e8127-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.818 DEBUG:boto:path=/qskrps-46/obj8 2023-05-24T21:27:03.818 DEBUG:boto:auth_path=/qskrps-46/obj8 2023-05-24T21:27:03.819 DEBUG:boto:path=/qskrps-46/obj8?versionId=null 2023-05-24T21:27:03.819 DEBUG:boto:auth_path=/qskrps-46/obj8?versionId=null 2023-05-24T21:27:03.819 DEBUG:boto:Method: GET 2023-05-24T21:27:03.820 DEBUG:boto:Path: /qskrps-46/obj8?versionId=null 2023-05-24T21:27:03.820 DEBUG:boto:Data: 2023-05-24T21:27:03.821 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.821 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:03.821 DEBUG:boto:Port: 8000 2023-05-24T21:27:03.822 DEBUG:boto:Params: {} 2023-05-24T21:27:03.822 DEBUG:boto:Token: None 2023-05-24T21:27:03.822 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj8?versionId=null 2023-05-24T21:27:03.823 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:vay6iU1myehR4nFG+2End3txAUQ= 2023-05-24T21:27:03.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:vay6iU1myehR4nFG+2End3txAUQ=', 'Content-Length': '0'} 2023-05-24T21:27:03.825 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009a65d7b4bed4633c-00646e8127-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.826 DEBUG:boto:path=/qskrps-46/obj8 2023-05-24T21:27:03.826 DEBUG:boto:auth_path=/qskrps-46/obj8 2023-05-24T21:27:03.826 DEBUG:boto:path=/qskrps-46/obj8?versionId=null 2023-05-24T21:27:03.827 DEBUG:boto:auth_path=/qskrps-46/obj8?versionId=null 2023-05-24T21:27:03.827 DEBUG:boto:Method: HEAD 2023-05-24T21:27:03.827 DEBUG:boto:Path: /qskrps-46/obj8?versionId=null 2023-05-24T21:27:03.828 DEBUG:boto:Data: 2023-05-24T21:27:03.828 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.828 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:03.829 DEBUG:boto:Port: 8001 2023-05-24T21:27:03.829 DEBUG:boto:Params: {} 2023-05-24T21:27:03.829 DEBUG:boto:Token: None 2023-05-24T21:27:03.830 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj8?versionId=null 2023-05-24T21:27:03.830 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ke3iJ2h5nZtdx4G+xHqzsmX1EOg= 2023-05-24T21:27:03.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ke3iJ2h5nZtdx4G+xHqzsmX1EOg=', 'Content-Length': '0'} 2023-05-24T21:27:03.832 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dcafa3c1ab950eef-00646e8127-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.833 DEBUG:boto:path=/qskrps-46/obj8 2023-05-24T21:27:03.833 DEBUG:boto:auth_path=/qskrps-46/obj8 2023-05-24T21:27:03.833 DEBUG:boto:path=/qskrps-46/obj8?versionId=null 2023-05-24T21:27:03.834 DEBUG:boto:auth_path=/qskrps-46/obj8?versionId=null 2023-05-24T21:27:03.834 DEBUG:boto:Method: HEAD 2023-05-24T21:27:03.834 DEBUG:boto:Path: /qskrps-46/obj8?versionId=null 2023-05-24T21:27:03.835 DEBUG:boto:Data: 2023-05-24T21:27:03.835 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.835 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:03.836 DEBUG:boto:Port: 8000 2023-05-24T21:27:03.836 DEBUG:boto:Params: {} 2023-05-24T21:27:03.836 DEBUG:boto:Token: None 2023-05-24T21:27:03.837 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj8?versionId=null 2023-05-24T21:27:03.837 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ke3iJ2h5nZtdx4G+xHqzsmX1EOg= 2023-05-24T21:27:03.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ke3iJ2h5nZtdx4G+xHqzsmX1EOg=', 'Content-Length': '0'} 2023-05-24T21:27:03.839 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a4b5a3080ef0ee73-00646e8127-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-05-24T21:27:03.840 DEBUG:boto:path=/qskrps-46/obj8 2023-05-24T21:27:03.840 DEBUG:boto:auth_path=/qskrps-46/obj8 2023-05-24T21:27:03.841 DEBUG:boto:path=/qskrps-46/obj8?versionId=null 2023-05-24T21:27:03.841 DEBUG:boto:auth_path=/qskrps-46/obj8?versionId=null 2023-05-24T21:27:03.841 DEBUG:boto:Method: GET 2023-05-24T21:27:03.842 DEBUG:boto:Path: /qskrps-46/obj8?versionId=null 2023-05-24T21:27:03.842 DEBUG:boto:Data: 2023-05-24T21:27:03.843 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.843 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:03.843 DEBUG:boto:Port: 8001 2023-05-24T21:27:03.844 DEBUG:boto:Params: {} 2023-05-24T21:27:03.844 DEBUG:boto:Token: None 2023-05-24T21:27:03.844 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj8?versionId=null 2023-05-24T21:27:03.845 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:vay6iU1myehR4nFG+2End3txAUQ= 2023-05-24T21:27:03.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:vay6iU1myehR4nFG+2End3txAUQ=', 'Content-Length': '0'} 2023-05-24T21:27:03.847 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005da2204a07649a07-00646e8127-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.848 DEBUG:boto:path=/qskrps-46/obj8 2023-05-24T21:27:03.848 DEBUG:boto:auth_path=/qskrps-46/obj8 2023-05-24T21:27:03.848 DEBUG:boto:path=/qskrps-46/obj8?versionId=null 2023-05-24T21:27:03.849 DEBUG:boto:auth_path=/qskrps-46/obj8?versionId=null 2023-05-24T21:27:03.849 DEBUG:boto:Method: GET 2023-05-24T21:27:03.849 DEBUG:boto:Path: /qskrps-46/obj8?versionId=null 2023-05-24T21:27:03.850 DEBUG:boto:Data: 2023-05-24T21:27:03.850 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.851 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:03.851 DEBUG:boto:Port: 8000 2023-05-24T21:27:03.851 DEBUG:boto:Params: {} 2023-05-24T21:27:03.852 DEBUG:boto:Token: None 2023-05-24T21:27:03.852 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj8?versionId=null 2023-05-24T21:27:03.852 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:vay6iU1myehR4nFG+2End3txAUQ= 2023-05-24T21:27:03.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:vay6iU1myehR4nFG+2End3txAUQ=', 'Content-Length': '0'} 2023-05-24T21:27:03.855 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b5073b7e878bb4f1-00646e8127-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.896 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:27:03.897 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:27:03.897 DEBUG:boto:path=/qskrps-46/?prefix=obj8 2023-05-24T21:27:03.897 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj8 2023-05-24T21:27:03.898 DEBUG:boto:Method: GET 2023-05-24T21:27:03.898 DEBUG:boto:Path: /qskrps-46/?prefix=obj8 2023-05-24T21:27:03.899 DEBUG:boto:Data: 2023-05-24T21:27:03.899 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.899 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:03.900 DEBUG:boto:Port: 8001 2023-05-24T21:27:03.900 DEBUG:boto:Params: {} 2023-05-24T21:27:03.900 DEBUG:boto:Token: None 2023-05-24T21:27:03.901 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/ 2023-05-24T21:27:03.901 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I= 2023-05-24T21:27:03.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I=', 'Content-Length': '0'} 2023-05-24T21:27:03.904 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000873bf47ec573be52-00646e8127-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.904 DEBUG:boto:b'qskrps-46obj81000falseobj82023-05-24T21:26:22.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:03.905 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:27:03.905 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:27:03.906 DEBUG:boto:path=/qskrps-46/?prefix=obj8 2023-05-24T21:27:03.906 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj8 2023-05-24T21:27:03.906 DEBUG:boto:Method: GET 2023-05-24T21:27:03.907 DEBUG:boto:Path: /qskrps-46/?prefix=obj8 2023-05-24T21:27:03.907 DEBUG:boto:Data: 2023-05-24T21:27:03.907 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.908 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:03.908 DEBUG:boto:Port: 8000 2023-05-24T21:27:03.909 DEBUG:boto:Params: {} 2023-05-24T21:27:03.909 DEBUG:boto:Token: None 2023-05-24T21:27:03.909 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/ 2023-05-24T21:27:03.910 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I= 2023-05-24T21:27:03.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I=', 'Content-Length': '0'} 2023-05-24T21:27:03.912 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046ff041c84d31528-00646e8127-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.913 DEBUG:boto:b'qskrps-46obj81000falseobj82023-05-24T21:26:22.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:03.914 DEBUG:boto:path=/qskrps-46/obj8 2023-05-24T21:27:03.914 DEBUG:boto:auth_path=/qskrps-46/obj8 2023-05-24T21:27:03.914 DEBUG:boto:Method: HEAD 2023-05-24T21:27:03.915 DEBUG:boto:Path: /qskrps-46/obj8 2023-05-24T21:27:03.915 DEBUG:boto:Data: 2023-05-24T21:27:03.915 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.916 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:03.916 DEBUG:boto:Port: 8001 2023-05-24T21:27:03.916 DEBUG:boto:Params: {} 2023-05-24T21:27:03.917 DEBUG:boto:Token: None 2023-05-24T21:27:03.917 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj8 2023-05-24T21:27:03.918 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:gIXDNvuQz3v9M4wCDunMWNTN0lA= 2023-05-24T21:27:03.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:gIXDNvuQz3v9M4wCDunMWNTN0lA=', 'Content-Length': '0'} 2023-05-24T21:27:03.920 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000076b5c93cd68824a4-00646e8127-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.920 DEBUG:boto:path=/qskrps-46/obj8 2023-05-24T21:27:03.921 DEBUG:boto:auth_path=/qskrps-46/obj8 2023-05-24T21:27:03.921 DEBUG:boto:Method: HEAD 2023-05-24T21:27:03.921 DEBUG:boto:Path: /qskrps-46/obj8 2023-05-24T21:27:03.922 DEBUG:boto:Data: 2023-05-24T21:27:03.922 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.922 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:03.923 DEBUG:boto:Port: 8000 2023-05-24T21:27:03.923 DEBUG:boto:Params: {} 2023-05-24T21:27:03.923 DEBUG:boto:Token: None 2023-05-24T21:27:03.924 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj8 2023-05-24T21:27:03.924 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:gIXDNvuQz3v9M4wCDunMWNTN0lA= 2023-05-24T21:27:03.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:gIXDNvuQz3v9M4wCDunMWNTN0lA=', 'Content-Length': '0'} 2023-05-24T21:27:03.926 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000714118a88c71169e-00646e8127-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-05-24T21:27:03.927 DEBUG:boto:path=/qskrps-46/obj8 2023-05-24T21:27:03.928 DEBUG:boto:auth_path=/qskrps-46/obj8 2023-05-24T21:27:03.928 DEBUG:boto:Method: GET 2023-05-24T21:27:03.928 DEBUG:boto:Path: /qskrps-46/obj8 2023-05-24T21:27:03.929 DEBUG:boto:Data: 2023-05-24T21:27:03.929 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.930 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:03.930 DEBUG:boto:Port: 8001 2023-05-24T21:27:03.930 DEBUG:boto:Params: {} 2023-05-24T21:27:03.931 DEBUG:boto:Token: None 2023-05-24T21:27:03.931 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj8 2023-05-24T21:27:03.931 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:VpUMecbD+cd0dKqKvB5BiE7O8sQ= 2023-05-24T21:27:03.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:VpUMecbD+cd0dKqKvB5BiE7O8sQ=', 'Content-Length': '0'} 2023-05-24T21:27:03.934 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004ad456ecc9dd7936-00646e8127-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.934 DEBUG:boto:path=/qskrps-46/obj8 2023-05-24T21:27:03.935 DEBUG:boto:auth_path=/qskrps-46/obj8 2023-05-24T21:27:03.935 DEBUG:boto:Method: GET 2023-05-24T21:27:03.935 DEBUG:boto:Path: /qskrps-46/obj8 2023-05-24T21:27:03.936 DEBUG:boto:Data: 2023-05-24T21:27:03.936 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.937 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:03.937 DEBUG:boto:Port: 8000 2023-05-24T21:27:03.937 DEBUG:boto:Params: {} 2023-05-24T21:27:03.938 DEBUG:boto:Token: None 2023-05-24T21:27:03.938 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/obj8 2023-05-24T21:27:03.938 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:VpUMecbD+cd0dKqKvB5BiE7O8sQ= 2023-05-24T21:27:03.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:VpUMecbD+cd0dKqKvB5BiE7O8sQ=', 'Content-Length': '0'} 2023-05-24T21:27:03.941 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006b9bf7406a9d30c4-00646e8127-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.941 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:27:03.942 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:27:03.942 DEBUG:boto:path=/qskrps-46/?prefix=obj8 2023-05-24T21:27:03.942 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj8 2023-05-24T21:27:03.943 DEBUG:boto:Method: GET 2023-05-24T21:27:03.943 DEBUG:boto:Path: /qskrps-46/?prefix=obj8 2023-05-24T21:27:03.944 DEBUG:boto:Data: 2023-05-24T21:27:03.944 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.944 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:03.945 DEBUG:boto:Port: 8001 2023-05-24T21:27:03.945 DEBUG:boto:Params: {} 2023-05-24T21:27:03.945 DEBUG:boto:Token: None 2023-05-24T21:27:03.946 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/ 2023-05-24T21:27:03.946 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I= 2023-05-24T21:27:03.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I=', 'Content-Length': '0'} 2023-05-24T21:27:03.949 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb17fe5c5c1ba956-00646e8127-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:03.949 DEBUG:boto:b'qskrps-46obj81000falseobj82023-05-24T21:26:22.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:03.950 DEBUG:boto:path=/qskrps-46/ 2023-05-24T21:27:03.950 DEBUG:boto:auth_path=/qskrps-46/ 2023-05-24T21:27:03.951 DEBUG:boto:path=/qskrps-46/?prefix=obj8 2023-05-24T21:27:03.951 DEBUG:boto:auth_path=/qskrps-46/?prefix=obj8 2023-05-24T21:27:03.951 DEBUG:boto:Method: GET 2023-05-24T21:27:03.952 DEBUG:boto:Path: /qskrps-46/?prefix=obj8 2023-05-24T21:27:03.952 DEBUG:boto:Data: 2023-05-24T21:27:03.952 DEBUG:boto:Headers: {} 2023-05-24T21:27:03.953 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:03.953 DEBUG:boto:Port: 8000 2023-05-24T21:27:03.953 DEBUG:boto:Params: {} 2023-05-24T21:27:03.954 DEBUG:boto:Token: None 2023-05-24T21:27:03.954 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:03 GMT /qskrps-46/ 2023-05-24T21:27:03.955 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I= 2023-05-24T21:27:03.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4v6apnr6ICzF3mm0UnwoWbIfV6I=', 'Content-Length': '0'} 2023-05-24T21:27:03.957 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025b9ea810abe7648-00646e8127-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.000 DEBUG:boto:b'qskrps-46obj81000falseobj82023-05-24T21:26:22.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:04.001 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qskrps-46 zones={a2, a1} 2023-05-24T21:27:04.001 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=qskrps-47 2023-05-24T21:27:04.002 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-47', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:27:04.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-47', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:27:04.002 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-47 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:27:04.063 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:27:04.064 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:27:04.111 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:27:04.105+0000 7f4303cf2780 1 bucket sync caught up with source: 2023-05-24T21:27:04.111 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [00000000004.15.6, 00000000002.49.1, 00000000004.24.6, 00000000002.19.1, 00000000004.52.6, 00000000002.106.1, 00000000002.18.1, 00000000004.32.6, 00000000002.107.1, 00000000002.23.1, 00000000002.30.1] 2023-05-24T21:27:04.111 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [00000000004.15.6, 00000000002.49.1, 00000000004.24.6, 00000000002.19.1, 00000000004.52.6, 00000000002.106.1, 00000000002.18.1, 00000000004.32.6, 00000000002.107.1, 00000000002.23.1, 00000000002.30.1] 2023-05-24T21:27:04.111 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:27:04.105+0000 7f4303cf2780 0 bucket checkpoint complete 2023-05-24T21:27:04.125 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=qskrps-47 2023-05-24T21:27:04.126 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-47', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:27:04.126 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-47', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:27:04.127 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-47 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:27:04.189 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:27:04.189 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:27:04.226 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:27:04.218+0000 7f5eca6d0780 1 bucket sync caught up with source: 2023-05-24T21:27:04.226 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [00000000002.329.6, , 00000000002.510.6, , 00000000002.539.6, , , 00000000002.852.6, , , ] 2023-05-24T21:27:04.227 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [00000000002.329.6, , 00000000002.510.6, , 00000000002.539.6, , , 00000000002.852.6, , , ] 2023-05-24T21:27:04.227 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:27:04.218+0000 7f5eca6d0780 0 bucket checkpoint complete 2023-05-24T21:27:04.236 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qskrps-47 zones={a2, a1} 2023-05-24T21:27:04.237 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:27:04.237 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:27:04.237 DEBUG:boto:Method: HEAD 2023-05-24T21:27:04.238 DEBUG:boto:Path: /qskrps-47/ 2023-05-24T21:27:04.238 DEBUG:boto:Data: 2023-05-24T21:27:04.238 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.239 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:04.239 DEBUG:boto:Port: 8001 2023-05-24T21:27:04.239 DEBUG:boto:Params: {} 2023-05-24T21:27:04.240 DEBUG:boto:Token: None 2023-05-24T21:27:04.240 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:04 GMT /qskrps-47/ 2023-05-24T21:27:04.241 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:7n9kjUxPVCEt46pa/MlAo9GVH6k= 2023-05-24T21:27:04.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:7n9kjUxPVCEt46pa/MlAo9GVH6k=', 'Content-Length': '0'} 2023-05-24T21:27:04.244 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000de3e0d39feb8788-00646e8128-1625-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.245 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:27:04.245 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:27:04.246 DEBUG:boto:Method: HEAD 2023-05-24T21:27:04.246 DEBUG:boto:Path: /qskrps-47/ 2023-05-24T21:27:04.246 DEBUG:boto:Data: 2023-05-24T21:27:04.247 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.247 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:04.247 DEBUG:boto:Port: 8000 2023-05-24T21:27:04.248 DEBUG:boto:Params: {} 2023-05-24T21:27:04.248 DEBUG:boto:Token: None 2023-05-24T21:27:04.248 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:04 GMT /qskrps-47/ 2023-05-24T21:27:04.249 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:7n9kjUxPVCEt46pa/MlAo9GVH6k= 2023-05-24T21:27:04.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:7n9kjUxPVCEt46pa/MlAo9GVH6k=', 'Content-Length': '0'} 2023-05-24T21:27:04.255 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f06e5187739fb06c-00646e8128-1627-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.255 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-24T21:27:04.256 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:27:04.256 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:27:04.256 DEBUG:boto:path=/qskrps-47/?versions&max-keys=999 2023-05-24T21:27:04.257 DEBUG:boto:auth_path=/qskrps-47/?versions&max-keys=999 2023-05-24T21:27:04.257 DEBUG:boto:Method: GET 2023-05-24T21:27:04.257 DEBUG:boto:Path: /qskrps-47/?versions&max-keys=999 2023-05-24T21:27:04.258 DEBUG:boto:Data: 2023-05-24T21:27:04.258 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.258 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:04.259 DEBUG:boto:Port: 8001 2023-05-24T21:27:04.259 DEBUG:boto:Params: {} 2023-05-24T21:27:04.259 DEBUG:boto:Token: None 2023-05-24T21:27:04.260 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:04 GMT /qskrps-47/?versions 2023-05-24T21:27:04.260 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:8016dZlj76wFp5gICiJZLQ+7+Kk= 2023-05-24T21:27:04.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:8016dZlj76wFp5gICiJZLQ+7+Kk=', 'Content-Length': '0'} 2023-05-24T21:27:04.268 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000036d5b77f36b1dda-00646e8128-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.269 DEBUG:boto:b'qskrps-47999falseobj1nulltrue2023-05-24T21:25:49.049Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-24T21:25:49.067Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-24T21:25:49.084Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-24T21:25:49.101Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-05-24T21:26:22.170Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-05-24T21:26:22.187Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-05-24T21:26:22.240Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-05-24T21:26:22.259Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:04.270 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-05-24T21:27:04.270 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-05-24T21:27:04.271 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-05-24T21:27:04.271 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-05-24T21:27:04.271 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-05-24T21:27:04.271 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-05-24T21:27:04.272 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-05-24T21:27:04.272 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-05-24T21:27:04.272 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-24T21:27:04.273 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:27:04.273 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:27:04.273 DEBUG:boto:path=/qskrps-47/?versions&max-keys=999 2023-05-24T21:27:04.274 DEBUG:boto:auth_path=/qskrps-47/?versions&max-keys=999 2023-05-24T21:27:04.274 DEBUG:boto:Method: GET 2023-05-24T21:27:04.274 DEBUG:boto:Path: /qskrps-47/?versions&max-keys=999 2023-05-24T21:27:04.275 DEBUG:boto:Data: 2023-05-24T21:27:04.275 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.275 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:04.276 DEBUG:boto:Port: 8000 2023-05-24T21:27:04.276 DEBUG:boto:Params: {} 2023-05-24T21:27:04.277 DEBUG:boto:Token: None 2023-05-24T21:27:04.277 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:04 GMT /qskrps-47/?versions 2023-05-24T21:27:04.277 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:8016dZlj76wFp5gICiJZLQ+7+Kk= 2023-05-24T21:27:04.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:8016dZlj76wFp5gICiJZLQ+7+Kk=', 'Content-Length': '0'} 2023-05-24T21:27:04.282 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d00c9ea298d6aa0-00646e8128-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.284 DEBUG:boto:b'qskrps-47999falseobj1nulltrue2023-05-24T21:25:49.049Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-24T21:25:49.067Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-24T21:25:49.084Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-24T21:25:49.101Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-05-24T21:26:22.170Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-05-24T21:26:22.187Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-05-24T21:26:22.240Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-05-24T21:26:22.259Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:04.285 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-05-24T21:27:04.285 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-05-24T21:27:04.285 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-05-24T21:27:04.286 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-05-24T21:27:04.286 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-05-24T21:27:04.286 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-05-24T21:27:04.287 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-05-24T21:27:04.287 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-05-24T21:27:04.287 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:27:04.288 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:27:04.288 DEBUG:boto:path=/qskrps-47/?versions&max-keys=999 2023-05-24T21:27:04.288 DEBUG:boto:auth_path=/qskrps-47/?versions&max-keys=999 2023-05-24T21:27:04.289 DEBUG:boto:Method: GET 2023-05-24T21:27:04.289 DEBUG:boto:Path: /qskrps-47/?versions&max-keys=999 2023-05-24T21:27:04.289 DEBUG:boto:Data: 2023-05-24T21:27:04.290 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.290 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:04.290 DEBUG:boto:Port: 8001 2023-05-24T21:27:04.291 DEBUG:boto:Params: {} 2023-05-24T21:27:04.291 DEBUG:boto:Token: None 2023-05-24T21:27:04.291 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:04 GMT /qskrps-47/?versions 2023-05-24T21:27:04.292 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:8016dZlj76wFp5gICiJZLQ+7+Kk= 2023-05-24T21:27:04.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:8016dZlj76wFp5gICiJZLQ+7+Kk=', 'Content-Length': '0'} 2023-05-24T21:27:04.300 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af4565e82d37ee08-00646e8128-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.301 DEBUG:boto:b'qskrps-47999falseobj1nulltrue2023-05-24T21:25:49.049Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-24T21:25:49.067Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-24T21:25:49.084Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-24T21:25:49.101Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-05-24T21:26:22.170Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-05-24T21:26:22.187Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-05-24T21:26:22.240Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-05-24T21:26:22.259Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:04.302 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:27:04.302 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:27:04.302 DEBUG:boto:path=/qskrps-47/?versions&max-keys=999 2023-05-24T21:27:04.303 DEBUG:boto:auth_path=/qskrps-47/?versions&max-keys=999 2023-05-24T21:27:04.303 DEBUG:boto:Method: GET 2023-05-24T21:27:04.303 DEBUG:boto:Path: /qskrps-47/?versions&max-keys=999 2023-05-24T21:27:04.304 DEBUG:boto:Data: 2023-05-24T21:27:04.304 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.304 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:04.305 DEBUG:boto:Port: 8000 2023-05-24T21:27:04.305 DEBUG:boto:Params: {} 2023-05-24T21:27:04.306 DEBUG:boto:Token: None 2023-05-24T21:27:04.306 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:04 GMT /qskrps-47/?versions 2023-05-24T21:27:04.306 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:8016dZlj76wFp5gICiJZLQ+7+Kk= 2023-05-24T21:27:04.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:8016dZlj76wFp5gICiJZLQ+7+Kk=', 'Content-Length': '0'} 2023-05-24T21:27:04.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054981f72cc124d9c-00646e8128-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.313 DEBUG:boto:b'qskrps-47999falseobj1nulltrue2023-05-24T21:25:49.049Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-24T21:25:49.067Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-24T21:25:49.084Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-24T21:25:49.101Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-05-24T21:26:22.170Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-05-24T21:26:22.187Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-05-24T21:26:22.240Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-05-24T21:26:22.259Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:04.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-24T21:27:04.315 DEBUG:boto:path=/qskrps-47/obj1 2023-05-24T21:27:04.315 DEBUG:boto:auth_path=/qskrps-47/obj1 2023-05-24T21:27:04.315 DEBUG:boto:path=/qskrps-47/obj1?versionId=null 2023-05-24T21:27:04.316 DEBUG:boto:auth_path=/qskrps-47/obj1?versionId=null 2023-05-24T21:27:04.316 DEBUG:boto:Method: GET 2023-05-24T21:27:04.317 DEBUG:boto:Path: /qskrps-47/obj1?versionId=null 2023-05-24T21:27:04.317 DEBUG:boto:Data: 2023-05-24T21:27:04.317 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.318 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:04.318 DEBUG:boto:Port: 8001 2023-05-24T21:27:04.318 DEBUG:boto:Params: {} 2023-05-24T21:27:04.319 DEBUG:boto:Token: None 2023-05-24T21:27:04.319 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:04 GMT /qskrps-47/obj1?versionId=null 2023-05-24T21:27:04.320 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:C67mknoHzahi3tUNc1fTkDZPDj4= 2023-05-24T21:27:04.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:C67mknoHzahi3tUNc1fTkDZPDj4=', 'Content-Length': '0'} 2023-05-24T21:27:04.322 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000986c9cc1a39efdf8-00646e8128-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.364 DEBUG:boto:path=/qskrps-47/obj1 2023-05-24T21:27:04.365 DEBUG:boto:auth_path=/qskrps-47/obj1 2023-05-24T21:27:04.365 DEBUG:boto:path=/qskrps-47/obj1?versionId=null 2023-05-24T21:27:04.366 DEBUG:boto:auth_path=/qskrps-47/obj1?versionId=null 2023-05-24T21:27:04.366 DEBUG:boto:Method: GET 2023-05-24T21:27:04.366 DEBUG:boto:Path: /qskrps-47/obj1?versionId=null 2023-05-24T21:27:04.367 DEBUG:boto:Data: 2023-05-24T21:27:04.367 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.368 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:04.368 DEBUG:boto:Port: 8000 2023-05-24T21:27:04.368 DEBUG:boto:Params: {} 2023-05-24T21:27:04.369 DEBUG:boto:Token: None 2023-05-24T21:27:04.369 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:04 GMT /qskrps-47/obj1?versionId=null 2023-05-24T21:27:04.370 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:C67mknoHzahi3tUNc1fTkDZPDj4= 2023-05-24T21:27:04.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:C67mknoHzahi3tUNc1fTkDZPDj4=', 'Content-Length': '0'} 2023-05-24T21:27:04.372 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000089697d27b38bbc68-00646e8128-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.373 DEBUG:boto:path=/qskrps-47/obj1 2023-05-24T21:27:04.374 DEBUG:boto:auth_path=/qskrps-47/obj1 2023-05-24T21:27:04.374 DEBUG:boto:path=/qskrps-47/obj1?versionId=null 2023-05-24T21:27:04.374 DEBUG:boto:auth_path=/qskrps-47/obj1?versionId=null 2023-05-24T21:27:04.375 DEBUG:boto:Method: HEAD 2023-05-24T21:27:04.375 DEBUG:boto:Path: /qskrps-47/obj1?versionId=null 2023-05-24T21:27:04.375 DEBUG:boto:Data: 2023-05-24T21:27:04.376 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.376 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:04.376 DEBUG:boto:Port: 8001 2023-05-24T21:27:04.377 DEBUG:boto:Params: {} 2023-05-24T21:27:04.377 DEBUG:boto:Token: None 2023-05-24T21:27:04.377 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:04 GMT /qskrps-47/obj1?versionId=null 2023-05-24T21:27:04.378 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:T/Gk7dnEtWS9Qs7cIw0ZkMTK6lg= 2023-05-24T21:27:04.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:T/Gk7dnEtWS9Qs7cIw0ZkMTK6lg=', 'Content-Length': '0'} 2023-05-24T21:27:04.381 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008f43a179052ff678-00646e8128-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.381 DEBUG:boto:path=/qskrps-47/obj1 2023-05-24T21:27:04.382 DEBUG:boto:auth_path=/qskrps-47/obj1 2023-05-24T21:27:04.382 DEBUG:boto:path=/qskrps-47/obj1?versionId=null 2023-05-24T21:27:04.382 DEBUG:boto:auth_path=/qskrps-47/obj1?versionId=null 2023-05-24T21:27:04.383 DEBUG:boto:Method: HEAD 2023-05-24T21:27:04.383 DEBUG:boto:Path: /qskrps-47/obj1?versionId=null 2023-05-24T21:27:04.383 DEBUG:boto:Data: 2023-05-24T21:27:04.384 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.384 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:04.384 DEBUG:boto:Port: 8000 2023-05-24T21:27:04.385 DEBUG:boto:Params: {} 2023-05-24T21:27:04.385 DEBUG:boto:Token: None 2023-05-24T21:27:04.385 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:04 GMT /qskrps-47/obj1?versionId=null 2023-05-24T21:27:04.386 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:T/Gk7dnEtWS9Qs7cIw0ZkMTK6lg= 2023-05-24T21:27:04.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:T/Gk7dnEtWS9Qs7cIw0ZkMTK6lg=', 'Content-Length': '0'} 2023-05-24T21:27:04.388 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005ee596036a228f72-00646e8128-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-24T21:27:04.389 DEBUG:boto:path=/qskrps-47/obj1 2023-05-24T21:27:04.389 DEBUG:boto:auth_path=/qskrps-47/obj1 2023-05-24T21:27:04.389 DEBUG:boto:path=/qskrps-47/obj1?versionId=null 2023-05-24T21:27:04.390 DEBUG:boto:auth_path=/qskrps-47/obj1?versionId=null 2023-05-24T21:27:04.390 DEBUG:boto:Method: GET 2023-05-24T21:27:04.391 DEBUG:boto:Path: /qskrps-47/obj1?versionId=null 2023-05-24T21:27:04.391 DEBUG:boto:Data: 2023-05-24T21:27:04.391 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.392 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:04.392 DEBUG:boto:Port: 8001 2023-05-24T21:27:04.392 DEBUG:boto:Params: {} 2023-05-24T21:27:04.393 DEBUG:boto:Token: None 2023-05-24T21:27:04.393 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:04 GMT /qskrps-47/obj1?versionId=null 2023-05-24T21:27:04.394 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:C67mknoHzahi3tUNc1fTkDZPDj4= 2023-05-24T21:27:04.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:C67mknoHzahi3tUNc1fTkDZPDj4=', 'Content-Length': '0'} 2023-05-24T21:27:04.397 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000135e127b3ca6dfb7-00646e8128-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.398 DEBUG:boto:path=/qskrps-47/obj1 2023-05-24T21:27:04.399 DEBUG:boto:auth_path=/qskrps-47/obj1 2023-05-24T21:27:04.399 DEBUG:boto:path=/qskrps-47/obj1?versionId=null 2023-05-24T21:27:04.399 DEBUG:boto:auth_path=/qskrps-47/obj1?versionId=null 2023-05-24T21:27:04.400 DEBUG:boto:Method: GET 2023-05-24T21:27:04.400 DEBUG:boto:Path: /qskrps-47/obj1?versionId=null 2023-05-24T21:27:04.401 DEBUG:boto:Data: 2023-05-24T21:27:04.401 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.401 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:04.402 DEBUG:boto:Port: 8000 2023-05-24T21:27:04.402 DEBUG:boto:Params: {} 2023-05-24T21:27:04.402 DEBUG:boto:Token: None 2023-05-24T21:27:04.403 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:04 GMT /qskrps-47/obj1?versionId=null 2023-05-24T21:27:04.403 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:C67mknoHzahi3tUNc1fTkDZPDj4= 2023-05-24T21:27:04.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:C67mknoHzahi3tUNc1fTkDZPDj4=', 'Content-Length': '0'} 2023-05-24T21:27:04.407 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bc2dc98bd361b4ae-00646e8128-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.448 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:27:04.449 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:27:04.449 DEBUG:boto:path=/qskrps-47/?prefix=obj1 2023-05-24T21:27:04.450 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj1 2023-05-24T21:27:04.450 DEBUG:boto:Method: GET 2023-05-24T21:27:04.450 DEBUG:boto:Path: /qskrps-47/?prefix=obj1 2023-05-24T21:27:04.451 DEBUG:boto:Data: 2023-05-24T21:27:04.451 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.451 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:04.452 DEBUG:boto:Port: 8001 2023-05-24T21:27:04.452 DEBUG:boto:Params: {} 2023-05-24T21:27:04.453 DEBUG:boto:Token: None 2023-05-24T21:27:04.453 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:04 GMT /qskrps-47/ 2023-05-24T21:27:04.453 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:JZjeXLi2iW/RoyoLgKjPhYTMNvw= 2023-05-24T21:27:04.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:JZjeXLi2iW/RoyoLgKjPhYTMNvw=', 'Content-Length': '0'} 2023-05-24T21:27:04.456 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9650f128df04253-00646e8128-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.457 DEBUG:boto:b'qskrps-47obj11000falseobj12023-05-24T21:25:49.049Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:04.458 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:27:04.458 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:27:04.459 DEBUG:boto:path=/qskrps-47/?prefix=obj1 2023-05-24T21:27:04.459 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj1 2023-05-24T21:27:04.459 DEBUG:boto:Method: GET 2023-05-24T21:27:04.460 DEBUG:boto:Path: /qskrps-47/?prefix=obj1 2023-05-24T21:27:04.460 DEBUG:boto:Data: 2023-05-24T21:27:04.460 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.461 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:04.461 DEBUG:boto:Port: 8000 2023-05-24T21:27:04.461 DEBUG:boto:Params: {} 2023-05-24T21:27:04.462 DEBUG:boto:Token: None 2023-05-24T21:27:04.462 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:04 GMT /qskrps-47/ 2023-05-24T21:27:04.463 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:JZjeXLi2iW/RoyoLgKjPhYTMNvw= 2023-05-24T21:27:04.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:JZjeXLi2iW/RoyoLgKjPhYTMNvw=', 'Content-Length': '0'} 2023-05-24T21:27:04.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057e71ed6ad984fe9-00646e8128-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.467 DEBUG:boto:b'qskrps-47obj11000falseobj12023-05-24T21:25:49.049Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:04.468 DEBUG:boto:path=/qskrps-47/obj1 2023-05-24T21:27:04.468 DEBUG:boto:auth_path=/qskrps-47/obj1 2023-05-24T21:27:04.469 DEBUG:boto:Method: HEAD 2023-05-24T21:27:04.469 DEBUG:boto:Path: /qskrps-47/obj1 2023-05-24T21:27:04.469 DEBUG:boto:Data: 2023-05-24T21:27:04.470 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.470 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:04.470 DEBUG:boto:Port: 8001 2023-05-24T21:27:04.471 DEBUG:boto:Params: {} 2023-05-24T21:27:04.471 DEBUG:boto:Token: None 2023-05-24T21:27:04.471 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:04 GMT /qskrps-47/obj1 2023-05-24T21:27:04.472 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Jr6fFNLsFa+OmFZKsakihkShcGY= 2023-05-24T21:27:04.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Jr6fFNLsFa+OmFZKsakihkShcGY=', 'Content-Length': '0'} 2023-05-24T21:27:04.474 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000015b207f85f8e665-00646e8128-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.475 DEBUG:boto:path=/qskrps-47/obj1 2023-05-24T21:27:04.475 DEBUG:boto:auth_path=/qskrps-47/obj1 2023-05-24T21:27:04.476 DEBUG:boto:Method: HEAD 2023-05-24T21:27:04.476 DEBUG:boto:Path: /qskrps-47/obj1 2023-05-24T21:27:04.476 DEBUG:boto:Data: 2023-05-24T21:27:04.477 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.477 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:04.477 DEBUG:boto:Port: 8000 2023-05-24T21:27:04.478 DEBUG:boto:Params: {} 2023-05-24T21:27:04.478 DEBUG:boto:Token: None 2023-05-24T21:27:04.478 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:04 GMT /qskrps-47/obj1 2023-05-24T21:27:04.479 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Jr6fFNLsFa+OmFZKsakihkShcGY= 2023-05-24T21:27:04.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Jr6fFNLsFa+OmFZKsakihkShcGY=', 'Content-Length': '0'} 2023-05-24T21:27:04.482 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c84e72e30b7abc91-00646e8128-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-24T21:27:04.483 DEBUG:boto:path=/qskrps-47/obj1 2023-05-24T21:27:04.483 DEBUG:boto:auth_path=/qskrps-47/obj1 2023-05-24T21:27:04.484 DEBUG:boto:Method: GET 2023-05-24T21:27:04.484 DEBUG:boto:Path: /qskrps-47/obj1 2023-05-24T21:27:04.484 DEBUG:boto:Data: 2023-05-24T21:27:04.485 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.485 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:04.485 DEBUG:boto:Port: 8001 2023-05-24T21:27:04.486 DEBUG:boto:Params: {} 2023-05-24T21:27:04.486 DEBUG:boto:Token: None 2023-05-24T21:27:04.487 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:04 GMT /qskrps-47/obj1 2023-05-24T21:27:04.487 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Rp/kRV1X+UeGsb9GZr2SFlitzcM= 2023-05-24T21:27:04.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Rp/kRV1X+UeGsb9GZr2SFlitzcM=', 'Content-Length': '0'} 2023-05-24T21:27:04.490 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ca18ace28b9d40c2-00646e8128-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.537 DEBUG:boto:path=/qskrps-47/obj1 2023-05-24T21:27:04.538 DEBUG:boto:auth_path=/qskrps-47/obj1 2023-05-24T21:27:04.538 DEBUG:boto:Method: GET 2023-05-24T21:27:04.539 DEBUG:boto:Path: /qskrps-47/obj1 2023-05-24T21:27:04.539 DEBUG:boto:Data: 2023-05-24T21:27:04.539 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.540 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:04.540 DEBUG:boto:Port: 8000 2023-05-24T21:27:04.541 DEBUG:boto:Params: {} 2023-05-24T21:27:04.541 DEBUG:boto:Token: None 2023-05-24T21:27:04.541 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:04 GMT /qskrps-47/obj1 2023-05-24T21:27:04.542 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Rp/kRV1X+UeGsb9GZr2SFlitzcM= 2023-05-24T21:27:04.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Rp/kRV1X+UeGsb9GZr2SFlitzcM=', 'Content-Length': '0'} 2023-05-24T21:27:04.544 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000092e4b80d804e7e9e-00646e8128-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.545 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:27:04.545 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:27:04.546 DEBUG:boto:path=/qskrps-47/?prefix=obj1 2023-05-24T21:27:04.546 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj1 2023-05-24T21:27:04.547 DEBUG:boto:Method: GET 2023-05-24T21:27:04.547 DEBUG:boto:Path: /qskrps-47/?prefix=obj1 2023-05-24T21:27:04.548 DEBUG:boto:Data: 2023-05-24T21:27:04.548 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.548 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:04.549 DEBUG:boto:Port: 8001 2023-05-24T21:27:04.549 DEBUG:boto:Params: {} 2023-05-24T21:27:04.550 DEBUG:boto:Token: None 2023-05-24T21:27:04.550 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:04 GMT /qskrps-47/ 2023-05-24T21:27:04.551 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:JZjeXLi2iW/RoyoLgKjPhYTMNvw= 2023-05-24T21:27:04.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:JZjeXLi2iW/RoyoLgKjPhYTMNvw=', 'Content-Length': '0'} 2023-05-24T21:27:04.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000333f071710d98982-00646e8128-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.555 DEBUG:boto:b'qskrps-47obj11000falseobj12023-05-24T21:25:49.049Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:04.556 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:27:04.556 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:27:04.556 DEBUG:boto:path=/qskrps-47/?prefix=obj1 2023-05-24T21:27:04.557 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj1 2023-05-24T21:27:04.557 DEBUG:boto:Method: GET 2023-05-24T21:27:04.557 DEBUG:boto:Path: /qskrps-47/?prefix=obj1 2023-05-24T21:27:04.558 DEBUG:boto:Data: 2023-05-24T21:27:04.558 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.558 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:04.559 DEBUG:boto:Port: 8000 2023-05-24T21:27:04.559 DEBUG:boto:Params: {} 2023-05-24T21:27:04.560 DEBUG:boto:Token: None 2023-05-24T21:27:04.560 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:04 GMT /qskrps-47/ 2023-05-24T21:27:04.560 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:JZjeXLi2iW/RoyoLgKjPhYTMNvw= 2023-05-24T21:27:04.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:JZjeXLi2iW/RoyoLgKjPhYTMNvw=', 'Content-Length': '0'} 2023-05-24T21:27:04.563 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f295b54abb24dc7-00646e8128-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.604 DEBUG:boto:b'qskrps-47obj11000falseobj12023-05-24T21:25:49.049Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:04.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-24T21:27:04.605 DEBUG:boto:path=/qskrps-47/obj2 2023-05-24T21:27:04.606 DEBUG:boto:auth_path=/qskrps-47/obj2 2023-05-24T21:27:04.606 DEBUG:boto:path=/qskrps-47/obj2?versionId=null 2023-05-24T21:27:04.607 DEBUG:boto:auth_path=/qskrps-47/obj2?versionId=null 2023-05-24T21:27:04.607 DEBUG:boto:Method: GET 2023-05-24T21:27:04.607 DEBUG:boto:Path: /qskrps-47/obj2?versionId=null 2023-05-24T21:27:04.608 DEBUG:boto:Data: 2023-05-24T21:27:04.608 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.608 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:04.609 DEBUG:boto:Port: 8001 2023-05-24T21:27:04.609 DEBUG:boto:Params: {} 2023-05-24T21:27:04.609 DEBUG:boto:Token: None 2023-05-24T21:27:04.610 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:04 GMT /qskrps-47/obj2?versionId=null 2023-05-24T21:27:04.610 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4vBJtZWrnF4rFRFl3gHrEXQZK+c= 2023-05-24T21:27:04.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4vBJtZWrnF4rFRFl3gHrEXQZK+c=', 'Content-Length': '0'} 2023-05-24T21:27:04.613 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d25850f37ef76922-00646e8128-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.613 DEBUG:boto:path=/qskrps-47/obj2 2023-05-24T21:27:04.614 DEBUG:boto:auth_path=/qskrps-47/obj2 2023-05-24T21:27:04.614 DEBUG:boto:path=/qskrps-47/obj2?versionId=null 2023-05-24T21:27:04.614 DEBUG:boto:auth_path=/qskrps-47/obj2?versionId=null 2023-05-24T21:27:04.615 DEBUG:boto:Method: GET 2023-05-24T21:27:04.615 DEBUG:boto:Path: /qskrps-47/obj2?versionId=null 2023-05-24T21:27:04.615 DEBUG:boto:Data: 2023-05-24T21:27:04.616 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.616 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:04.616 DEBUG:boto:Port: 8000 2023-05-24T21:27:04.617 DEBUG:boto:Params: {} 2023-05-24T21:27:04.617 DEBUG:boto:Token: None 2023-05-24T21:27:04.618 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:04 GMT /qskrps-47/obj2?versionId=null 2023-05-24T21:27:04.618 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4vBJtZWrnF4rFRFl3gHrEXQZK+c= 2023-05-24T21:27:04.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4vBJtZWrnF4rFRFl3gHrEXQZK+c=', 'Content-Length': '0'} 2023-05-24T21:27:04.620 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007be11bb58041aa46-00646e8128-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.621 DEBUG:boto:path=/qskrps-47/obj2 2023-05-24T21:27:04.621 DEBUG:boto:auth_path=/qskrps-47/obj2 2023-05-24T21:27:04.621 DEBUG:boto:path=/qskrps-47/obj2?versionId=null 2023-05-24T21:27:04.622 DEBUG:boto:auth_path=/qskrps-47/obj2?versionId=null 2023-05-24T21:27:04.622 DEBUG:boto:Method: HEAD 2023-05-24T21:27:04.623 DEBUG:boto:Path: /qskrps-47/obj2?versionId=null 2023-05-24T21:27:04.623 DEBUG:boto:Data: 2023-05-24T21:27:04.623 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.623 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:04.624 DEBUG:boto:Port: 8001 2023-05-24T21:27:04.624 DEBUG:boto:Params: {} 2023-05-24T21:27:04.625 DEBUG:boto:Token: None 2023-05-24T21:27:04.625 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:04 GMT /qskrps-47/obj2?versionId=null 2023-05-24T21:27:04.625 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:MOI1KFL/udnZhGcMjhgqwPgd7vo= 2023-05-24T21:27:04.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:MOI1KFL/udnZhGcMjhgqwPgd7vo=', 'Content-Length': '0'} 2023-05-24T21:27:04.628 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000667e3bf7392e6bc5-00646e8128-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.629 DEBUG:boto:path=/qskrps-47/obj2 2023-05-24T21:27:04.629 DEBUG:boto:auth_path=/qskrps-47/obj2 2023-05-24T21:27:04.629 DEBUG:boto:path=/qskrps-47/obj2?versionId=null 2023-05-24T21:27:04.630 DEBUG:boto:auth_path=/qskrps-47/obj2?versionId=null 2023-05-24T21:27:04.630 DEBUG:boto:Method: HEAD 2023-05-24T21:27:04.630 DEBUG:boto:Path: /qskrps-47/obj2?versionId=null 2023-05-24T21:27:04.631 DEBUG:boto:Data: 2023-05-24T21:27:04.631 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.631 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:04.632 DEBUG:boto:Port: 8000 2023-05-24T21:27:04.632 DEBUG:boto:Params: {} 2023-05-24T21:27:04.632 DEBUG:boto:Token: None 2023-05-24T21:27:04.633 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:04 GMT /qskrps-47/obj2?versionId=null 2023-05-24T21:27:04.633 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:MOI1KFL/udnZhGcMjhgqwPgd7vo= 2023-05-24T21:27:04.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:MOI1KFL/udnZhGcMjhgqwPgd7vo=', 'Content-Length': '0'} 2023-05-24T21:27:04.635 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000235cb39da8b62b98-00646e8128-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-24T21:27:04.636 DEBUG:boto:path=/qskrps-47/obj2 2023-05-24T21:27:04.636 DEBUG:boto:auth_path=/qskrps-47/obj2 2023-05-24T21:27:04.637 DEBUG:boto:path=/qskrps-47/obj2?versionId=null 2023-05-24T21:27:04.637 DEBUG:boto:auth_path=/qskrps-47/obj2?versionId=null 2023-05-24T21:27:04.637 DEBUG:boto:Method: GET 2023-05-24T21:27:04.638 DEBUG:boto:Path: /qskrps-47/obj2?versionId=null 2023-05-24T21:27:04.638 DEBUG:boto:Data: 2023-05-24T21:27:04.638 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.639 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:04.639 DEBUG:boto:Port: 8001 2023-05-24T21:27:04.639 DEBUG:boto:Params: {} 2023-05-24T21:27:04.640 DEBUG:boto:Token: None 2023-05-24T21:27:04.640 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:04 GMT /qskrps-47/obj2?versionId=null 2023-05-24T21:27:04.641 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4vBJtZWrnF4rFRFl3gHrEXQZK+c= 2023-05-24T21:27:04.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4vBJtZWrnF4rFRFl3gHrEXQZK+c=', 'Content-Length': '0'} 2023-05-24T21:27:04.643 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007625f19a919ba797-00646e8128-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.643 DEBUG:boto:path=/qskrps-47/obj2 2023-05-24T21:27:04.644 DEBUG:boto:auth_path=/qskrps-47/obj2 2023-05-24T21:27:04.644 DEBUG:boto:path=/qskrps-47/obj2?versionId=null 2023-05-24T21:27:04.645 DEBUG:boto:auth_path=/qskrps-47/obj2?versionId=null 2023-05-24T21:27:04.645 DEBUG:boto:Method: GET 2023-05-24T21:27:04.645 DEBUG:boto:Path: /qskrps-47/obj2?versionId=null 2023-05-24T21:27:04.646 DEBUG:boto:Data: 2023-05-24T21:27:04.646 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.646 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:04.647 DEBUG:boto:Port: 8000 2023-05-24T21:27:04.647 DEBUG:boto:Params: {} 2023-05-24T21:27:04.647 DEBUG:boto:Token: None 2023-05-24T21:27:04.648 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:04 GMT /qskrps-47/obj2?versionId=null 2023-05-24T21:27:04.648 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4vBJtZWrnF4rFRFl3gHrEXQZK+c= 2023-05-24T21:27:04.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4vBJtZWrnF4rFRFl3gHrEXQZK+c=', 'Content-Length': '0'} 2023-05-24T21:27:04.650 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001bc61bf2e791cb61-00646e8128-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.651 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:27:04.651 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:27:04.652 DEBUG:boto:path=/qskrps-47/?prefix=obj2 2023-05-24T21:27:04.652 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj2 2023-05-24T21:27:04.652 DEBUG:boto:Method: GET 2023-05-24T21:27:04.653 DEBUG:boto:Path: /qskrps-47/?prefix=obj2 2023-05-24T21:27:04.653 DEBUG:boto:Data: 2023-05-24T21:27:04.653 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.654 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:04.654 DEBUG:boto:Port: 8001 2023-05-24T21:27:04.654 DEBUG:boto:Params: {} 2023-05-24T21:27:04.655 DEBUG:boto:Token: None 2023-05-24T21:27:04.655 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:04 GMT /qskrps-47/ 2023-05-24T21:27:04.656 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:JZjeXLi2iW/RoyoLgKjPhYTMNvw= 2023-05-24T21:27:04.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:JZjeXLi2iW/RoyoLgKjPhYTMNvw=', 'Content-Length': '0'} 2023-05-24T21:27:04.658 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a49f3e7d1314a36-00646e8128-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.658 DEBUG:boto:b'qskrps-47obj21000falseobj22023-05-24T21:25:49.067Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:04.659 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:27:04.659 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:27:04.660 DEBUG:boto:path=/qskrps-47/?prefix=obj2 2023-05-24T21:27:04.660 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj2 2023-05-24T21:27:04.661 DEBUG:boto:Method: GET 2023-05-24T21:27:04.661 DEBUG:boto:Path: /qskrps-47/?prefix=obj2 2023-05-24T21:27:04.661 DEBUG:boto:Data: 2023-05-24T21:27:04.662 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.662 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:04.662 DEBUG:boto:Port: 8000 2023-05-24T21:27:04.663 DEBUG:boto:Params: {} 2023-05-24T21:27:04.663 DEBUG:boto:Token: None 2023-05-24T21:27:04.663 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:04 GMT /qskrps-47/ 2023-05-24T21:27:04.664 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:JZjeXLi2iW/RoyoLgKjPhYTMNvw= 2023-05-24T21:27:04.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:JZjeXLi2iW/RoyoLgKjPhYTMNvw=', 'Content-Length': '0'} 2023-05-24T21:27:04.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f8ce7bfd953f82a-00646e8128-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.667 DEBUG:boto:b'qskrps-47obj21000falseobj22023-05-24T21:25:49.067Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:04.667 DEBUG:boto:path=/qskrps-47/obj2 2023-05-24T21:27:04.668 DEBUG:boto:auth_path=/qskrps-47/obj2 2023-05-24T21:27:04.668 DEBUG:boto:Method: HEAD 2023-05-24T21:27:04.669 DEBUG:boto:Path: /qskrps-47/obj2 2023-05-24T21:27:04.669 DEBUG:boto:Data: 2023-05-24T21:27:04.669 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.670 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:04.670 DEBUG:boto:Port: 8001 2023-05-24T21:27:04.670 DEBUG:boto:Params: {} 2023-05-24T21:27:04.671 DEBUG:boto:Token: None 2023-05-24T21:27:04.671 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:04 GMT /qskrps-47/obj2 2023-05-24T21:27:04.671 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:+iB4zToa6NmjRYTPgnyeLz7C82E= 2023-05-24T21:27:04.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:+iB4zToa6NmjRYTPgnyeLz7C82E=', 'Content-Length': '0'} 2023-05-24T21:27:04.673 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005097a5a7659bf22e-00646e8128-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.674 DEBUG:boto:path=/qskrps-47/obj2 2023-05-24T21:27:04.674 DEBUG:boto:auth_path=/qskrps-47/obj2 2023-05-24T21:27:04.675 DEBUG:boto:Method: HEAD 2023-05-24T21:27:04.675 DEBUG:boto:Path: /qskrps-47/obj2 2023-05-24T21:27:04.675 DEBUG:boto:Data: 2023-05-24T21:27:04.676 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.676 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:04.676 DEBUG:boto:Port: 8000 2023-05-24T21:27:04.677 DEBUG:boto:Params: {} 2023-05-24T21:27:04.677 DEBUG:boto:Token: None 2023-05-24T21:27:04.677 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:04 GMT /qskrps-47/obj2 2023-05-24T21:27:04.678 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:+iB4zToa6NmjRYTPgnyeLz7C82E= 2023-05-24T21:27:04.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:+iB4zToa6NmjRYTPgnyeLz7C82E=', 'Content-Length': '0'} 2023-05-24T21:27:04.679 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d6044b5d0fd1cbe5-00646e8128-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.680 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-24T21:27:04.680 DEBUG:boto:path=/qskrps-47/obj2 2023-05-24T21:27:04.681 DEBUG:boto:auth_path=/qskrps-47/obj2 2023-05-24T21:27:04.681 DEBUG:boto:Method: GET 2023-05-24T21:27:04.682 DEBUG:boto:Path: /qskrps-47/obj2 2023-05-24T21:27:04.682 DEBUG:boto:Data: 2023-05-24T21:27:04.682 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.683 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:04.683 DEBUG:boto:Port: 8001 2023-05-24T21:27:04.683 DEBUG:boto:Params: {} 2023-05-24T21:27:04.684 DEBUG:boto:Token: None 2023-05-24T21:27:04.684 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:04 GMT /qskrps-47/obj2 2023-05-24T21:27:04.685 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:q+sgKMh453xY2oFaoQ3HAYTHJv8= 2023-05-24T21:27:04.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:q+sgKMh453xY2oFaoQ3HAYTHJv8=', 'Content-Length': '0'} 2023-05-24T21:27:04.687 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007928006dfdca2e50-00646e8128-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.687 DEBUG:boto:path=/qskrps-47/obj2 2023-05-24T21:27:04.688 DEBUG:boto:auth_path=/qskrps-47/obj2 2023-05-24T21:27:04.688 DEBUG:boto:Method: GET 2023-05-24T21:27:04.688 DEBUG:boto:Path: /qskrps-47/obj2 2023-05-24T21:27:04.689 DEBUG:boto:Data: 2023-05-24T21:27:04.689 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.689 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:04.690 DEBUG:boto:Port: 8000 2023-05-24T21:27:04.690 DEBUG:boto:Params: {} 2023-05-24T21:27:04.691 DEBUG:boto:Token: None 2023-05-24T21:27:04.691 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:04 GMT /qskrps-47/obj2 2023-05-24T21:27:04.691 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:q+sgKMh453xY2oFaoQ3HAYTHJv8= 2023-05-24T21:27:04.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:q+sgKMh453xY2oFaoQ3HAYTHJv8=', 'Content-Length': '0'} 2023-05-24T21:27:04.693 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000024b10a21335f8d4-00646e8128-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.694 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:27:04.694 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:27:04.695 DEBUG:boto:path=/qskrps-47/?prefix=obj2 2023-05-24T21:27:04.695 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj2 2023-05-24T21:27:04.696 DEBUG:boto:Method: GET 2023-05-24T21:27:04.696 DEBUG:boto:Path: /qskrps-47/?prefix=obj2 2023-05-24T21:27:04.696 DEBUG:boto:Data: 2023-05-24T21:27:04.697 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.697 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:04.697 DEBUG:boto:Port: 8001 2023-05-24T21:27:04.698 DEBUG:boto:Params: {} 2023-05-24T21:27:04.698 DEBUG:boto:Token: None 2023-05-24T21:27:04.699 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:04 GMT /qskrps-47/ 2023-05-24T21:27:04.699 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:JZjeXLi2iW/RoyoLgKjPhYTMNvw= 2023-05-24T21:27:04.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:JZjeXLi2iW/RoyoLgKjPhYTMNvw=', 'Content-Length': '0'} 2023-05-24T21:27:04.701 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fdda8d2037694a0-00646e8128-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.702 DEBUG:boto:b'qskrps-47obj21000falseobj22023-05-24T21:25:49.067Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:04.702 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:27:04.703 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:27:04.703 DEBUG:boto:path=/qskrps-47/?prefix=obj2 2023-05-24T21:27:04.704 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj2 2023-05-24T21:27:04.704 DEBUG:boto:Method: GET 2023-05-24T21:27:04.704 DEBUG:boto:Path: /qskrps-47/?prefix=obj2 2023-05-24T21:27:04.705 DEBUG:boto:Data: 2023-05-24T21:27:04.705 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.705 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:04.706 DEBUG:boto:Port: 8000 2023-05-24T21:27:04.706 DEBUG:boto:Params: {} 2023-05-24T21:27:04.706 DEBUG:boto:Token: None 2023-05-24T21:27:04.707 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:04 GMT /qskrps-47/ 2023-05-24T21:27:04.707 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:JZjeXLi2iW/RoyoLgKjPhYTMNvw= 2023-05-24T21:27:04.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:JZjeXLi2iW/RoyoLgKjPhYTMNvw=', 'Content-Length': '0'} 2023-05-24T21:27:04.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057c7f5abb285685b-00646e8128-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.710 DEBUG:boto:b'qskrps-47obj21000falseobj22023-05-24T21:25:49.067Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:04.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-24T21:27:04.711 DEBUG:boto:path=/qskrps-47/obj3 2023-05-24T21:27:04.711 DEBUG:boto:auth_path=/qskrps-47/obj3 2023-05-24T21:27:04.712 DEBUG:boto:path=/qskrps-47/obj3?versionId=null 2023-05-24T21:27:04.712 DEBUG:boto:auth_path=/qskrps-47/obj3?versionId=null 2023-05-24T21:27:04.712 DEBUG:boto:Method: GET 2023-05-24T21:27:04.713 DEBUG:boto:Path: /qskrps-47/obj3?versionId=null 2023-05-24T21:27:04.713 DEBUG:boto:Data: 2023-05-24T21:27:04.714 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.714 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:04.714 DEBUG:boto:Port: 8001 2023-05-24T21:27:04.715 DEBUG:boto:Params: {} 2023-05-24T21:27:04.715 DEBUG:boto:Token: None 2023-05-24T21:27:04.715 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:04 GMT /qskrps-47/obj3?versionId=null 2023-05-24T21:27:04.716 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:e94Bh3UmPuhQwDC9d6jg+fSKOX8= 2023-05-24T21:27:04.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:e94Bh3UmPuhQwDC9d6jg+fSKOX8=', 'Content-Length': '0'} 2023-05-24T21:27:04.718 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000a5e6828ce97217d-00646e8128-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.760 DEBUG:boto:path=/qskrps-47/obj3 2023-05-24T21:27:04.761 DEBUG:boto:auth_path=/qskrps-47/obj3 2023-05-24T21:27:04.761 DEBUG:boto:path=/qskrps-47/obj3?versionId=null 2023-05-24T21:27:04.761 DEBUG:boto:auth_path=/qskrps-47/obj3?versionId=null 2023-05-24T21:27:04.762 DEBUG:boto:Method: GET 2023-05-24T21:27:04.762 DEBUG:boto:Path: /qskrps-47/obj3?versionId=null 2023-05-24T21:27:04.763 DEBUG:boto:Data: 2023-05-24T21:27:04.763 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.763 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:04.764 DEBUG:boto:Port: 8000 2023-05-24T21:27:04.764 DEBUG:boto:Params: {} 2023-05-24T21:27:04.764 DEBUG:boto:Token: None 2023-05-24T21:27:04.765 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:04 GMT /qskrps-47/obj3?versionId=null 2023-05-24T21:27:04.765 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:e94Bh3UmPuhQwDC9d6jg+fSKOX8= 2023-05-24T21:27:04.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:e94Bh3UmPuhQwDC9d6jg+fSKOX8=', 'Content-Length': '0'} 2023-05-24T21:27:04.767 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000058b6eed7c11af408-00646e8128-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.768 DEBUG:boto:path=/qskrps-47/obj3 2023-05-24T21:27:04.768 DEBUG:boto:auth_path=/qskrps-47/obj3 2023-05-24T21:27:04.769 DEBUG:boto:path=/qskrps-47/obj3?versionId=null 2023-05-24T21:27:04.769 DEBUG:boto:auth_path=/qskrps-47/obj3?versionId=null 2023-05-24T21:27:04.769 DEBUG:boto:Method: HEAD 2023-05-24T21:27:04.770 DEBUG:boto:Path: /qskrps-47/obj3?versionId=null 2023-05-24T21:27:04.770 DEBUG:boto:Data: 2023-05-24T21:27:04.770 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.771 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:04.771 DEBUG:boto:Port: 8001 2023-05-24T21:27:04.771 DEBUG:boto:Params: {} 2023-05-24T21:27:04.772 DEBUG:boto:Token: None 2023-05-24T21:27:04.772 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:04 GMT /qskrps-47/obj3?versionId=null 2023-05-24T21:27:04.773 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:/4PWqov7QTFDpn71/6vDMQWdDjI= 2023-05-24T21:27:04.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:/4PWqov7QTFDpn71/6vDMQWdDjI=', 'Content-Length': '0'} 2023-05-24T21:27:04.774 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005dcb755d39e02c55-00646e8128-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.775 DEBUG:boto:path=/qskrps-47/obj3 2023-05-24T21:27:04.775 DEBUG:boto:auth_path=/qskrps-47/obj3 2023-05-24T21:27:04.776 DEBUG:boto:path=/qskrps-47/obj3?versionId=null 2023-05-24T21:27:04.776 DEBUG:boto:auth_path=/qskrps-47/obj3?versionId=null 2023-05-24T21:27:04.776 DEBUG:boto:Method: HEAD 2023-05-24T21:27:04.777 DEBUG:boto:Path: /qskrps-47/obj3?versionId=null 2023-05-24T21:27:04.777 DEBUG:boto:Data: 2023-05-24T21:27:04.777 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.778 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:04.778 DEBUG:boto:Port: 8000 2023-05-24T21:27:04.778 DEBUG:boto:Params: {} 2023-05-24T21:27:04.779 DEBUG:boto:Token: None 2023-05-24T21:27:04.779 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:04 GMT /qskrps-47/obj3?versionId=null 2023-05-24T21:27:04.779 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:/4PWqov7QTFDpn71/6vDMQWdDjI= 2023-05-24T21:27:04.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:/4PWqov7QTFDpn71/6vDMQWdDjI=', 'Content-Length': '0'} 2023-05-24T21:27:04.781 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000704c5053032b6c6c-00646e8128-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-24T21:27:04.782 DEBUG:boto:path=/qskrps-47/obj3 2023-05-24T21:27:04.783 DEBUG:boto:auth_path=/qskrps-47/obj3 2023-05-24T21:27:04.783 DEBUG:boto:path=/qskrps-47/obj3?versionId=null 2023-05-24T21:27:04.783 DEBUG:boto:auth_path=/qskrps-47/obj3?versionId=null 2023-05-24T21:27:04.784 DEBUG:boto:Method: GET 2023-05-24T21:27:04.784 DEBUG:boto:Path: /qskrps-47/obj3?versionId=null 2023-05-24T21:27:04.785 DEBUG:boto:Data: 2023-05-24T21:27:04.785 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.785 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:04.786 DEBUG:boto:Port: 8001 2023-05-24T21:27:04.786 DEBUG:boto:Params: {} 2023-05-24T21:27:04.786 DEBUG:boto:Token: None 2023-05-24T21:27:04.787 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:04 GMT /qskrps-47/obj3?versionId=null 2023-05-24T21:27:04.787 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:e94Bh3UmPuhQwDC9d6jg+fSKOX8= 2023-05-24T21:27:04.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:e94Bh3UmPuhQwDC9d6jg+fSKOX8=', 'Content-Length': '0'} 2023-05-24T21:27:04.789 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007eb9284e61279c6b-00646e8128-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.790 DEBUG:boto:path=/qskrps-47/obj3 2023-05-24T21:27:04.790 DEBUG:boto:auth_path=/qskrps-47/obj3 2023-05-24T21:27:04.790 DEBUG:boto:path=/qskrps-47/obj3?versionId=null 2023-05-24T21:27:04.791 DEBUG:boto:auth_path=/qskrps-47/obj3?versionId=null 2023-05-24T21:27:04.791 DEBUG:boto:Method: GET 2023-05-24T21:27:04.792 DEBUG:boto:Path: /qskrps-47/obj3?versionId=null 2023-05-24T21:27:04.792 DEBUG:boto:Data: 2023-05-24T21:27:04.792 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.793 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:04.793 DEBUG:boto:Port: 8000 2023-05-24T21:27:04.793 DEBUG:boto:Params: {} 2023-05-24T21:27:04.794 DEBUG:boto:Token: None 2023-05-24T21:27:04.794 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:04 GMT /qskrps-47/obj3?versionId=null 2023-05-24T21:27:04.795 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:e94Bh3UmPuhQwDC9d6jg+fSKOX8= 2023-05-24T21:27:04.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:e94Bh3UmPuhQwDC9d6jg+fSKOX8=', 'Content-Length': '0'} 2023-05-24T21:27:04.797 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006cbdee229fc43541-00646e8128-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.797 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:27:04.798 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:27:04.798 DEBUG:boto:path=/qskrps-47/?prefix=obj3 2023-05-24T21:27:04.798 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj3 2023-05-24T21:27:04.799 DEBUG:boto:Method: GET 2023-05-24T21:27:04.799 DEBUG:boto:Path: /qskrps-47/?prefix=obj3 2023-05-24T21:27:04.800 DEBUG:boto:Data: 2023-05-24T21:27:04.800 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.800 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:04.801 DEBUG:boto:Port: 8001 2023-05-24T21:27:04.801 DEBUG:boto:Params: {} 2023-05-24T21:27:04.801 DEBUG:boto:Token: None 2023-05-24T21:27:04.802 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:04 GMT /qskrps-47/ 2023-05-24T21:27:04.802 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:JZjeXLi2iW/RoyoLgKjPhYTMNvw= 2023-05-24T21:27:04.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:JZjeXLi2iW/RoyoLgKjPhYTMNvw=', 'Content-Length': '0'} 2023-05-24T21:27:04.804 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ababa2ac7403baf-00646e8128-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.805 DEBUG:boto:b'qskrps-47obj31000falseobj32023-05-24T21:25:49.084Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:04.806 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:27:04.806 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:27:04.806 DEBUG:boto:path=/qskrps-47/?prefix=obj3 2023-05-24T21:27:04.807 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj3 2023-05-24T21:27:04.807 DEBUG:boto:Method: GET 2023-05-24T21:27:04.808 DEBUG:boto:Path: /qskrps-47/?prefix=obj3 2023-05-24T21:27:04.808 DEBUG:boto:Data: 2023-05-24T21:27:04.808 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.809 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:04.809 DEBUG:boto:Port: 8000 2023-05-24T21:27:04.809 DEBUG:boto:Params: {} 2023-05-24T21:27:04.810 DEBUG:boto:Token: None 2023-05-24T21:27:04.810 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:04 GMT /qskrps-47/ 2023-05-24T21:27:04.811 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:JZjeXLi2iW/RoyoLgKjPhYTMNvw= 2023-05-24T21:27:04.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:JZjeXLi2iW/RoyoLgKjPhYTMNvw=', 'Content-Length': '0'} 2023-05-24T21:27:04.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1ee7868968f9f1e-00646e8128-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.856 DEBUG:boto:b'qskrps-47obj31000falseobj32023-05-24T21:25:49.084Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:04.857 DEBUG:boto:path=/qskrps-47/obj3 2023-05-24T21:27:04.857 DEBUG:boto:auth_path=/qskrps-47/obj3 2023-05-24T21:27:04.858 DEBUG:boto:Method: HEAD 2023-05-24T21:27:04.858 DEBUG:boto:Path: /qskrps-47/obj3 2023-05-24T21:27:04.858 DEBUG:boto:Data: 2023-05-24T21:27:04.859 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.859 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:04.859 DEBUG:boto:Port: 8001 2023-05-24T21:27:04.860 DEBUG:boto:Params: {} 2023-05-24T21:27:04.860 DEBUG:boto:Token: None 2023-05-24T21:27:04.860 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:04 GMT /qskrps-47/obj3 2023-05-24T21:27:04.861 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:3JMYk6R2CkSBwE+PeHhuQn5IMUs= 2023-05-24T21:27:04.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:3JMYk6R2CkSBwE+PeHhuQn5IMUs=', 'Content-Length': '0'} 2023-05-24T21:27:04.863 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000522bd20e2d293ced-00646e8128-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.864 DEBUG:boto:path=/qskrps-47/obj3 2023-05-24T21:27:04.864 DEBUG:boto:auth_path=/qskrps-47/obj3 2023-05-24T21:27:04.864 DEBUG:boto:Method: HEAD 2023-05-24T21:27:04.865 DEBUG:boto:Path: /qskrps-47/obj3 2023-05-24T21:27:04.865 DEBUG:boto:Data: 2023-05-24T21:27:04.865 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.866 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:04.866 DEBUG:boto:Port: 8000 2023-05-24T21:27:04.866 DEBUG:boto:Params: {} 2023-05-24T21:27:04.867 DEBUG:boto:Token: None 2023-05-24T21:27:04.867 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:04 GMT /qskrps-47/obj3 2023-05-24T21:27:04.867 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:3JMYk6R2CkSBwE+PeHhuQn5IMUs= 2023-05-24T21:27:04.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:3JMYk6R2CkSBwE+PeHhuQn5IMUs=', 'Content-Length': '0'} 2023-05-24T21:27:04.869 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b2c5610cadc53e8-00646e8128-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-24T21:27:04.870 DEBUG:boto:path=/qskrps-47/obj3 2023-05-24T21:27:04.871 DEBUG:boto:auth_path=/qskrps-47/obj3 2023-05-24T21:27:04.871 DEBUG:boto:Method: GET 2023-05-24T21:27:04.871 DEBUG:boto:Path: /qskrps-47/obj3 2023-05-24T21:27:04.872 DEBUG:boto:Data: 2023-05-24T21:27:04.872 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.873 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:04.873 DEBUG:boto:Port: 8001 2023-05-24T21:27:04.873 DEBUG:boto:Params: {} 2023-05-24T21:27:04.874 DEBUG:boto:Token: None 2023-05-24T21:27:04.874 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:04 GMT /qskrps-47/obj3 2023-05-24T21:27:04.874 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4JlAHpOykPmfDY7yK3nRjASetg0= 2023-05-24T21:27:04.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4JlAHpOykPmfDY7yK3nRjASetg0=', 'Content-Length': '0'} 2023-05-24T21:27:04.876 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004c0c20f3b05f757c-00646e8128-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.877 DEBUG:boto:path=/qskrps-47/obj3 2023-05-24T21:27:04.877 DEBUG:boto:auth_path=/qskrps-47/obj3 2023-05-24T21:27:04.878 DEBUG:boto:Method: GET 2023-05-24T21:27:04.878 DEBUG:boto:Path: /qskrps-47/obj3 2023-05-24T21:27:04.878 DEBUG:boto:Data: 2023-05-24T21:27:04.879 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.879 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:04.879 DEBUG:boto:Port: 8000 2023-05-24T21:27:04.880 DEBUG:boto:Params: {} 2023-05-24T21:27:04.880 DEBUG:boto:Token: None 2023-05-24T21:27:04.881 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:04 GMT /qskrps-47/obj3 2023-05-24T21:27:04.881 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4JlAHpOykPmfDY7yK3nRjASetg0= 2023-05-24T21:27:04.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4JlAHpOykPmfDY7yK3nRjASetg0=', 'Content-Length': '0'} 2023-05-24T21:27:04.883 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000592c92fe161bd97c-00646e8128-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.884 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:27:04.884 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:27:04.885 DEBUG:boto:path=/qskrps-47/?prefix=obj3 2023-05-24T21:27:04.885 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj3 2023-05-24T21:27:04.885 DEBUG:boto:Method: GET 2023-05-24T21:27:04.886 DEBUG:boto:Path: /qskrps-47/?prefix=obj3 2023-05-24T21:27:04.886 DEBUG:boto:Data: 2023-05-24T21:27:04.886 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.887 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:04.887 DEBUG:boto:Port: 8001 2023-05-24T21:27:04.887 DEBUG:boto:Params: {} 2023-05-24T21:27:04.888 DEBUG:boto:Token: None 2023-05-24T21:27:04.888 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:04 GMT /qskrps-47/ 2023-05-24T21:27:04.889 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:JZjeXLi2iW/RoyoLgKjPhYTMNvw= 2023-05-24T21:27:04.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:JZjeXLi2iW/RoyoLgKjPhYTMNvw=', 'Content-Length': '0'} 2023-05-24T21:27:04.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f108e013b73e8982-00646e8128-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.892 DEBUG:boto:b'qskrps-47obj31000falseobj32023-05-24T21:25:49.084Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:04.892 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:27:04.893 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:27:04.893 DEBUG:boto:path=/qskrps-47/?prefix=obj3 2023-05-24T21:27:04.893 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj3 2023-05-24T21:27:04.894 DEBUG:boto:Method: GET 2023-05-24T21:27:04.894 DEBUG:boto:Path: /qskrps-47/?prefix=obj3 2023-05-24T21:27:04.894 DEBUG:boto:Data: 2023-05-24T21:27:04.895 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.895 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:04.895 DEBUG:boto:Port: 8000 2023-05-24T21:27:04.896 DEBUG:boto:Params: {} 2023-05-24T21:27:04.896 DEBUG:boto:Token: None 2023-05-24T21:27:04.897 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:04 GMT /qskrps-47/ 2023-05-24T21:27:04.897 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:JZjeXLi2iW/RoyoLgKjPhYTMNvw= 2023-05-24T21:27:04.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:JZjeXLi2iW/RoyoLgKjPhYTMNvw=', 'Content-Length': '0'} 2023-05-24T21:27:04.899 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000def40fbf56a0cc9-00646e8128-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.900 DEBUG:boto:b'qskrps-47obj31000falseobj32023-05-24T21:25:49.084Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:04.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-24T21:27:04.901 DEBUG:boto:path=/qskrps-47/obj4 2023-05-24T21:27:04.901 DEBUG:boto:auth_path=/qskrps-47/obj4 2023-05-24T21:27:04.901 DEBUG:boto:path=/qskrps-47/obj4?versionId=null 2023-05-24T21:27:04.902 DEBUG:boto:auth_path=/qskrps-47/obj4?versionId=null 2023-05-24T21:27:04.902 DEBUG:boto:Method: GET 2023-05-24T21:27:04.903 DEBUG:boto:Path: /qskrps-47/obj4?versionId=null 2023-05-24T21:27:04.903 DEBUG:boto:Data: 2023-05-24T21:27:04.903 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.904 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:04.904 DEBUG:boto:Port: 8001 2023-05-24T21:27:04.904 DEBUG:boto:Params: {} 2023-05-24T21:27:04.905 DEBUG:boto:Token: None 2023-05-24T21:27:04.905 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:04 GMT /qskrps-47/obj4?versionId=null 2023-05-24T21:27:04.906 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:2mWpc6p6b9kmz0ceYXDRoQg/udo= 2023-05-24T21:27:04.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:2mWpc6p6b9kmz0ceYXDRoQg/udo=', 'Content-Length': '0'} 2023-05-24T21:27:04.908 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000964bc489b77bd399-00646e8128-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.908 DEBUG:boto:path=/qskrps-47/obj4 2023-05-24T21:27:04.909 DEBUG:boto:auth_path=/qskrps-47/obj4 2023-05-24T21:27:04.909 DEBUG:boto:path=/qskrps-47/obj4?versionId=null 2023-05-24T21:27:04.909 DEBUG:boto:auth_path=/qskrps-47/obj4?versionId=null 2023-05-24T21:27:04.910 DEBUG:boto:Method: GET 2023-05-24T21:27:04.910 DEBUG:boto:Path: /qskrps-47/obj4?versionId=null 2023-05-24T21:27:04.910 DEBUG:boto:Data: 2023-05-24T21:27:04.911 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.911 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:04.911 DEBUG:boto:Port: 8000 2023-05-24T21:27:04.912 DEBUG:boto:Params: {} 2023-05-24T21:27:04.912 DEBUG:boto:Token: None 2023-05-24T21:27:04.913 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:04 GMT /qskrps-47/obj4?versionId=null 2023-05-24T21:27:04.913 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:2mWpc6p6b9kmz0ceYXDRoQg/udo= 2023-05-24T21:27:04.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:2mWpc6p6b9kmz0ceYXDRoQg/udo=', 'Content-Length': '0'} 2023-05-24T21:27:04.915 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f308ccb8c151a3e7-00646e8128-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.916 DEBUG:boto:path=/qskrps-47/obj4 2023-05-24T21:27:04.916 DEBUG:boto:auth_path=/qskrps-47/obj4 2023-05-24T21:27:04.916 DEBUG:boto:path=/qskrps-47/obj4?versionId=null 2023-05-24T21:27:04.917 DEBUG:boto:auth_path=/qskrps-47/obj4?versionId=null 2023-05-24T21:27:04.917 DEBUG:boto:Method: HEAD 2023-05-24T21:27:04.917 DEBUG:boto:Path: /qskrps-47/obj4?versionId=null 2023-05-24T21:27:04.918 DEBUG:boto:Data: 2023-05-24T21:27:04.918 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.918 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:04.919 DEBUG:boto:Port: 8001 2023-05-24T21:27:04.919 DEBUG:boto:Params: {} 2023-05-24T21:27:04.919 DEBUG:boto:Token: None 2023-05-24T21:27:04.920 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:04 GMT /qskrps-47/obj4?versionId=null 2023-05-24T21:27:04.920 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:pCzoQo2UAE1UaqZMm+eSXCIC9eg= 2023-05-24T21:27:04.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:pCzoQo2UAE1UaqZMm+eSXCIC9eg=', 'Content-Length': '0'} 2023-05-24T21:27:04.922 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000658835160c8b4abf-00646e8128-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.923 DEBUG:boto:path=/qskrps-47/obj4 2023-05-24T21:27:04.923 DEBUG:boto:auth_path=/qskrps-47/obj4 2023-05-24T21:27:04.923 DEBUG:boto:path=/qskrps-47/obj4?versionId=null 2023-05-24T21:27:04.924 DEBUG:boto:auth_path=/qskrps-47/obj4?versionId=null 2023-05-24T21:27:04.924 DEBUG:boto:Method: HEAD 2023-05-24T21:27:04.924 DEBUG:boto:Path: /qskrps-47/obj4?versionId=null 2023-05-24T21:27:04.925 DEBUG:boto:Data: 2023-05-24T21:27:04.925 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.925 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:04.926 DEBUG:boto:Port: 8000 2023-05-24T21:27:04.926 DEBUG:boto:Params: {} 2023-05-24T21:27:04.926 DEBUG:boto:Token: None 2023-05-24T21:27:04.927 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:04 GMT /qskrps-47/obj4?versionId=null 2023-05-24T21:27:04.927 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:pCzoQo2UAE1UaqZMm+eSXCIC9eg= 2023-05-24T21:27:04.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:pCzoQo2UAE1UaqZMm+eSXCIC9eg=', 'Content-Length': '0'} 2023-05-24T21:27:04.929 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f43bcaa2b1fdc8c6-00646e8128-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-24T21:27:04.930 DEBUG:boto:path=/qskrps-47/obj4 2023-05-24T21:27:04.930 DEBUG:boto:auth_path=/qskrps-47/obj4 2023-05-24T21:27:04.931 DEBUG:boto:path=/qskrps-47/obj4?versionId=null 2023-05-24T21:27:04.931 DEBUG:boto:auth_path=/qskrps-47/obj4?versionId=null 2023-05-24T21:27:04.931 DEBUG:boto:Method: GET 2023-05-24T21:27:04.932 DEBUG:boto:Path: /qskrps-47/obj4?versionId=null 2023-05-24T21:27:04.932 DEBUG:boto:Data: 2023-05-24T21:27:04.933 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.933 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:04.933 DEBUG:boto:Port: 8001 2023-05-24T21:27:04.934 DEBUG:boto:Params: {} 2023-05-24T21:27:04.934 DEBUG:boto:Token: None 2023-05-24T21:27:04.934 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:04 GMT /qskrps-47/obj4?versionId=null 2023-05-24T21:27:04.935 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:2mWpc6p6b9kmz0ceYXDRoQg/udo= 2023-05-24T21:27:04.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:2mWpc6p6b9kmz0ceYXDRoQg/udo=', 'Content-Length': '0'} 2023-05-24T21:27:04.937 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009eec489e597887df-00646e8128-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.937 DEBUG:boto:path=/qskrps-47/obj4 2023-05-24T21:27:04.938 DEBUG:boto:auth_path=/qskrps-47/obj4 2023-05-24T21:27:04.938 DEBUG:boto:path=/qskrps-47/obj4?versionId=null 2023-05-24T21:27:04.938 DEBUG:boto:auth_path=/qskrps-47/obj4?versionId=null 2023-05-24T21:27:04.939 DEBUG:boto:Method: GET 2023-05-24T21:27:04.939 DEBUG:boto:Path: /qskrps-47/obj4?versionId=null 2023-05-24T21:27:04.939 DEBUG:boto:Data: 2023-05-24T21:27:04.940 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.940 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:04.941 DEBUG:boto:Port: 8000 2023-05-24T21:27:04.941 DEBUG:boto:Params: {} 2023-05-24T21:27:04.941 DEBUG:boto:Token: None 2023-05-24T21:27:04.942 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:04 GMT /qskrps-47/obj4?versionId=null 2023-05-24T21:27:04.942 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:2mWpc6p6b9kmz0ceYXDRoQg/udo= 2023-05-24T21:27:04.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:2mWpc6p6b9kmz0ceYXDRoQg/udo=', 'Content-Length': '0'} 2023-05-24T21:27:04.944 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000061d23a5e5aa4a5c0-00646e8128-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.945 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:27:04.945 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:27:04.946 DEBUG:boto:path=/qskrps-47/?prefix=obj4 2023-05-24T21:27:04.946 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj4 2023-05-24T21:27:04.946 DEBUG:boto:Method: GET 2023-05-24T21:27:04.947 DEBUG:boto:Path: /qskrps-47/?prefix=obj4 2023-05-24T21:27:04.947 DEBUG:boto:Data: 2023-05-24T21:27:04.947 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.948 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:04.948 DEBUG:boto:Port: 8001 2023-05-24T21:27:04.948 DEBUG:boto:Params: {} 2023-05-24T21:27:04.949 DEBUG:boto:Token: None 2023-05-24T21:27:04.949 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:04 GMT /qskrps-47/ 2023-05-24T21:27:04.950 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:JZjeXLi2iW/RoyoLgKjPhYTMNvw= 2023-05-24T21:27:04.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:JZjeXLi2iW/RoyoLgKjPhYTMNvw=', 'Content-Length': '0'} 2023-05-24T21:27:04.952 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c66224c51132cbec-00646e8128-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:04.992 DEBUG:boto:b'qskrps-47obj41000falseobj42023-05-24T21:25:49.101Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:04.993 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:27:04.993 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:27:04.994 DEBUG:boto:path=/qskrps-47/?prefix=obj4 2023-05-24T21:27:04.994 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj4 2023-05-24T21:27:04.994 DEBUG:boto:Method: GET 2023-05-24T21:27:04.995 DEBUG:boto:Path: /qskrps-47/?prefix=obj4 2023-05-24T21:27:04.995 DEBUG:boto:Data: 2023-05-24T21:27:04.995 DEBUG:boto:Headers: {} 2023-05-24T21:27:04.996 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:04.996 DEBUG:boto:Port: 8000 2023-05-24T21:27:04.996 DEBUG:boto:Params: {} 2023-05-24T21:27:04.997 DEBUG:boto:Token: None 2023-05-24T21:27:04.997 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:04 GMT /qskrps-47/ 2023-05-24T21:27:04.998 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:JZjeXLi2iW/RoyoLgKjPhYTMNvw= 2023-05-24T21:27:04.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:04 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:JZjeXLi2iW/RoyoLgKjPhYTMNvw=', 'Content-Length': '0'} 2023-05-24T21:27:05.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001879e7ab74ca14d4-00646e8128-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.001 DEBUG:boto:b'qskrps-47obj41000falseobj42023-05-24T21:25:49.101Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:05.002 DEBUG:boto:path=/qskrps-47/obj4 2023-05-24T21:27:05.002 DEBUG:boto:auth_path=/qskrps-47/obj4 2023-05-24T21:27:05.002 DEBUG:boto:Method: HEAD 2023-05-24T21:27:05.003 DEBUG:boto:Path: /qskrps-47/obj4 2023-05-24T21:27:05.003 DEBUG:boto:Data: 2023-05-24T21:27:05.003 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.004 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:05.004 DEBUG:boto:Port: 8001 2023-05-24T21:27:05.004 DEBUG:boto:Params: {} 2023-05-24T21:27:05.005 DEBUG:boto:Token: None 2023-05-24T21:27:05.005 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:05 GMT /qskrps-47/obj4 2023-05-24T21:27:05.005 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:MXhxdxFq97w0BF/OfhPi72Nd1iw= 2023-05-24T21:27:05.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:MXhxdxFq97w0BF/OfhPi72Nd1iw=', 'Content-Length': '0'} 2023-05-24T21:27:05.007 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000087e87589bd59ad31-00646e8129-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.008 DEBUG:boto:path=/qskrps-47/obj4 2023-05-24T21:27:05.008 DEBUG:boto:auth_path=/qskrps-47/obj4 2023-05-24T21:27:05.009 DEBUG:boto:Method: HEAD 2023-05-24T21:27:05.009 DEBUG:boto:Path: /qskrps-47/obj4 2023-05-24T21:27:05.009 DEBUG:boto:Data: 2023-05-24T21:27:05.010 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.010 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:05.010 DEBUG:boto:Port: 8000 2023-05-24T21:27:05.011 DEBUG:boto:Params: {} 2023-05-24T21:27:05.011 DEBUG:boto:Token: None 2023-05-24T21:27:05.011 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:05 GMT /qskrps-47/obj4 2023-05-24T21:27:05.012 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:MXhxdxFq97w0BF/OfhPi72Nd1iw= 2023-05-24T21:27:05.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:MXhxdxFq97w0BF/OfhPi72Nd1iw=', 'Content-Length': '0'} 2023-05-24T21:27:05.014 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000080eb00b7054e29ad-00646e8129-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-24T21:27:05.015 DEBUG:boto:path=/qskrps-47/obj4 2023-05-24T21:27:05.015 DEBUG:boto:auth_path=/qskrps-47/obj4 2023-05-24T21:27:05.016 DEBUG:boto:Method: GET 2023-05-24T21:27:05.016 DEBUG:boto:Path: /qskrps-47/obj4 2023-05-24T21:27:05.016 DEBUG:boto:Data: 2023-05-24T21:27:05.017 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.017 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:05.018 DEBUG:boto:Port: 8001 2023-05-24T21:27:05.018 DEBUG:boto:Params: {} 2023-05-24T21:27:05.018 DEBUG:boto:Token: None 2023-05-24T21:27:05.019 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:05 GMT /qskrps-47/obj4 2023-05-24T21:27:05.019 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:qsrtl9wH/xcuEOdmThurm20R3cM= 2023-05-24T21:27:05.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qsrtl9wH/xcuEOdmThurm20R3cM=', 'Content-Length': '0'} 2023-05-24T21:27:05.021 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004e7d5581d2fd94af-00646e8129-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.022 DEBUG:boto:path=/qskrps-47/obj4 2023-05-24T21:27:05.022 DEBUG:boto:auth_path=/qskrps-47/obj4 2023-05-24T21:27:05.023 DEBUG:boto:Method: GET 2023-05-24T21:27:05.023 DEBUG:boto:Path: /qskrps-47/obj4 2023-05-24T21:27:05.023 DEBUG:boto:Data: 2023-05-24T21:27:05.024 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.024 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:05.025 DEBUG:boto:Port: 8000 2023-05-24T21:27:05.025 DEBUG:boto:Params: {} 2023-05-24T21:27:05.025 DEBUG:boto:Token: None 2023-05-24T21:27:05.026 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:05 GMT /qskrps-47/obj4 2023-05-24T21:27:05.026 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:qsrtl9wH/xcuEOdmThurm20R3cM= 2023-05-24T21:27:05.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qsrtl9wH/xcuEOdmThurm20R3cM=', 'Content-Length': '0'} 2023-05-24T21:27:05.028 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:25:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000838a4765c883e64e-00646e8129-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.029 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:27:05.029 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:27:05.029 DEBUG:boto:path=/qskrps-47/?prefix=obj4 2023-05-24T21:27:05.030 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj4 2023-05-24T21:27:05.030 DEBUG:boto:Method: GET 2023-05-24T21:27:05.030 DEBUG:boto:Path: /qskrps-47/?prefix=obj4 2023-05-24T21:27:05.031 DEBUG:boto:Data: 2023-05-24T21:27:05.031 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.032 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:05.032 DEBUG:boto:Port: 8001 2023-05-24T21:27:05.032 DEBUG:boto:Params: {} 2023-05-24T21:27:05.033 DEBUG:boto:Token: None 2023-05-24T21:27:05.033 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:05 GMT /qskrps-47/ 2023-05-24T21:27:05.034 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:w6FoUD0FGV59W6YrGMc6YlVuCgY= 2023-05-24T21:27:05.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:w6FoUD0FGV59W6YrGMc6YlVuCgY=', 'Content-Length': '0'} 2023-05-24T21:27:05.036 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030c4a27db7fde618-00646e8129-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.037 DEBUG:boto:b'qskrps-47obj41000falseobj42023-05-24T21:25:49.101Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:05.037 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:27:05.038 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:27:05.038 DEBUG:boto:path=/qskrps-47/?prefix=obj4 2023-05-24T21:27:05.038 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj4 2023-05-24T21:27:05.039 DEBUG:boto:Method: GET 2023-05-24T21:27:05.039 DEBUG:boto:Path: /qskrps-47/?prefix=obj4 2023-05-24T21:27:05.040 DEBUG:boto:Data: 2023-05-24T21:27:05.040 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.040 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:05.041 DEBUG:boto:Port: 8000 2023-05-24T21:27:05.041 DEBUG:boto:Params: {} 2023-05-24T21:27:05.041 DEBUG:boto:Token: None 2023-05-24T21:27:05.042 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:05 GMT /qskrps-47/ 2023-05-24T21:27:05.042 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:w6FoUD0FGV59W6YrGMc6YlVuCgY= 2023-05-24T21:27:05.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:w6FoUD0FGV59W6YrGMc6YlVuCgY=', 'Content-Length': '0'} 2023-05-24T21:27:05.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca8b14c237d6a652-00646e8129-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.088 DEBUG:boto:b'qskrps-47obj41000falseobj42023-05-24T21:25:49.101Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:05.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-05-24T21:27:05.089 DEBUG:boto:path=/qskrps-47/obj5 2023-05-24T21:27:05.090 DEBUG:boto:auth_path=/qskrps-47/obj5 2023-05-24T21:27:05.090 DEBUG:boto:path=/qskrps-47/obj5?versionId=null 2023-05-24T21:27:05.090 DEBUG:boto:auth_path=/qskrps-47/obj5?versionId=null 2023-05-24T21:27:05.091 DEBUG:boto:Method: GET 2023-05-24T21:27:05.091 DEBUG:boto:Path: /qskrps-47/obj5?versionId=null 2023-05-24T21:27:05.091 DEBUG:boto:Data: 2023-05-24T21:27:05.092 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.092 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:05.092 DEBUG:boto:Port: 8001 2023-05-24T21:27:05.093 DEBUG:boto:Params: {} 2023-05-24T21:27:05.093 DEBUG:boto:Token: None 2023-05-24T21:27:05.094 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:05 GMT /qskrps-47/obj5?versionId=null 2023-05-24T21:27:05.094 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:DkHKJ3EoX6YdlvjU3uZewlIOGQA= 2023-05-24T21:27:05.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:DkHKJ3EoX6YdlvjU3uZewlIOGQA=', 'Content-Length': '0'} 2023-05-24T21:27:05.096 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eda69f29f5a0460b-00646e8129-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.097 DEBUG:boto:path=/qskrps-47/obj5 2023-05-24T21:27:05.097 DEBUG:boto:auth_path=/qskrps-47/obj5 2023-05-24T21:27:05.098 DEBUG:boto:path=/qskrps-47/obj5?versionId=null 2023-05-24T21:27:05.098 DEBUG:boto:auth_path=/qskrps-47/obj5?versionId=null 2023-05-24T21:27:05.098 DEBUG:boto:Method: GET 2023-05-24T21:27:05.099 DEBUG:boto:Path: /qskrps-47/obj5?versionId=null 2023-05-24T21:27:05.099 DEBUG:boto:Data: 2023-05-24T21:27:05.099 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.100 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:05.100 DEBUG:boto:Port: 8000 2023-05-24T21:27:05.100 DEBUG:boto:Params: {} 2023-05-24T21:27:05.101 DEBUG:boto:Token: None 2023-05-24T21:27:05.101 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:05 GMT /qskrps-47/obj5?versionId=null 2023-05-24T21:27:05.102 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:DkHKJ3EoX6YdlvjU3uZewlIOGQA= 2023-05-24T21:27:05.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:DkHKJ3EoX6YdlvjU3uZewlIOGQA=', 'Content-Length': '0'} 2023-05-24T21:27:05.104 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000071b6f83cffecb584-00646e8129-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.105 DEBUG:boto:path=/qskrps-47/obj5 2023-05-24T21:27:05.105 DEBUG:boto:auth_path=/qskrps-47/obj5 2023-05-24T21:27:05.105 DEBUG:boto:path=/qskrps-47/obj5?versionId=null 2023-05-24T21:27:05.106 DEBUG:boto:auth_path=/qskrps-47/obj5?versionId=null 2023-05-24T21:27:05.106 DEBUG:boto:Method: HEAD 2023-05-24T21:27:05.106 DEBUG:boto:Path: /qskrps-47/obj5?versionId=null 2023-05-24T21:27:05.107 DEBUG:boto:Data: 2023-05-24T21:27:05.107 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.107 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:05.108 DEBUG:boto:Port: 8001 2023-05-24T21:27:05.108 DEBUG:boto:Params: {} 2023-05-24T21:27:05.108 DEBUG:boto:Token: None 2023-05-24T21:27:05.109 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:05 GMT /qskrps-47/obj5?versionId=null 2023-05-24T21:27:05.109 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:KdpEK44PVUDeItd3RffoOzgU+Fg= 2023-05-24T21:27:05.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:KdpEK44PVUDeItd3RffoOzgU+Fg=', 'Content-Length': '0'} 2023-05-24T21:27:05.111 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cbd1313df6ae5779-00646e8129-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.111 DEBUG:boto:path=/qskrps-47/obj5 2023-05-24T21:27:05.112 DEBUG:boto:auth_path=/qskrps-47/obj5 2023-05-24T21:27:05.112 DEBUG:boto:path=/qskrps-47/obj5?versionId=null 2023-05-24T21:27:05.112 DEBUG:boto:auth_path=/qskrps-47/obj5?versionId=null 2023-05-24T21:27:05.113 DEBUG:boto:Method: HEAD 2023-05-24T21:27:05.113 DEBUG:boto:Path: /qskrps-47/obj5?versionId=null 2023-05-24T21:27:05.113 DEBUG:boto:Data: 2023-05-24T21:27:05.114 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.114 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:05.114 DEBUG:boto:Port: 8000 2023-05-24T21:27:05.115 DEBUG:boto:Params: {} 2023-05-24T21:27:05.115 DEBUG:boto:Token: None 2023-05-24T21:27:05.116 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:05 GMT /qskrps-47/obj5?versionId=null 2023-05-24T21:27:05.116 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:KdpEK44PVUDeItd3RffoOzgU+Fg= 2023-05-24T21:27:05.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:KdpEK44PVUDeItd3RffoOzgU+Fg=', 'Content-Length': '0'} 2023-05-24T21:27:05.118 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e7c4e5f8409c218e-00646e8129-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-05-24T21:27:05.119 DEBUG:boto:path=/qskrps-47/obj5 2023-05-24T21:27:05.119 DEBUG:boto:auth_path=/qskrps-47/obj5 2023-05-24T21:27:05.119 DEBUG:boto:path=/qskrps-47/obj5?versionId=null 2023-05-24T21:27:05.120 DEBUG:boto:auth_path=/qskrps-47/obj5?versionId=null 2023-05-24T21:27:05.120 DEBUG:boto:Method: GET 2023-05-24T21:27:05.121 DEBUG:boto:Path: /qskrps-47/obj5?versionId=null 2023-05-24T21:27:05.121 DEBUG:boto:Data: 2023-05-24T21:27:05.121 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.122 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:05.122 DEBUG:boto:Port: 8001 2023-05-24T21:27:05.122 DEBUG:boto:Params: {} 2023-05-24T21:27:05.123 DEBUG:boto:Token: None 2023-05-24T21:27:05.123 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:05 GMT /qskrps-47/obj5?versionId=null 2023-05-24T21:27:05.123 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:DkHKJ3EoX6YdlvjU3uZewlIOGQA= 2023-05-24T21:27:05.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:DkHKJ3EoX6YdlvjU3uZewlIOGQA=', 'Content-Length': '0'} 2023-05-24T21:27:05.125 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d62ee54b68e75682-00646e8129-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.126 DEBUG:boto:path=/qskrps-47/obj5 2023-05-24T21:27:05.126 DEBUG:boto:auth_path=/qskrps-47/obj5 2023-05-24T21:27:05.127 DEBUG:boto:path=/qskrps-47/obj5?versionId=null 2023-05-24T21:27:05.127 DEBUG:boto:auth_path=/qskrps-47/obj5?versionId=null 2023-05-24T21:27:05.128 DEBUG:boto:Method: GET 2023-05-24T21:27:05.128 DEBUG:boto:Path: /qskrps-47/obj5?versionId=null 2023-05-24T21:27:05.128 DEBUG:boto:Data: 2023-05-24T21:27:05.129 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.129 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:05.130 DEBUG:boto:Port: 8000 2023-05-24T21:27:05.130 DEBUG:boto:Params: {} 2023-05-24T21:27:05.130 DEBUG:boto:Token: None 2023-05-24T21:27:05.131 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:05 GMT /qskrps-47/obj5?versionId=null 2023-05-24T21:27:05.131 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:DkHKJ3EoX6YdlvjU3uZewlIOGQA= 2023-05-24T21:27:05.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:DkHKJ3EoX6YdlvjU3uZewlIOGQA=', 'Content-Length': '0'} 2023-05-24T21:27:05.133 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d10e11baa126f655-00646e8129-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.134 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:27:05.134 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:27:05.135 DEBUG:boto:path=/qskrps-47/?prefix=obj5 2023-05-24T21:27:05.135 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj5 2023-05-24T21:27:05.135 DEBUG:boto:Method: GET 2023-05-24T21:27:05.136 DEBUG:boto:Path: /qskrps-47/?prefix=obj5 2023-05-24T21:27:05.136 DEBUG:boto:Data: 2023-05-24T21:27:05.136 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.137 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:05.137 DEBUG:boto:Port: 8001 2023-05-24T21:27:05.138 DEBUG:boto:Params: {} 2023-05-24T21:27:05.138 DEBUG:boto:Token: None 2023-05-24T21:27:05.138 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:05 GMT /qskrps-47/ 2023-05-24T21:27:05.139 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:w6FoUD0FGV59W6YrGMc6YlVuCgY= 2023-05-24T21:27:05.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:w6FoUD0FGV59W6YrGMc6YlVuCgY=', 'Content-Length': '0'} 2023-05-24T21:27:05.141 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6c4684976271e1e-00646e8129-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.142 DEBUG:boto:b'qskrps-47obj51000falseobj52023-05-24T21:26:22.170Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:05.142 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:27:05.143 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:27:05.143 DEBUG:boto:path=/qskrps-47/?prefix=obj5 2023-05-24T21:27:05.143 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj5 2023-05-24T21:27:05.144 DEBUG:boto:Method: GET 2023-05-24T21:27:05.144 DEBUG:boto:Path: /qskrps-47/?prefix=obj5 2023-05-24T21:27:05.145 DEBUG:boto:Data: 2023-05-24T21:27:05.145 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.145 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:05.146 DEBUG:boto:Port: 8000 2023-05-24T21:27:05.146 DEBUG:boto:Params: {} 2023-05-24T21:27:05.146 DEBUG:boto:Token: None 2023-05-24T21:27:05.147 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:05 GMT /qskrps-47/ 2023-05-24T21:27:05.147 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:w6FoUD0FGV59W6YrGMc6YlVuCgY= 2023-05-24T21:27:05.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:w6FoUD0FGV59W6YrGMc6YlVuCgY=', 'Content-Length': '0'} 2023-05-24T21:27:05.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c6ce8b0f2904228-00646e8129-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.150 DEBUG:boto:b'qskrps-47obj51000falseobj52023-05-24T21:26:22.170Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:05.151 DEBUG:boto:path=/qskrps-47/obj5 2023-05-24T21:27:05.151 DEBUG:boto:auth_path=/qskrps-47/obj5 2023-05-24T21:27:05.152 DEBUG:boto:Method: HEAD 2023-05-24T21:27:05.152 DEBUG:boto:Path: /qskrps-47/obj5 2023-05-24T21:27:05.152 DEBUG:boto:Data: 2023-05-24T21:27:05.153 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.153 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:05.153 DEBUG:boto:Port: 8001 2023-05-24T21:27:05.154 DEBUG:boto:Params: {} 2023-05-24T21:27:05.154 DEBUG:boto:Token: None 2023-05-24T21:27:05.154 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:05 GMT /qskrps-47/obj5 2023-05-24T21:27:05.155 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:wU2iGyYuK8YDz75LE8nScNSSTww= 2023-05-24T21:27:05.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wU2iGyYuK8YDz75LE8nScNSSTww=', 'Content-Length': '0'} 2023-05-24T21:27:05.157 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a4fa3332b8885a82-00646e8129-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.157 DEBUG:boto:path=/qskrps-47/obj5 2023-05-24T21:27:05.158 DEBUG:boto:auth_path=/qskrps-47/obj5 2023-05-24T21:27:05.158 DEBUG:boto:Method: HEAD 2023-05-24T21:27:05.158 DEBUG:boto:Path: /qskrps-47/obj5 2023-05-24T21:27:05.159 DEBUG:boto:Data: 2023-05-24T21:27:05.159 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.159 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:05.160 DEBUG:boto:Port: 8000 2023-05-24T21:27:05.160 DEBUG:boto:Params: {} 2023-05-24T21:27:05.160 DEBUG:boto:Token: None 2023-05-24T21:27:05.161 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:05 GMT /qskrps-47/obj5 2023-05-24T21:27:05.161 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:wU2iGyYuK8YDz75LE8nScNSSTww= 2023-05-24T21:27:05.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wU2iGyYuK8YDz75LE8nScNSSTww=', 'Content-Length': '0'} 2023-05-24T21:27:05.163 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000ed6b031a63893ad-00646e8129-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-05-24T21:27:05.164 DEBUG:boto:path=/qskrps-47/obj5 2023-05-24T21:27:05.164 DEBUG:boto:auth_path=/qskrps-47/obj5 2023-05-24T21:27:05.165 DEBUG:boto:Method: GET 2023-05-24T21:27:05.165 DEBUG:boto:Path: /qskrps-47/obj5 2023-05-24T21:27:05.166 DEBUG:boto:Data: 2023-05-24T21:27:05.166 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.166 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:05.167 DEBUG:boto:Port: 8001 2023-05-24T21:27:05.167 DEBUG:boto:Params: {} 2023-05-24T21:27:05.167 DEBUG:boto:Token: None 2023-05-24T21:27:05.168 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:05 GMT /qskrps-47/obj5 2023-05-24T21:27:05.168 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:nYYiWoxAa90tuuEOcegE0ryVR/k= 2023-05-24T21:27:05.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:nYYiWoxAa90tuuEOcegE0ryVR/k=', 'Content-Length': '0'} 2023-05-24T21:27:05.170 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d8c1311583f99f89-00646e8129-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.212 DEBUG:boto:path=/qskrps-47/obj5 2023-05-24T21:27:05.213 DEBUG:boto:auth_path=/qskrps-47/obj5 2023-05-24T21:27:05.213 DEBUG:boto:Method: GET 2023-05-24T21:27:05.213 DEBUG:boto:Path: /qskrps-47/obj5 2023-05-24T21:27:05.214 DEBUG:boto:Data: 2023-05-24T21:27:05.214 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.214 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:05.215 DEBUG:boto:Port: 8000 2023-05-24T21:27:05.215 DEBUG:boto:Params: {} 2023-05-24T21:27:05.215 DEBUG:boto:Token: None 2023-05-24T21:27:05.216 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:05 GMT /qskrps-47/obj5 2023-05-24T21:27:05.216 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:nYYiWoxAa90tuuEOcegE0ryVR/k= 2023-05-24T21:27:05.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:nYYiWoxAa90tuuEOcegE0ryVR/k=', 'Content-Length': '0'} 2023-05-24T21:27:05.218 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c91090b498d99067-00646e8129-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.219 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:27:05.219 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:27:05.220 DEBUG:boto:path=/qskrps-47/?prefix=obj5 2023-05-24T21:27:05.220 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj5 2023-05-24T21:27:05.221 DEBUG:boto:Method: GET 2023-05-24T21:27:05.221 DEBUG:boto:Path: /qskrps-47/?prefix=obj5 2023-05-24T21:27:05.221 DEBUG:boto:Data: 2023-05-24T21:27:05.222 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.222 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:05.222 DEBUG:boto:Port: 8001 2023-05-24T21:27:05.223 DEBUG:boto:Params: {} 2023-05-24T21:27:05.223 DEBUG:boto:Token: None 2023-05-24T21:27:05.223 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:05 GMT /qskrps-47/ 2023-05-24T21:27:05.224 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:w6FoUD0FGV59W6YrGMc6YlVuCgY= 2023-05-24T21:27:05.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:w6FoUD0FGV59W6YrGMc6YlVuCgY=', 'Content-Length': '0'} 2023-05-24T21:27:05.226 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5ff69d9cf01eee3-00646e8129-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.227 DEBUG:boto:b'qskrps-47obj51000falseobj52023-05-24T21:26:22.170Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:05.227 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:27:05.228 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:27:05.228 DEBUG:boto:path=/qskrps-47/?prefix=obj5 2023-05-24T21:27:05.228 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj5 2023-05-24T21:27:05.229 DEBUG:boto:Method: GET 2023-05-24T21:27:05.229 DEBUG:boto:Path: /qskrps-47/?prefix=obj5 2023-05-24T21:27:05.229 DEBUG:boto:Data: 2023-05-24T21:27:05.230 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.230 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:05.231 DEBUG:boto:Port: 8000 2023-05-24T21:27:05.231 DEBUG:boto:Params: {} 2023-05-24T21:27:05.231 DEBUG:boto:Token: None 2023-05-24T21:27:05.232 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:05 GMT /qskrps-47/ 2023-05-24T21:27:05.232 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:w6FoUD0FGV59W6YrGMc6YlVuCgY= 2023-05-24T21:27:05.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:w6FoUD0FGV59W6YrGMc6YlVuCgY=', 'Content-Length': '0'} 2023-05-24T21:27:05.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067877dbfc2850a4a-00646e8129-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.235 DEBUG:boto:b'qskrps-47obj51000falseobj52023-05-24T21:26:22.170Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:05.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-05-24T21:27:05.236 DEBUG:boto:path=/qskrps-47/obj6 2023-05-24T21:27:05.236 DEBUG:boto:auth_path=/qskrps-47/obj6 2023-05-24T21:27:05.237 DEBUG:boto:path=/qskrps-47/obj6?versionId=null 2023-05-24T21:27:05.237 DEBUG:boto:auth_path=/qskrps-47/obj6?versionId=null 2023-05-24T21:27:05.237 DEBUG:boto:Method: GET 2023-05-24T21:27:05.238 DEBUG:boto:Path: /qskrps-47/obj6?versionId=null 2023-05-24T21:27:05.238 DEBUG:boto:Data: 2023-05-24T21:27:05.238 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.239 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:05.239 DEBUG:boto:Port: 8001 2023-05-24T21:27:05.239 DEBUG:boto:Params: {} 2023-05-24T21:27:05.240 DEBUG:boto:Token: None 2023-05-24T21:27:05.240 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:05 GMT /qskrps-47/obj6?versionId=null 2023-05-24T21:27:05.241 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:jFUh/ZT+z9U7+2r1dAMK8lFhlUM= 2023-05-24T21:27:05.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:jFUh/ZT+z9U7+2r1dAMK8lFhlUM=', 'Content-Length': '0'} 2023-05-24T21:27:05.243 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002a6af5ce7eb467ff-00646e8129-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.243 DEBUG:boto:path=/qskrps-47/obj6 2023-05-24T21:27:05.244 DEBUG:boto:auth_path=/qskrps-47/obj6 2023-05-24T21:27:05.244 DEBUG:boto:path=/qskrps-47/obj6?versionId=null 2023-05-24T21:27:05.244 DEBUG:boto:auth_path=/qskrps-47/obj6?versionId=null 2023-05-24T21:27:05.245 DEBUG:boto:Method: GET 2023-05-24T21:27:05.245 DEBUG:boto:Path: /qskrps-47/obj6?versionId=null 2023-05-24T21:27:05.246 DEBUG:boto:Data: 2023-05-24T21:27:05.246 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.246 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:05.247 DEBUG:boto:Port: 8000 2023-05-24T21:27:05.247 DEBUG:boto:Params: {} 2023-05-24T21:27:05.247 DEBUG:boto:Token: None 2023-05-24T21:27:05.248 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:05 GMT /qskrps-47/obj6?versionId=null 2023-05-24T21:27:05.248 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:jFUh/ZT+z9U7+2r1dAMK8lFhlUM= 2023-05-24T21:27:05.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:jFUh/ZT+z9U7+2r1dAMK8lFhlUM=', 'Content-Length': '0'} 2023-05-24T21:27:05.250 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000779d36580642805f-00646e8129-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.251 DEBUG:boto:path=/qskrps-47/obj6 2023-05-24T21:27:05.251 DEBUG:boto:auth_path=/qskrps-47/obj6 2023-05-24T21:27:05.252 DEBUG:boto:path=/qskrps-47/obj6?versionId=null 2023-05-24T21:27:05.252 DEBUG:boto:auth_path=/qskrps-47/obj6?versionId=null 2023-05-24T21:27:05.252 DEBUG:boto:Method: HEAD 2023-05-24T21:27:05.253 DEBUG:boto:Path: /qskrps-47/obj6?versionId=null 2023-05-24T21:27:05.253 DEBUG:boto:Data: 2023-05-24T21:27:05.253 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.254 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:05.254 DEBUG:boto:Port: 8001 2023-05-24T21:27:05.254 DEBUG:boto:Params: {} 2023-05-24T21:27:05.255 DEBUG:boto:Token: None 2023-05-24T21:27:05.255 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:05 GMT /qskrps-47/obj6?versionId=null 2023-05-24T21:27:05.255 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:JNSppb1T4f3iOWbyxCe01H5d268= 2023-05-24T21:27:05.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:JNSppb1T4f3iOWbyxCe01H5d268=', 'Content-Length': '0'} 2023-05-24T21:27:05.257 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000337b9f96a466dd1b-00646e8129-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.258 DEBUG:boto:path=/qskrps-47/obj6 2023-05-24T21:27:05.258 DEBUG:boto:auth_path=/qskrps-47/obj6 2023-05-24T21:27:05.258 DEBUG:boto:path=/qskrps-47/obj6?versionId=null 2023-05-24T21:27:05.259 DEBUG:boto:auth_path=/qskrps-47/obj6?versionId=null 2023-05-24T21:27:05.259 DEBUG:boto:Method: HEAD 2023-05-24T21:27:05.259 DEBUG:boto:Path: /qskrps-47/obj6?versionId=null 2023-05-24T21:27:05.260 DEBUG:boto:Data: 2023-05-24T21:27:05.260 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.260 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:05.261 DEBUG:boto:Port: 8000 2023-05-24T21:27:05.261 DEBUG:boto:Params: {} 2023-05-24T21:27:05.262 DEBUG:boto:Token: None 2023-05-24T21:27:05.262 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:05 GMT /qskrps-47/obj6?versionId=null 2023-05-24T21:27:05.262 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:JNSppb1T4f3iOWbyxCe01H5d268= 2023-05-24T21:27:05.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:JNSppb1T4f3iOWbyxCe01H5d268=', 'Content-Length': '0'} 2023-05-24T21:27:05.264 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d76f16621e9fd79-00646e8129-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-05-24T21:27:05.265 DEBUG:boto:path=/qskrps-47/obj6 2023-05-24T21:27:05.266 DEBUG:boto:auth_path=/qskrps-47/obj6 2023-05-24T21:27:05.266 DEBUG:boto:path=/qskrps-47/obj6?versionId=null 2023-05-24T21:27:05.266 DEBUG:boto:auth_path=/qskrps-47/obj6?versionId=null 2023-05-24T21:27:05.267 DEBUG:boto:Method: GET 2023-05-24T21:27:05.267 DEBUG:boto:Path: /qskrps-47/obj6?versionId=null 2023-05-24T21:27:05.267 DEBUG:boto:Data: 2023-05-24T21:27:05.268 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.268 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:05.268 DEBUG:boto:Port: 8001 2023-05-24T21:27:05.269 DEBUG:boto:Params: {} 2023-05-24T21:27:05.269 DEBUG:boto:Token: None 2023-05-24T21:27:05.270 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:05 GMT /qskrps-47/obj6?versionId=null 2023-05-24T21:27:05.270 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:jFUh/ZT+z9U7+2r1dAMK8lFhlUM= 2023-05-24T21:27:05.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:jFUh/ZT+z9U7+2r1dAMK8lFhlUM=', 'Content-Length': '0'} 2023-05-24T21:27:05.272 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000175a230cdffe225c-00646e8129-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.272 DEBUG:boto:path=/qskrps-47/obj6 2023-05-24T21:27:05.273 DEBUG:boto:auth_path=/qskrps-47/obj6 2023-05-24T21:27:05.273 DEBUG:boto:path=/qskrps-47/obj6?versionId=null 2023-05-24T21:27:05.274 DEBUG:boto:auth_path=/qskrps-47/obj6?versionId=null 2023-05-24T21:27:05.274 DEBUG:boto:Method: GET 2023-05-24T21:27:05.274 DEBUG:boto:Path: /qskrps-47/obj6?versionId=null 2023-05-24T21:27:05.275 DEBUG:boto:Data: 2023-05-24T21:27:05.275 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.275 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:05.276 DEBUG:boto:Port: 8000 2023-05-24T21:27:05.276 DEBUG:boto:Params: {} 2023-05-24T21:27:05.277 DEBUG:boto:Token: None 2023-05-24T21:27:05.277 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:05 GMT /qskrps-47/obj6?versionId=null 2023-05-24T21:27:05.277 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:jFUh/ZT+z9U7+2r1dAMK8lFhlUM= 2023-05-24T21:27:05.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:jFUh/ZT+z9U7+2r1dAMK8lFhlUM=', 'Content-Length': '0'} 2023-05-24T21:27:05.279 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000259814c130956ca5-00646e8129-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.280 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:27:05.280 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:27:05.281 DEBUG:boto:path=/qskrps-47/?prefix=obj6 2023-05-24T21:27:05.281 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj6 2023-05-24T21:27:05.281 DEBUG:boto:Method: GET 2023-05-24T21:27:05.282 DEBUG:boto:Path: /qskrps-47/?prefix=obj6 2023-05-24T21:27:05.282 DEBUG:boto:Data: 2023-05-24T21:27:05.282 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.283 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:05.283 DEBUG:boto:Port: 8001 2023-05-24T21:27:05.283 DEBUG:boto:Params: {} 2023-05-24T21:27:05.284 DEBUG:boto:Token: None 2023-05-24T21:27:05.284 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:05 GMT /qskrps-47/ 2023-05-24T21:27:05.285 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:w6FoUD0FGV59W6YrGMc6YlVuCgY= 2023-05-24T21:27:05.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:w6FoUD0FGV59W6YrGMc6YlVuCgY=', 'Content-Length': '0'} 2023-05-24T21:27:05.287 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000961a1a33bef6948a-00646e8129-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.288 DEBUG:boto:b'qskrps-47obj61000falseobj62023-05-24T21:26:22.187Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:05.288 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:27:05.289 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:27:05.289 DEBUG:boto:path=/qskrps-47/?prefix=obj6 2023-05-24T21:27:05.289 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj6 2023-05-24T21:27:05.290 DEBUG:boto:Method: GET 2023-05-24T21:27:05.290 DEBUG:boto:Path: /qskrps-47/?prefix=obj6 2023-05-24T21:27:05.291 DEBUG:boto:Data: 2023-05-24T21:27:05.291 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.291 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:05.292 DEBUG:boto:Port: 8000 2023-05-24T21:27:05.292 DEBUG:boto:Params: {} 2023-05-24T21:27:05.292 DEBUG:boto:Token: None 2023-05-24T21:27:05.293 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:05 GMT /qskrps-47/ 2023-05-24T21:27:05.293 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:w6FoUD0FGV59W6YrGMc6YlVuCgY= 2023-05-24T21:27:05.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:w6FoUD0FGV59W6YrGMc6YlVuCgY=', 'Content-Length': '0'} 2023-05-24T21:27:05.296 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093e630a48b70a04c-00646e8129-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.296 DEBUG:boto:b'qskrps-47obj61000falseobj62023-05-24T21:26:22.187Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:05.297 DEBUG:boto:path=/qskrps-47/obj6 2023-05-24T21:27:05.297 DEBUG:boto:auth_path=/qskrps-47/obj6 2023-05-24T21:27:05.297 DEBUG:boto:Method: HEAD 2023-05-24T21:27:05.298 DEBUG:boto:Path: /qskrps-47/obj6 2023-05-24T21:27:05.298 DEBUG:boto:Data: 2023-05-24T21:27:05.298 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.299 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:05.299 DEBUG:boto:Port: 8001 2023-05-24T21:27:05.299 DEBUG:boto:Params: {} 2023-05-24T21:27:05.300 DEBUG:boto:Token: None 2023-05-24T21:27:05.300 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:05 GMT /qskrps-47/obj6 2023-05-24T21:27:05.301 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:AbtWLzho5G4v1m7dVQdrMRaFUF8= 2023-05-24T21:27:05.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:AbtWLzho5G4v1m7dVQdrMRaFUF8=', 'Content-Length': '0'} 2023-05-24T21:27:05.302 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000162505a058951a42-00646e8129-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.303 DEBUG:boto:path=/qskrps-47/obj6 2023-05-24T21:27:05.303 DEBUG:boto:auth_path=/qskrps-47/obj6 2023-05-24T21:27:05.304 DEBUG:boto:Method: HEAD 2023-05-24T21:27:05.304 DEBUG:boto:Path: /qskrps-47/obj6 2023-05-24T21:27:05.304 DEBUG:boto:Data: 2023-05-24T21:27:05.305 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.305 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:05.305 DEBUG:boto:Port: 8000 2023-05-24T21:27:05.306 DEBUG:boto:Params: {} 2023-05-24T21:27:05.306 DEBUG:boto:Token: None 2023-05-24T21:27:05.306 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:05 GMT /qskrps-47/obj6 2023-05-24T21:27:05.307 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:AbtWLzho5G4v1m7dVQdrMRaFUF8= 2023-05-24T21:27:05.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:AbtWLzho5G4v1m7dVQdrMRaFUF8=', 'Content-Length': '0'} 2023-05-24T21:27:05.308 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ab96e68f519c8f87-00646e8129-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-05-24T21:27:05.309 DEBUG:boto:path=/qskrps-47/obj6 2023-05-24T21:27:05.310 DEBUG:boto:auth_path=/qskrps-47/obj6 2023-05-24T21:27:05.310 DEBUG:boto:Method: GET 2023-05-24T21:27:05.311 DEBUG:boto:Path: /qskrps-47/obj6 2023-05-24T21:27:05.311 DEBUG:boto:Data: 2023-05-24T21:27:05.311 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.312 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:05.312 DEBUG:boto:Port: 8001 2023-05-24T21:27:05.312 DEBUG:boto:Params: {} 2023-05-24T21:27:05.313 DEBUG:boto:Token: None 2023-05-24T21:27:05.313 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:05 GMT /qskrps-47/obj6 2023-05-24T21:27:05.314 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:2VeI3N33zi6huxMe0zbN/TVqQUo= 2023-05-24T21:27:05.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:2VeI3N33zi6huxMe0zbN/TVqQUo=', 'Content-Length': '0'} 2023-05-24T21:27:05.316 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000be89daf59ee7fdbe-00646e8129-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.316 DEBUG:boto:path=/qskrps-47/obj6 2023-05-24T21:27:05.317 DEBUG:boto:auth_path=/qskrps-47/obj6 2023-05-24T21:27:05.317 DEBUG:boto:Method: GET 2023-05-24T21:27:05.317 DEBUG:boto:Path: /qskrps-47/obj6 2023-05-24T21:27:05.318 DEBUG:boto:Data: 2023-05-24T21:27:05.318 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.318 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:05.319 DEBUG:boto:Port: 8000 2023-05-24T21:27:05.319 DEBUG:boto:Params: {} 2023-05-24T21:27:05.319 DEBUG:boto:Token: None 2023-05-24T21:27:05.320 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:05 GMT /qskrps-47/obj6 2023-05-24T21:27:05.320 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:2VeI3N33zi6huxMe0zbN/TVqQUo= 2023-05-24T21:27:05.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:2VeI3N33zi6huxMe0zbN/TVqQUo=', 'Content-Length': '0'} 2023-05-24T21:27:05.322 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000080c5b8cae6b959a9-00646e8129-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.364 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:27:05.365 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:27:05.365 DEBUG:boto:path=/qskrps-47/?prefix=obj6 2023-05-24T21:27:05.365 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj6 2023-05-24T21:27:05.366 DEBUG:boto:Method: GET 2023-05-24T21:27:05.366 DEBUG:boto:Path: /qskrps-47/?prefix=obj6 2023-05-24T21:27:05.366 DEBUG:boto:Data: 2023-05-24T21:27:05.367 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.367 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:05.367 DEBUG:boto:Port: 8001 2023-05-24T21:27:05.368 DEBUG:boto:Params: {} 2023-05-24T21:27:05.368 DEBUG:boto:Token: None 2023-05-24T21:27:05.369 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:05 GMT /qskrps-47/ 2023-05-24T21:27:05.369 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:w6FoUD0FGV59W6YrGMc6YlVuCgY= 2023-05-24T21:27:05.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:w6FoUD0FGV59W6YrGMc6YlVuCgY=', 'Content-Length': '0'} 2023-05-24T21:27:05.372 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027a4be4c430091ed-00646e8129-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.372 DEBUG:boto:b'qskrps-47obj61000falseobj62023-05-24T21:26:22.187Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:05.373 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:27:05.373 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:27:05.374 DEBUG:boto:path=/qskrps-47/?prefix=obj6 2023-05-24T21:27:05.374 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj6 2023-05-24T21:27:05.374 DEBUG:boto:Method: GET 2023-05-24T21:27:05.375 DEBUG:boto:Path: /qskrps-47/?prefix=obj6 2023-05-24T21:27:05.375 DEBUG:boto:Data: 2023-05-24T21:27:05.375 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.376 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:05.376 DEBUG:boto:Port: 8000 2023-05-24T21:27:05.376 DEBUG:boto:Params: {} 2023-05-24T21:27:05.377 DEBUG:boto:Token: None 2023-05-24T21:27:05.377 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:05 GMT /qskrps-47/ 2023-05-24T21:27:05.378 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:w6FoUD0FGV59W6YrGMc6YlVuCgY= 2023-05-24T21:27:05.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:w6FoUD0FGV59W6YrGMc6YlVuCgY=', 'Content-Length': '0'} 2023-05-24T21:27:05.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000652288e7680b77c0-00646e8129-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.381 DEBUG:boto:b'qskrps-47obj61000falseobj62023-05-24T21:26:22.187Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:05.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-05-24T21:27:05.382 DEBUG:boto:path=/qskrps-47/obj7 2023-05-24T21:27:05.382 DEBUG:boto:auth_path=/qskrps-47/obj7 2023-05-24T21:27:05.382 DEBUG:boto:path=/qskrps-47/obj7?versionId=null 2023-05-24T21:27:05.383 DEBUG:boto:auth_path=/qskrps-47/obj7?versionId=null 2023-05-24T21:27:05.383 DEBUG:boto:Method: GET 2023-05-24T21:27:05.383 DEBUG:boto:Path: /qskrps-47/obj7?versionId=null 2023-05-24T21:27:05.384 DEBUG:boto:Data: 2023-05-24T21:27:05.384 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.385 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:05.385 DEBUG:boto:Port: 8001 2023-05-24T21:27:05.385 DEBUG:boto:Params: {} 2023-05-24T21:27:05.386 DEBUG:boto:Token: None 2023-05-24T21:27:05.386 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:05 GMT /qskrps-47/obj7?versionId=null 2023-05-24T21:27:05.386 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:LydvOol7BpufvcBNuV5WN4CAFHA= 2023-05-24T21:27:05.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:LydvOol7BpufvcBNuV5WN4CAFHA=', 'Content-Length': '0'} 2023-05-24T21:27:05.388 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b0209bf7193f3c3f-00646e8129-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.389 DEBUG:boto:path=/qskrps-47/obj7 2023-05-24T21:27:05.389 DEBUG:boto:auth_path=/qskrps-47/obj7 2023-05-24T21:27:05.390 DEBUG:boto:path=/qskrps-47/obj7?versionId=null 2023-05-24T21:27:05.390 DEBUG:boto:auth_path=/qskrps-47/obj7?versionId=null 2023-05-24T21:27:05.390 DEBUG:boto:Method: GET 2023-05-24T21:27:05.391 DEBUG:boto:Path: /qskrps-47/obj7?versionId=null 2023-05-24T21:27:05.391 DEBUG:boto:Data: 2023-05-24T21:27:05.391 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.392 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:05.392 DEBUG:boto:Port: 8000 2023-05-24T21:27:05.393 DEBUG:boto:Params: {} 2023-05-24T21:27:05.393 DEBUG:boto:Token: None 2023-05-24T21:27:05.393 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:05 GMT /qskrps-47/obj7?versionId=null 2023-05-24T21:27:05.394 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:LydvOol7BpufvcBNuV5WN4CAFHA= 2023-05-24T21:27:05.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:LydvOol7BpufvcBNuV5WN4CAFHA=', 'Content-Length': '0'} 2023-05-24T21:27:05.396 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000057160ff0768a0cf1-00646e8129-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.397 DEBUG:boto:path=/qskrps-47/obj7 2023-05-24T21:27:05.397 DEBUG:boto:auth_path=/qskrps-47/obj7 2023-05-24T21:27:05.397 DEBUG:boto:path=/qskrps-47/obj7?versionId=null 2023-05-24T21:27:05.398 DEBUG:boto:auth_path=/qskrps-47/obj7?versionId=null 2023-05-24T21:27:05.398 DEBUG:boto:Method: HEAD 2023-05-24T21:27:05.398 DEBUG:boto:Path: /qskrps-47/obj7?versionId=null 2023-05-24T21:27:05.399 DEBUG:boto:Data: 2023-05-24T21:27:05.399 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.399 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:05.400 DEBUG:boto:Port: 8001 2023-05-24T21:27:05.400 DEBUG:boto:Params: {} 2023-05-24T21:27:05.400 DEBUG:boto:Token: None 2023-05-24T21:27:05.401 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:05 GMT /qskrps-47/obj7?versionId=null 2023-05-24T21:27:05.401 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4m9Tajq4N/MGpgZlV0oYYHplAUs= 2023-05-24T21:27:05.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4m9Tajq4N/MGpgZlV0oYYHplAUs=', 'Content-Length': '0'} 2023-05-24T21:27:05.403 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b2ca7e6fcd3ea0f5-00646e8129-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.403 DEBUG:boto:path=/qskrps-47/obj7 2023-05-24T21:27:05.404 DEBUG:boto:auth_path=/qskrps-47/obj7 2023-05-24T21:27:05.404 DEBUG:boto:path=/qskrps-47/obj7?versionId=null 2023-05-24T21:27:05.405 DEBUG:boto:auth_path=/qskrps-47/obj7?versionId=null 2023-05-24T21:27:05.405 DEBUG:boto:Method: HEAD 2023-05-24T21:27:05.405 DEBUG:boto:Path: /qskrps-47/obj7?versionId=null 2023-05-24T21:27:05.406 DEBUG:boto:Data: 2023-05-24T21:27:05.406 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.406 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:05.407 DEBUG:boto:Port: 8000 2023-05-24T21:27:05.407 DEBUG:boto:Params: {} 2023-05-24T21:27:05.407 DEBUG:boto:Token: None 2023-05-24T21:27:05.408 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:05 GMT /qskrps-47/obj7?versionId=null 2023-05-24T21:27:05.408 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4m9Tajq4N/MGpgZlV0oYYHplAUs= 2023-05-24T21:27:05.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4m9Tajq4N/MGpgZlV0oYYHplAUs=', 'Content-Length': '0'} 2023-05-24T21:27:05.410 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000db7e43880f118ef1-00646e8129-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-05-24T21:27:05.411 DEBUG:boto:path=/qskrps-47/obj7 2023-05-24T21:27:05.411 DEBUG:boto:auth_path=/qskrps-47/obj7 2023-05-24T21:27:05.412 DEBUG:boto:path=/qskrps-47/obj7?versionId=null 2023-05-24T21:27:05.412 DEBUG:boto:auth_path=/qskrps-47/obj7?versionId=null 2023-05-24T21:27:05.412 DEBUG:boto:Method: GET 2023-05-24T21:27:05.413 DEBUG:boto:Path: /qskrps-47/obj7?versionId=null 2023-05-24T21:27:05.413 DEBUG:boto:Data: 2023-05-24T21:27:05.413 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.414 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:05.414 DEBUG:boto:Port: 8001 2023-05-24T21:27:05.414 DEBUG:boto:Params: {} 2023-05-24T21:27:05.415 DEBUG:boto:Token: None 2023-05-24T21:27:05.415 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:05 GMT /qskrps-47/obj7?versionId=null 2023-05-24T21:27:05.416 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:LydvOol7BpufvcBNuV5WN4CAFHA= 2023-05-24T21:27:05.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:LydvOol7BpufvcBNuV5WN4CAFHA=', 'Content-Length': '0'} 2023-05-24T21:27:05.418 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b2fcd5b59e746bbf-00646e8129-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.418 DEBUG:boto:path=/qskrps-47/obj7 2023-05-24T21:27:05.419 DEBUG:boto:auth_path=/qskrps-47/obj7 2023-05-24T21:27:05.419 DEBUG:boto:path=/qskrps-47/obj7?versionId=null 2023-05-24T21:27:05.419 DEBUG:boto:auth_path=/qskrps-47/obj7?versionId=null 2023-05-24T21:27:05.420 DEBUG:boto:Method: GET 2023-05-24T21:27:05.420 DEBUG:boto:Path: /qskrps-47/obj7?versionId=null 2023-05-24T21:27:05.420 DEBUG:boto:Data: 2023-05-24T21:27:05.421 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.421 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:05.421 DEBUG:boto:Port: 8000 2023-05-24T21:27:05.422 DEBUG:boto:Params: {} 2023-05-24T21:27:05.422 DEBUG:boto:Token: None 2023-05-24T21:27:05.423 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:05 GMT /qskrps-47/obj7?versionId=null 2023-05-24T21:27:05.423 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:LydvOol7BpufvcBNuV5WN4CAFHA= 2023-05-24T21:27:05.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:LydvOol7BpufvcBNuV5WN4CAFHA=', 'Content-Length': '0'} 2023-05-24T21:27:05.425 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001c7adf5c1bd1451a-00646e8129-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.468 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:27:05.469 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:27:05.470 DEBUG:boto:path=/qskrps-47/?prefix=obj7 2023-05-24T21:27:05.470 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj7 2023-05-24T21:27:05.470 DEBUG:boto:Method: GET 2023-05-24T21:27:05.471 DEBUG:boto:Path: /qskrps-47/?prefix=obj7 2023-05-24T21:27:05.471 DEBUG:boto:Data: 2023-05-24T21:27:05.472 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.472 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:05.472 DEBUG:boto:Port: 8001 2023-05-24T21:27:05.473 DEBUG:boto:Params: {} 2023-05-24T21:27:05.473 DEBUG:boto:Token: None 2023-05-24T21:27:05.474 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:05 GMT /qskrps-47/ 2023-05-24T21:27:05.474 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:w6FoUD0FGV59W6YrGMc6YlVuCgY= 2023-05-24T21:27:05.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:w6FoUD0FGV59W6YrGMc6YlVuCgY=', 'Content-Length': '0'} 2023-05-24T21:27:05.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049b443558c918969-00646e8129-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.478 DEBUG:boto:b'qskrps-47obj71000falseobj72023-05-24T21:26:22.240Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:05.478 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:27:05.479 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:27:05.479 DEBUG:boto:path=/qskrps-47/?prefix=obj7 2023-05-24T21:27:05.479 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj7 2023-05-24T21:27:05.480 DEBUG:boto:Method: GET 2023-05-24T21:27:05.480 DEBUG:boto:Path: /qskrps-47/?prefix=obj7 2023-05-24T21:27:05.481 DEBUG:boto:Data: 2023-05-24T21:27:05.481 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.481 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:05.482 DEBUG:boto:Port: 8000 2023-05-24T21:27:05.482 DEBUG:boto:Params: {} 2023-05-24T21:27:05.482 DEBUG:boto:Token: None 2023-05-24T21:27:05.483 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:05 GMT /qskrps-47/ 2023-05-24T21:27:05.483 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:w6FoUD0FGV59W6YrGMc6YlVuCgY= 2023-05-24T21:27:05.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:w6FoUD0FGV59W6YrGMc6YlVuCgY=', 'Content-Length': '0'} 2023-05-24T21:27:05.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a1da2bd96d84f4d-00646e8129-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.487 DEBUG:boto:b'qskrps-47obj71000falseobj72023-05-24T21:26:22.240Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:05.487 DEBUG:boto:path=/qskrps-47/obj7 2023-05-24T21:27:05.488 DEBUG:boto:auth_path=/qskrps-47/obj7 2023-05-24T21:27:05.488 DEBUG:boto:Method: HEAD 2023-05-24T21:27:05.489 DEBUG:boto:Path: /qskrps-47/obj7 2023-05-24T21:27:05.489 DEBUG:boto:Data: 2023-05-24T21:27:05.489 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.490 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:05.490 DEBUG:boto:Port: 8001 2023-05-24T21:27:05.490 DEBUG:boto:Params: {} 2023-05-24T21:27:05.491 DEBUG:boto:Token: None 2023-05-24T21:27:05.491 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:05 GMT /qskrps-47/obj7 2023-05-24T21:27:05.491 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:2b2FBTdw1SswHSEw33Ic/cA+AjI= 2023-05-24T21:27:05.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:2b2FBTdw1SswHSEw33Ic/cA+AjI=', 'Content-Length': '0'} 2023-05-24T21:27:05.494 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c5cf313c342e51e-00646e8129-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.494 DEBUG:boto:path=/qskrps-47/obj7 2023-05-24T21:27:05.495 DEBUG:boto:auth_path=/qskrps-47/obj7 2023-05-24T21:27:05.495 DEBUG:boto:Method: HEAD 2023-05-24T21:27:05.495 DEBUG:boto:Path: /qskrps-47/obj7 2023-05-24T21:27:05.496 DEBUG:boto:Data: 2023-05-24T21:27:05.496 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.496 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:05.497 DEBUG:boto:Port: 8000 2023-05-24T21:27:05.497 DEBUG:boto:Params: {} 2023-05-24T21:27:05.497 DEBUG:boto:Token: None 2023-05-24T21:27:05.498 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:05 GMT /qskrps-47/obj7 2023-05-24T21:27:05.498 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:2b2FBTdw1SswHSEw33Ic/cA+AjI= 2023-05-24T21:27:05.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:2b2FBTdw1SswHSEw33Ic/cA+AjI=', 'Content-Length': '0'} 2023-05-24T21:27:05.500 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000068de9d333acc3d8d-00646e8129-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-05-24T21:27:05.501 DEBUG:boto:path=/qskrps-47/obj7 2023-05-24T21:27:05.502 DEBUG:boto:auth_path=/qskrps-47/obj7 2023-05-24T21:27:05.502 DEBUG:boto:Method: GET 2023-05-24T21:27:05.502 DEBUG:boto:Path: /qskrps-47/obj7 2023-05-24T21:27:05.503 DEBUG:boto:Data: 2023-05-24T21:27:05.503 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.503 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:05.504 DEBUG:boto:Port: 8001 2023-05-24T21:27:05.504 DEBUG:boto:Params: {} 2023-05-24T21:27:05.505 DEBUG:boto:Token: None 2023-05-24T21:27:05.505 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:05 GMT /qskrps-47/obj7 2023-05-24T21:27:05.506 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:/N3OqYXJ6AqLk6lsg91X0wYC0ag= 2023-05-24T21:27:05.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:/N3OqYXJ6AqLk6lsg91X0wYC0ag=', 'Content-Length': '0'} 2023-05-24T21:27:05.508 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dfcfd93e8e437f46-00646e8129-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.548 DEBUG:boto:path=/qskrps-47/obj7 2023-05-24T21:27:05.549 DEBUG:boto:auth_path=/qskrps-47/obj7 2023-05-24T21:27:05.549 DEBUG:boto:Method: GET 2023-05-24T21:27:05.550 DEBUG:boto:Path: /qskrps-47/obj7 2023-05-24T21:27:05.550 DEBUG:boto:Data: 2023-05-24T21:27:05.550 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.551 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:05.551 DEBUG:boto:Port: 8000 2023-05-24T21:27:05.552 DEBUG:boto:Params: {} 2023-05-24T21:27:05.552 DEBUG:boto:Token: None 2023-05-24T21:27:05.552 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:05 GMT /qskrps-47/obj7 2023-05-24T21:27:05.553 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:/N3OqYXJ6AqLk6lsg91X0wYC0ag= 2023-05-24T21:27:05.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:/N3OqYXJ6AqLk6lsg91X0wYC0ag=', 'Content-Length': '0'} 2023-05-24T21:27:05.555 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007824253d3982ffc1-00646e8129-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.556 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:27:05.556 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:27:05.557 DEBUG:boto:path=/qskrps-47/?prefix=obj7 2023-05-24T21:27:05.557 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj7 2023-05-24T21:27:05.558 DEBUG:boto:Method: GET 2023-05-24T21:27:05.558 DEBUG:boto:Path: /qskrps-47/?prefix=obj7 2023-05-24T21:27:05.558 DEBUG:boto:Data: 2023-05-24T21:27:05.559 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.559 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:05.559 DEBUG:boto:Port: 8001 2023-05-24T21:27:05.560 DEBUG:boto:Params: {} 2023-05-24T21:27:05.560 DEBUG:boto:Token: None 2023-05-24T21:27:05.561 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:05 GMT /qskrps-47/ 2023-05-24T21:27:05.561 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:w6FoUD0FGV59W6YrGMc6YlVuCgY= 2023-05-24T21:27:05.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:w6FoUD0FGV59W6YrGMc6YlVuCgY=', 'Content-Length': '0'} 2023-05-24T21:27:05.564 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001ded9f393f2de74-00646e8129-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.564 DEBUG:boto:b'qskrps-47obj71000falseobj72023-05-24T21:26:22.240Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:05.565 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:27:05.566 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:27:05.566 DEBUG:boto:path=/qskrps-47/?prefix=obj7 2023-05-24T21:27:05.566 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj7 2023-05-24T21:27:05.567 DEBUG:boto:Method: GET 2023-05-24T21:27:05.567 DEBUG:boto:Path: /qskrps-47/?prefix=obj7 2023-05-24T21:27:05.568 DEBUG:boto:Data: 2023-05-24T21:27:05.568 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.568 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:05.569 DEBUG:boto:Port: 8000 2023-05-24T21:27:05.569 DEBUG:boto:Params: {} 2023-05-24T21:27:05.569 DEBUG:boto:Token: None 2023-05-24T21:27:05.570 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:05 GMT /qskrps-47/ 2023-05-24T21:27:05.570 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:w6FoUD0FGV59W6YrGMc6YlVuCgY= 2023-05-24T21:27:05.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:w6FoUD0FGV59W6YrGMc6YlVuCgY=', 'Content-Length': '0'} 2023-05-24T21:27:05.573 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076307e3a7a9e2de2-00646e8129-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.574 DEBUG:boto:b'qskrps-47obj71000falseobj72023-05-24T21:26:22.240Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:05.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-05-24T21:27:05.575 DEBUG:boto:path=/qskrps-47/obj8 2023-05-24T21:27:05.575 DEBUG:boto:auth_path=/qskrps-47/obj8 2023-05-24T21:27:05.576 DEBUG:boto:path=/qskrps-47/obj8?versionId=null 2023-05-24T21:27:05.576 DEBUG:boto:auth_path=/qskrps-47/obj8?versionId=null 2023-05-24T21:27:05.576 DEBUG:boto:Method: GET 2023-05-24T21:27:05.577 DEBUG:boto:Path: /qskrps-47/obj8?versionId=null 2023-05-24T21:27:05.577 DEBUG:boto:Data: 2023-05-24T21:27:05.577 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.578 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:05.578 DEBUG:boto:Port: 8001 2023-05-24T21:27:05.578 DEBUG:boto:Params: {} 2023-05-24T21:27:05.579 DEBUG:boto:Token: None 2023-05-24T21:27:05.579 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:05 GMT /qskrps-47/obj8?versionId=null 2023-05-24T21:27:05.580 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:mdr5cUgd2Cqpu63rDAc2hmurhy8= 2023-05-24T21:27:05.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:mdr5cUgd2Cqpu63rDAc2hmurhy8=', 'Content-Length': '0'} 2023-05-24T21:27:05.582 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e5b87b38ce654599-00646e8129-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.583 DEBUG:boto:path=/qskrps-47/obj8 2023-05-24T21:27:05.583 DEBUG:boto:auth_path=/qskrps-47/obj8 2023-05-24T21:27:05.584 DEBUG:boto:path=/qskrps-47/obj8?versionId=null 2023-05-24T21:27:05.584 DEBUG:boto:auth_path=/qskrps-47/obj8?versionId=null 2023-05-24T21:27:05.584 DEBUG:boto:Method: GET 2023-05-24T21:27:05.585 DEBUG:boto:Path: /qskrps-47/obj8?versionId=null 2023-05-24T21:27:05.585 DEBUG:boto:Data: 2023-05-24T21:27:05.586 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.586 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:05.586 DEBUG:boto:Port: 8000 2023-05-24T21:27:05.587 DEBUG:boto:Params: {} 2023-05-24T21:27:05.587 DEBUG:boto:Token: None 2023-05-24T21:27:05.587 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:05 GMT /qskrps-47/obj8?versionId=null 2023-05-24T21:27:05.588 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:mdr5cUgd2Cqpu63rDAc2hmurhy8= 2023-05-24T21:27:05.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:mdr5cUgd2Cqpu63rDAc2hmurhy8=', 'Content-Length': '0'} 2023-05-24T21:27:05.590 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b4d8197e5b85e0c1-00646e8129-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.591 DEBUG:boto:path=/qskrps-47/obj8 2023-05-24T21:27:05.591 DEBUG:boto:auth_path=/qskrps-47/obj8 2023-05-24T21:27:05.592 DEBUG:boto:path=/qskrps-47/obj8?versionId=null 2023-05-24T21:27:05.592 DEBUG:boto:auth_path=/qskrps-47/obj8?versionId=null 2023-05-24T21:27:05.592 DEBUG:boto:Method: HEAD 2023-05-24T21:27:05.593 DEBUG:boto:Path: /qskrps-47/obj8?versionId=null 2023-05-24T21:27:05.593 DEBUG:boto:Data: 2023-05-24T21:27:05.593 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.594 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:05.594 DEBUG:boto:Port: 8001 2023-05-24T21:27:05.594 DEBUG:boto:Params: {} 2023-05-24T21:27:05.595 DEBUG:boto:Token: None 2023-05-24T21:27:05.595 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:05 GMT /qskrps-47/obj8?versionId=null 2023-05-24T21:27:05.595 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:pWrkup8Vmek29Xmce1X+NE4aKeA= 2023-05-24T21:27:05.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:pWrkup8Vmek29Xmce1X+NE4aKeA=', 'Content-Length': '0'} 2023-05-24T21:27:05.597 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ae77e0e7c6c07b36-00646e8129-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.598 DEBUG:boto:path=/qskrps-47/obj8 2023-05-24T21:27:05.598 DEBUG:boto:auth_path=/qskrps-47/obj8 2023-05-24T21:27:05.599 DEBUG:boto:path=/qskrps-47/obj8?versionId=null 2023-05-24T21:27:05.599 DEBUG:boto:auth_path=/qskrps-47/obj8?versionId=null 2023-05-24T21:27:05.599 DEBUG:boto:Method: HEAD 2023-05-24T21:27:05.600 DEBUG:boto:Path: /qskrps-47/obj8?versionId=null 2023-05-24T21:27:05.600 DEBUG:boto:Data: 2023-05-24T21:27:05.601 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.601 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:05.601 DEBUG:boto:Port: 8000 2023-05-24T21:27:05.602 DEBUG:boto:Params: {} 2023-05-24T21:27:05.602 DEBUG:boto:Token: None 2023-05-24T21:27:05.603 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:05 GMT /qskrps-47/obj8?versionId=null 2023-05-24T21:27:05.603 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:pWrkup8Vmek29Xmce1X+NE4aKeA= 2023-05-24T21:27:05.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:pWrkup8Vmek29Xmce1X+NE4aKeA=', 'Content-Length': '0'} 2023-05-24T21:27:05.605 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cb1702837af36551-00646e8129-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-05-24T21:27:05.606 DEBUG:boto:path=/qskrps-47/obj8 2023-05-24T21:27:05.607 DEBUG:boto:auth_path=/qskrps-47/obj8 2023-05-24T21:27:05.607 DEBUG:boto:path=/qskrps-47/obj8?versionId=null 2023-05-24T21:27:05.607 DEBUG:boto:auth_path=/qskrps-47/obj8?versionId=null 2023-05-24T21:27:05.608 DEBUG:boto:Method: GET 2023-05-24T21:27:05.608 DEBUG:boto:Path: /qskrps-47/obj8?versionId=null 2023-05-24T21:27:05.608 DEBUG:boto:Data: 2023-05-24T21:27:05.609 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.609 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:05.610 DEBUG:boto:Port: 8001 2023-05-24T21:27:05.610 DEBUG:boto:Params: {} 2023-05-24T21:27:05.610 DEBUG:boto:Token: None 2023-05-24T21:27:05.611 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:05 GMT /qskrps-47/obj8?versionId=null 2023-05-24T21:27:05.611 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:mdr5cUgd2Cqpu63rDAc2hmurhy8= 2023-05-24T21:27:05.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:mdr5cUgd2Cqpu63rDAc2hmurhy8=', 'Content-Length': '0'} 2023-05-24T21:27:05.614 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001bea8d299809aa6d-00646e8129-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.614 DEBUG:boto:path=/qskrps-47/obj8 2023-05-24T21:27:05.615 DEBUG:boto:auth_path=/qskrps-47/obj8 2023-05-24T21:27:05.615 DEBUG:boto:path=/qskrps-47/obj8?versionId=null 2023-05-24T21:27:05.616 DEBUG:boto:auth_path=/qskrps-47/obj8?versionId=null 2023-05-24T21:27:05.616 DEBUG:boto:Method: GET 2023-05-24T21:27:05.617 DEBUG:boto:Path: /qskrps-47/obj8?versionId=null 2023-05-24T21:27:05.617 DEBUG:boto:Data: 2023-05-24T21:27:05.617 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.618 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:05.618 DEBUG:boto:Port: 8000 2023-05-24T21:27:05.619 DEBUG:boto:Params: {} 2023-05-24T21:27:05.619 DEBUG:boto:Token: None 2023-05-24T21:27:05.619 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:05 GMT /qskrps-47/obj8?versionId=null 2023-05-24T21:27:05.620 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:mdr5cUgd2Cqpu63rDAc2hmurhy8= 2023-05-24T21:27:05.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:mdr5cUgd2Cqpu63rDAc2hmurhy8=', 'Content-Length': '0'} 2023-05-24T21:27:05.622 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a2421fdabd6cb848-00646e8129-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.623 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:27:05.623 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:27:05.623 DEBUG:boto:path=/qskrps-47/?prefix=obj8 2023-05-24T21:27:05.624 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj8 2023-05-24T21:27:05.624 DEBUG:boto:Method: GET 2023-05-24T21:27:05.625 DEBUG:boto:Path: /qskrps-47/?prefix=obj8 2023-05-24T21:27:05.625 DEBUG:boto:Data: 2023-05-24T21:27:05.625 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.626 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:05.626 DEBUG:boto:Port: 8001 2023-05-24T21:27:05.626 DEBUG:boto:Params: {} 2023-05-24T21:27:05.627 DEBUG:boto:Token: None 2023-05-24T21:27:05.627 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:05 GMT /qskrps-47/ 2023-05-24T21:27:05.628 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:w6FoUD0FGV59W6YrGMc6YlVuCgY= 2023-05-24T21:27:05.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:w6FoUD0FGV59W6YrGMc6YlVuCgY=', 'Content-Length': '0'} 2023-05-24T21:27:05.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009088cfa549d01623-00646e8129-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.631 DEBUG:boto:b'qskrps-47obj81000falseobj82023-05-24T21:26:22.259Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:05.632 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:27:05.632 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:27:05.632 DEBUG:boto:path=/qskrps-47/?prefix=obj8 2023-05-24T21:27:05.633 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj8 2023-05-24T21:27:05.633 DEBUG:boto:Method: GET 2023-05-24T21:27:05.633 DEBUG:boto:Path: /qskrps-47/?prefix=obj8 2023-05-24T21:27:05.634 DEBUG:boto:Data: 2023-05-24T21:27:05.634 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.634 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:05.635 DEBUG:boto:Port: 8000 2023-05-24T21:27:05.635 DEBUG:boto:Params: {} 2023-05-24T21:27:05.636 DEBUG:boto:Token: None 2023-05-24T21:27:05.636 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:05 GMT /qskrps-47/ 2023-05-24T21:27:05.636 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:w6FoUD0FGV59W6YrGMc6YlVuCgY= 2023-05-24T21:27:05.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:w6FoUD0FGV59W6YrGMc6YlVuCgY=', 'Content-Length': '0'} 2023-05-24T21:27:05.639 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b651fc0a27af02bd-00646e8129-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.639 DEBUG:boto:b'qskrps-47obj81000falseobj82023-05-24T21:26:22.259Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:05.640 DEBUG:boto:path=/qskrps-47/obj8 2023-05-24T21:27:05.641 DEBUG:boto:auth_path=/qskrps-47/obj8 2023-05-24T21:27:05.641 DEBUG:boto:Method: HEAD 2023-05-24T21:27:05.641 DEBUG:boto:Path: /qskrps-47/obj8 2023-05-24T21:27:05.642 DEBUG:boto:Data: 2023-05-24T21:27:05.642 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.642 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:05.643 DEBUG:boto:Port: 8001 2023-05-24T21:27:05.643 DEBUG:boto:Params: {} 2023-05-24T21:27:05.643 DEBUG:boto:Token: None 2023-05-24T21:27:05.644 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:05 GMT /qskrps-47/obj8 2023-05-24T21:27:05.644 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:U+QUvcZLKMPADWWnfrqPVWtkbuc= 2023-05-24T21:27:05.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:U+QUvcZLKMPADWWnfrqPVWtkbuc=', 'Content-Length': '0'} 2023-05-24T21:27:05.646 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000066c3c5e362dc0e44-00646e8129-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.647 DEBUG:boto:path=/qskrps-47/obj8 2023-05-24T21:27:05.647 DEBUG:boto:auth_path=/qskrps-47/obj8 2023-05-24T21:27:05.647 DEBUG:boto:Method: HEAD 2023-05-24T21:27:05.648 DEBUG:boto:Path: /qskrps-47/obj8 2023-05-24T21:27:05.648 DEBUG:boto:Data: 2023-05-24T21:27:05.648 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.649 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:05.649 DEBUG:boto:Port: 8000 2023-05-24T21:27:05.649 DEBUG:boto:Params: {} 2023-05-24T21:27:05.650 DEBUG:boto:Token: None 2023-05-24T21:27:05.650 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:05 GMT /qskrps-47/obj8 2023-05-24T21:27:05.651 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:U+QUvcZLKMPADWWnfrqPVWtkbuc= 2023-05-24T21:27:05.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:U+QUvcZLKMPADWWnfrqPVWtkbuc=', 'Content-Length': '0'} 2023-05-24T21:27:05.653 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000809bc585631c76e7-00646e8129-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-05-24T21:27:05.654 DEBUG:boto:path=/qskrps-47/obj8 2023-05-24T21:27:05.654 DEBUG:boto:auth_path=/qskrps-47/obj8 2023-05-24T21:27:05.654 DEBUG:boto:Method: GET 2023-05-24T21:27:05.655 DEBUG:boto:Path: /qskrps-47/obj8 2023-05-24T21:27:05.655 DEBUG:boto:Data: 2023-05-24T21:27:05.655 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.656 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:05.656 DEBUG:boto:Port: 8001 2023-05-24T21:27:05.656 DEBUG:boto:Params: {} 2023-05-24T21:27:05.657 DEBUG:boto:Token: None 2023-05-24T21:27:05.657 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:05 GMT /qskrps-47/obj8 2023-05-24T21:27:05.658 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:PP1aodQZ16fLrRNPFE95EpqcMfs= 2023-05-24T21:27:05.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PP1aodQZ16fLrRNPFE95EpqcMfs=', 'Content-Length': '0'} 2023-05-24T21:27:05.660 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005f71d3b3d63db687-00646e8129-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.661 DEBUG:boto:path=/qskrps-47/obj8 2023-05-24T21:27:05.661 DEBUG:boto:auth_path=/qskrps-47/obj8 2023-05-24T21:27:05.661 DEBUG:boto:Method: GET 2023-05-24T21:27:05.662 DEBUG:boto:Path: /qskrps-47/obj8 2023-05-24T21:27:05.662 DEBUG:boto:Data: 2023-05-24T21:27:05.663 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.663 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:05.663 DEBUG:boto:Port: 8000 2023-05-24T21:27:05.664 DEBUG:boto:Params: {} 2023-05-24T21:27:05.664 DEBUG:boto:Token: None 2023-05-24T21:27:05.664 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:05 GMT /qskrps-47/obj8 2023-05-24T21:27:05.665 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:PP1aodQZ16fLrRNPFE95EpqcMfs= 2023-05-24T21:27:05.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PP1aodQZ16fLrRNPFE95EpqcMfs=', 'Content-Length': '0'} 2023-05-24T21:27:05.667 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:26:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000094f2604f155aa828-00646e8129-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.708 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:27:05.709 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:27:05.709 DEBUG:boto:path=/qskrps-47/?prefix=obj8 2023-05-24T21:27:05.710 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj8 2023-05-24T21:27:05.710 DEBUG:boto:Method: GET 2023-05-24T21:27:05.710 DEBUG:boto:Path: /qskrps-47/?prefix=obj8 2023-05-24T21:27:05.711 DEBUG:boto:Data: 2023-05-24T21:27:05.711 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.712 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:05.712 DEBUG:boto:Port: 8001 2023-05-24T21:27:05.712 DEBUG:boto:Params: {} 2023-05-24T21:27:05.713 DEBUG:boto:Token: None 2023-05-24T21:27:05.713 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:05 GMT /qskrps-47/ 2023-05-24T21:27:05.714 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:w6FoUD0FGV59W6YrGMc6YlVuCgY= 2023-05-24T21:27:05.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:w6FoUD0FGV59W6YrGMc6YlVuCgY=', 'Content-Length': '0'} 2023-05-24T21:27:05.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2513459d946983c-00646e8129-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.717 DEBUG:boto:b'qskrps-47obj81000falseobj82023-05-24T21:26:22.259Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:05.718 DEBUG:boto:path=/qskrps-47/ 2023-05-24T21:27:05.718 DEBUG:boto:auth_path=/qskrps-47/ 2023-05-24T21:27:05.718 DEBUG:boto:path=/qskrps-47/?prefix=obj8 2023-05-24T21:27:05.719 DEBUG:boto:auth_path=/qskrps-47/?prefix=obj8 2023-05-24T21:27:05.719 DEBUG:boto:Method: GET 2023-05-24T21:27:05.719 DEBUG:boto:Path: /qskrps-47/?prefix=obj8 2023-05-24T21:27:05.720 DEBUG:boto:Data: 2023-05-24T21:27:05.720 DEBUG:boto:Headers: {} 2023-05-24T21:27:05.721 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:05.721 DEBUG:boto:Port: 8000 2023-05-24T21:27:05.721 DEBUG:boto:Params: {} 2023-05-24T21:27:05.722 DEBUG:boto:Token: None 2023-05-24T21:27:05.722 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:05 GMT /qskrps-47/ 2023-05-24T21:27:05.723 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:w6FoUD0FGV59W6YrGMc6YlVuCgY= 2023-05-24T21:27:05.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:w6FoUD0FGV59W6YrGMc6YlVuCgY=', 'Content-Length': '0'} 2023-05-24T21:27:05.725 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff2e561ff97b1d22-00646e8129-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:05.726 DEBUG:boto:b'qskrps-47obj81000falseobj82023-05-24T21:26:22.259Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:05.726 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qskrps-47 zones={a2, a1} 2023-05-24T21:27:05.727 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-05-24T21:27:05.727 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:27:05.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:27:05.728 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:27:05.779 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:27:05.780 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:27:05.857 INFO:teuthology.orchestra.run.smithi071.stdout:[ 2023-05-24T21:27:05.857 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:27:05.858 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000005582", 2023-05-24T21:27:05.858 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:26:22.031494Z" 2023-05-24T21:27:05.858 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:27:05.858 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:27:05.858 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000004316", 2023-05-24T21:27:05.858 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:26:38.637472Z" 2023-05-24T21:27:05.859 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:27:05.859 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:27:05.859 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000002903", 2023-05-24T21:27:05.859 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:26:38.844590Z" 2023-05-24T21:27:05.859 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:27:05.859 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:27:05.859 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000008256", 2023-05-24T21:27:05.860 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:26:30.021686Z" 2023-05-24T21:27:05.860 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:27:05.860 INFO:teuthology.orchestra.run.smithi071.stdout:] 2023-05-24T21:27:05.861 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000005582', 1: '00000000000000000000:00000000000000004316', 2: '00000000000000000000:00000000000000002903', 3: '00000000000000000000:00000000000000008256'} 2023-05-24T21:27:05.861 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-05-24T21:27:05.862 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:27:05.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:27:05.863 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:27:05.913 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:27:05.914 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:27:05.972 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:27:05.972 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:27:05.972 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:27:05.972 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:27:05.973 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:27:05.973 INFO:teuthology.orchestra.run.smithi071.stdout: "instance_id": 13190874283549403954 2023-05-24T21:27:05.973 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:27:05.973 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:27:05.973 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:27:05.973 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:27:05.974 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:27:05.974 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:27:05.974 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000005582", 2023-05-24T21:27:05.974 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:27:05.974 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:27:05.974 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:27:05.974 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:26:22.031494Z" 2023-05-24T21:27:05.975 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:27:05.975 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:27:05.975 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:27:05.975 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:27:05.975 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:27:05.975 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:27:05.976 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000004316", 2023-05-24T21:27:05.976 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:27:05.976 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:27:05.976 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:27:05.976 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:26:38.637472Z" 2023-05-24T21:27:05.976 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:27:05.976 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:27:05.977 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:27:05.977 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:27:05.977 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:27:05.977 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:27:05.977 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000002903", 2023-05-24T21:27:05.977 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:27:05.977 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:27:05.978 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:27:05.978 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:26:38.844590Z" 2023-05-24T21:27:05.978 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:27:05.978 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:27:05.978 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:27:05.978 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:27:05.979 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:27:05.979 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:27:05.979 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000008256", 2023-05-24T21:27:05.979 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:27:05.979 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:27:05.979 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:27:05.979 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:26:30.021686Z" 2023-05-24T21:27:05.980 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:27:05.980 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:27:05.980 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:27:05.980 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:27:05.980 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:27:05.980 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 0, 2023-05-24T21:27:05.980 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 0 2023-05-24T21:27:05.981 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:27:05.981 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:27:05Z" 2023-05-24T21:27:05.981 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:27:05.982 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13190874283549403954 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005582", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:26:22.031494Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004316", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:26:38.637472Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002903", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:26:38.844590Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008256", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:26:30.021686Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-05-24T21:27:05Z" } 2023-05-24T21:27:05.982 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:26:22.031494Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004316', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:26:38.637472Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002903', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:26:38.844590Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008256', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:26:30.021686Z'}}] 2023-05-24T21:27:05.983 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000005582', 1: '00000000000000000000:00000000000000004316', 2: '00000000000000000000:00000000000000002903', 3: '00000000000000000000:00000000000000008256'} 2023-05-24T21:27:05.983 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000005582', 1: '00000000000000000000:00000000000000004316', 2: '00000000000000000000:00000000000000002903', 3: '00000000000000000000:00000000000000008256'} 2023-05-24T21:27:05.983 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-05-24T21:27:05.984 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-05-24T21:27:05.984 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:27:05.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:27:05.985 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:27:06.034 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:27:06.035 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:27:06.092 INFO:teuthology.orchestra.run.smithi071.stdout:[ 2023-05-24T21:27:06.093 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:27:06.093 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000003329", 2023-05-24T21:27:06.093 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:25:49.020046Z" 2023-05-24T21:27:06.093 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:27:06.093 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:27:06.093 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000002620", 2023-05-24T21:27:06.094 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:26:39.276828Z" 2023-05-24T21:27:06.094 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:27:06.094 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:27:06.094 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000001775", 2023-05-24T21:27:06.094 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:26:39.278679Z" 2023-05-24T21:27:06.094 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:27:06.095 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:27:06.095 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000006705", 2023-05-24T21:27:06.095 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:26:39.280349Z" 2023-05-24T21:27:06.095 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:27:06.095 INFO:teuthology.orchestra.run.smithi071.stdout:] 2023-05-24T21:27:06.096 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000003329', 1: '00000000000000000000:00000000000000002620', 2: '00000000000000000000:00000000000000001775', 3: '00000000000000000000:00000000000000006705'} 2023-05-24T21:27:06.097 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-05-24T21:27:06.097 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:27:06.098 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:27:06.098 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:27:06.149 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:27:06.150 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:27:06.206 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:27:06.207 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:27:06.207 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:27:06.207 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:27:06.207 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:27:06.208 INFO:teuthology.orchestra.run.smithi071.stdout: "instance_id": 6585121019900467362 2023-05-24T21:27:06.208 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:27:06.208 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:27:06.208 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:27:06.208 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:27:06.208 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:27:06.209 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:27:06.209 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000003329", 2023-05-24T21:27:06.209 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:27:06.209 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:27:06.209 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:27:06.209 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:25:49.020046Z" 2023-05-24T21:27:06.210 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:27:06.210 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:27:06.210 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:27:06.210 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:27:06.210 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:27:06.210 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:27:06.211 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000002620", 2023-05-24T21:27:06.211 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:27:06.211 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:27:06.211 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:27:06.211 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:26:39.276828Z" 2023-05-24T21:27:06.211 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:27:06.211 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:27:06.212 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:27:06.212 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:27:06.212 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:27:06.212 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:27:06.212 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000001775", 2023-05-24T21:27:06.212 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:27:06.212 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:27:06.213 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:27:06.213 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:26:39.278679Z" 2023-05-24T21:27:06.213 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:27:06.213 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:27:06.213 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:27:06.213 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:27:06.213 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:27:06.214 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:27:06.214 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000006705", 2023-05-24T21:27:06.214 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:27:06.214 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:27:06.214 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:27:06.214 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:26:39.280349Z" 2023-05-24T21:27:06.215 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:27:06.215 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:27:06.215 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:27:06.215 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:27:06.215 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:27:06.215 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 0, 2023-05-24T21:27:06.215 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 0 2023-05-24T21:27:06.216 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:27:06.216 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:27:06Z" 2023-05-24T21:27:06.216 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:27:06.217 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6585121019900467362 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003329", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:25:49.020046Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002620", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:26:39.276828Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001775", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:26:39.278679Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006705", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:26:39.280349Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-05-24T21:27:06Z" } 2023-05-24T21:27:06.217 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003329', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:25:49.020046Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002620', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:26:39.276828Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001775', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:26:39.278679Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006705', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:26:39.280349Z'}}] 2023-05-24T21:27:06.218 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003329', 1: '00000000000000000000:00000000000000002620', 2: '00000000000000000000:00000000000000001775', 3: '00000000000000000000:00000000000000006705'} 2023-05-24T21:27:06.218 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003329', 1: '00000000000000000000:00000000000000002620', 2: '00000000000000000000:00000000000000001775', 3: '00000000000000000000:00000000000000006705'} 2023-05-24T21:27:06.219 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-05-24T21:27:06.219 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... ok 2023-05-24T21:27:06.220 INFO:rgw_multi.tests:create bucket zone=a1 name=qskrps-48 2023-05-24T21:27:06.221 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:27:06.221 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:27:06.221 DEBUG:boto:Method: PUT 2023-05-24T21:27:06.222 DEBUG:boto:Path: /qskrps-48/ 2023-05-24T21:27:06.222 DEBUG:boto:Data: 2023-05-24T21:27:06.222 DEBUG:boto:Headers: {} 2023-05-24T21:27:06.223 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:06.223 DEBUG:boto:Port: 8000 2023-05-24T21:27:06.223 DEBUG:boto:Params: {} 2023-05-24T21:27:06.224 DEBUG:boto:Token: None 2023-05-24T21:27:06.224 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:27:06 GMT /qskrps-48/ 2023-05-24T21:27:06.225 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:cboCHTgTeIkg6xssRcBAQMrNLZ8= 2023-05-24T21:27:06.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:06 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:cboCHTgTeIkg6xssRcBAQMrNLZ8=', 'Content-Length': '0'} 2023-05-24T21:27:06.253 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ab0fd7360db79443-00646e812a-1627-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:06.254 INFO:rgw_multi.tests:create bucket zone=a2 name=qskrps-49 2023-05-24T21:27:06.254 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:27:06.254 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:27:06.255 DEBUG:boto:Method: PUT 2023-05-24T21:27:06.255 DEBUG:boto:Path: /qskrps-49/ 2023-05-24T21:27:06.255 DEBUG:boto:Data: 2023-05-24T21:27:06.256 DEBUG:boto:Headers: {} 2023-05-24T21:27:06.256 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:06.256 DEBUG:boto:Port: 8001 2023-05-24T21:27:06.257 DEBUG:boto:Params: {} 2023-05-24T21:27:06.257 DEBUG:boto:Token: None 2023-05-24T21:27:06.257 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:27:06 GMT /qskrps-49/ 2023-05-24T21:27:06.258 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:yaYd6zSuSZIGMO4oQ/Znw+ynwZo= 2023-05-24T21:27:06.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:06 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:yaYd6zSuSZIGMO4oQ/Znw+ynwZo=', 'Content-Length': '0'} 2023-05-24T21:27:06.300 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c21e42ed1901f71c-00646e812a-1625-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:06.301 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:27:06.301 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:27:06.302 DEBUG:boto:Method: HEAD 2023-05-24T21:27:06.302 DEBUG:boto:Path: /qskrps-48/ 2023-05-24T21:27:06.302 DEBUG:boto:Data: 2023-05-24T21:27:06.303 DEBUG:boto:Headers: {} 2023-05-24T21:27:06.303 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:06.303 DEBUG:boto:Port: 8000 2023-05-24T21:27:06.304 DEBUG:boto:Params: {} 2023-05-24T21:27:06.304 DEBUG:boto:Token: None 2023-05-24T21:27:06.305 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:06 GMT /qskrps-48/ 2023-05-24T21:27:06.305 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:V5A8nIK0Jgzr6CqeXTN5Ixbqflc= 2023-05-24T21:27:06.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:06 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:V5A8nIK0Jgzr6CqeXTN5Ixbqflc=', 'Content-Length': '0'} 2023-05-24T21:27:06.308 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000084b8bd1d662ed50f-00646e812a-1627-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:06.308 DEBUG:boto:path=/qskrps-48/obj1 2023-05-24T21:27:06.309 DEBUG:boto:auth_path=/qskrps-48/obj1 2023-05-24T21:27:06.309 DEBUG:boto:Method: PUT 2023-05-24T21:27:06.309 DEBUG:boto:Path: /qskrps-48/obj1 2023-05-24T21:27:06.310 DEBUG:boto:Data: 2023-05-24T21:27:06.310 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-24T21:27:06.310 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:06.311 DEBUG:boto:Port: 8000 2023-05-24T21:27:06.311 DEBUG:boto:Params: {} 2023-05-24T21:27:06.311 DEBUG:boto:Token: None 2023-05-24T21:27:06.312 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 24 May 2023 21:27:06 GMT /qskrps-48/obj1 2023-05-24T21:27:06.312 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:+KTnMJyY7s6ALGXuN3jNeI5K2TI= 2023-05-24T21:27:06.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:27:06 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:+KTnMJyY7s6ALGXuN3jNeI5K2TI='} 2023-05-24T21:27:06.321 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e7920931aaeebc79-00646e812a-1627-a1'), ('Date', 'Wed, 24 May 2023 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:06.322 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:27:06.322 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:27:06.322 DEBUG:boto:Method: HEAD 2023-05-24T21:27:06.323 DEBUG:boto:Path: /qskrps-48/ 2023-05-24T21:27:06.323 DEBUG:boto:Data: 2023-05-24T21:27:06.323 DEBUG:boto:Headers: {} 2023-05-24T21:27:06.324 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:06.324 DEBUG:boto:Port: 8000 2023-05-24T21:27:06.324 DEBUG:boto:Params: {} 2023-05-24T21:27:06.325 DEBUG:boto:Token: None 2023-05-24T21:27:06.325 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:06 GMT /qskrps-48/ 2023-05-24T21:27:06.326 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:V5A8nIK0Jgzr6CqeXTN5Ixbqflc= 2023-05-24T21:27:06.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:06 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:V5A8nIK0Jgzr6CqeXTN5Ixbqflc=', 'Content-Length': '0'} 2023-05-24T21:27:06.328 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003b9daf6eec5b4388-00646e812a-1627-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:06.329 DEBUG:boto:path=/qskrps-48/obj2 2023-05-24T21:27:06.329 DEBUG:boto:auth_path=/qskrps-48/obj2 2023-05-24T21:27:06.329 DEBUG:boto:Method: PUT 2023-05-24T21:27:06.330 DEBUG:boto:Path: /qskrps-48/obj2 2023-05-24T21:27:06.330 DEBUG:boto:Data: 2023-05-24T21:27:06.330 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-24T21:27:06.331 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:06.331 DEBUG:boto:Port: 8000 2023-05-24T21:27:06.331 DEBUG:boto:Params: {} 2023-05-24T21:27:06.332 DEBUG:boto:Token: None 2023-05-24T21:27:06.332 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 24 May 2023 21:27:06 GMT /qskrps-48/obj2 2023-05-24T21:27:06.332 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:6g32GV/2H54F1LIQHZHn3e+Cop8= 2023-05-24T21:27:06.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:27:06 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:6g32GV/2H54F1LIQHZHn3e+Cop8='} 2023-05-24T21:27:06.339 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004d712ae5cbab6e84-00646e812a-1627-a1'), ('Date', 'Wed, 24 May 2023 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:06.340 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:27:06.340 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:27:06.340 DEBUG:boto:Method: HEAD 2023-05-24T21:27:06.341 DEBUG:boto:Path: /qskrps-48/ 2023-05-24T21:27:06.341 DEBUG:boto:Data: 2023-05-24T21:27:06.341 DEBUG:boto:Headers: {} 2023-05-24T21:27:06.342 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:06.342 DEBUG:boto:Port: 8000 2023-05-24T21:27:06.342 DEBUG:boto:Params: {} 2023-05-24T21:27:06.343 DEBUG:boto:Token: None 2023-05-24T21:27:06.343 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:06 GMT /qskrps-48/ 2023-05-24T21:27:06.343 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:V5A8nIK0Jgzr6CqeXTN5Ixbqflc= 2023-05-24T21:27:06.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:06 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:V5A8nIK0Jgzr6CqeXTN5Ixbqflc=', 'Content-Length': '0'} 2023-05-24T21:27:06.346 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cd0d47d7919d6d9c-00646e812a-1627-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:06.347 DEBUG:boto:path=/qskrps-48/obj3 2023-05-24T21:27:06.347 DEBUG:boto:auth_path=/qskrps-48/obj3 2023-05-24T21:27:06.347 DEBUG:boto:Method: PUT 2023-05-24T21:27:06.348 DEBUG:boto:Path: /qskrps-48/obj3 2023-05-24T21:27:06.348 DEBUG:boto:Data: 2023-05-24T21:27:06.348 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-24T21:27:06.349 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:06.349 DEBUG:boto:Port: 8000 2023-05-24T21:27:06.349 DEBUG:boto:Params: {} 2023-05-24T21:27:06.350 DEBUG:boto:Token: None 2023-05-24T21:27:06.350 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 24 May 2023 21:27:06 GMT /qskrps-48/obj3 2023-05-24T21:27:06.351 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:88/fkgdsRr+8H+x6qB7XEBW3Z+Q= 2023-05-24T21:27:06.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:27:06 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:88/fkgdsRr+8H+x6qB7XEBW3Z+Q='} 2023-05-24T21:27:06.392 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004b9d0ac5b7e981be-00646e812a-1627-a1'), ('Date', 'Wed, 24 May 2023 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:06.393 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:27:06.394 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:27:06.394 DEBUG:boto:Method: HEAD 2023-05-24T21:27:06.394 DEBUG:boto:Path: /qskrps-48/ 2023-05-24T21:27:06.395 DEBUG:boto:Data: 2023-05-24T21:27:06.395 DEBUG:boto:Headers: {} 2023-05-24T21:27:06.395 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:06.396 DEBUG:boto:Port: 8000 2023-05-24T21:27:06.396 DEBUG:boto:Params: {} 2023-05-24T21:27:06.396 DEBUG:boto:Token: None 2023-05-24T21:27:06.397 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:06 GMT /qskrps-48/ 2023-05-24T21:27:06.397 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:V5A8nIK0Jgzr6CqeXTN5Ixbqflc= 2023-05-24T21:27:06.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:06 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:V5A8nIK0Jgzr6CqeXTN5Ixbqflc=', 'Content-Length': '0'} 2023-05-24T21:27:06.400 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000db8c5f4e4b5c29e1-00646e812a-1627-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:06.400 DEBUG:boto:path=/qskrps-48/obj4 2023-05-24T21:27:06.401 DEBUG:boto:auth_path=/qskrps-48/obj4 2023-05-24T21:27:06.401 DEBUG:boto:Method: PUT 2023-05-24T21:27:06.401 DEBUG:boto:Path: /qskrps-48/obj4 2023-05-24T21:27:06.402 DEBUG:boto:Data: 2023-05-24T21:27:06.402 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-24T21:27:06.402 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:06.403 DEBUG:boto:Port: 8000 2023-05-24T21:27:06.403 DEBUG:boto:Params: {} 2023-05-24T21:27:06.403 DEBUG:boto:Token: None 2023-05-24T21:27:06.404 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 24 May 2023 21:27:06 GMT /qskrps-48/obj4 2023-05-24T21:27:06.404 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:iWTR1p3Vn+MHu4/+9ngaTqSUOHY= 2023-05-24T21:27:06.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:27:06 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:iWTR1p3Vn+MHu4/+9ngaTqSUOHY='} 2023-05-24T21:27:06.410 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c3cb0168fa6a7cac-00646e812a-1627-a1'), ('Date', 'Wed, 24 May 2023 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:06.411 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:27:06.411 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:27:06.412 DEBUG:boto:Method: HEAD 2023-05-24T21:27:06.412 DEBUG:boto:Path: /qskrps-49/ 2023-05-24T21:27:06.412 DEBUG:boto:Data: 2023-05-24T21:27:06.413 DEBUG:boto:Headers: {} 2023-05-24T21:27:06.413 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:06.413 DEBUG:boto:Port: 8001 2023-05-24T21:27:06.414 DEBUG:boto:Params: {} 2023-05-24T21:27:06.414 DEBUG:boto:Token: None 2023-05-24T21:27:06.414 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:06 GMT /qskrps-49/ 2023-05-24T21:27:06.415 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:M5Ht46Haz/rqlYXANpRLYN5U11w= 2023-05-24T21:27:06.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:06 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:M5Ht46Haz/rqlYXANpRLYN5U11w=', 'Content-Length': '0'} 2023-05-24T21:27:06.418 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003aba7ab166b10b58-00646e812a-1625-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:06.418 DEBUG:boto:path=/qskrps-49/obj1 2023-05-24T21:27:06.419 DEBUG:boto:auth_path=/qskrps-49/obj1 2023-05-24T21:27:06.419 DEBUG:boto:Method: PUT 2023-05-24T21:27:06.419 DEBUG:boto:Path: /qskrps-49/obj1 2023-05-24T21:27:06.420 DEBUG:boto:Data: 2023-05-24T21:27:06.420 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-24T21:27:06.420 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:06.421 DEBUG:boto:Port: 8001 2023-05-24T21:27:06.421 DEBUG:boto:Params: {} 2023-05-24T21:27:06.421 DEBUG:boto:Token: None 2023-05-24T21:27:06.422 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 24 May 2023 21:27:06 GMT /qskrps-49/obj1 2023-05-24T21:27:06.422 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:/br2n9r6qVI4xnmgRP1CCiCLhAg= 2023-05-24T21:27:06.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:27:06 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:/br2n9r6qVI4xnmgRP1CCiCLhAg='} 2023-05-24T21:27:06.429 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c3b746195bdc6777-00646e812a-1625-a2'), ('Date', 'Wed, 24 May 2023 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:06.430 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:27:06.430 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:27:06.431 DEBUG:boto:Method: HEAD 2023-05-24T21:27:06.431 DEBUG:boto:Path: /qskrps-49/ 2023-05-24T21:27:06.431 DEBUG:boto:Data: 2023-05-24T21:27:06.432 DEBUG:boto:Headers: {} 2023-05-24T21:27:06.432 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:06.432 DEBUG:boto:Port: 8001 2023-05-24T21:27:06.433 DEBUG:boto:Params: {} 2023-05-24T21:27:06.433 DEBUG:boto:Token: None 2023-05-24T21:27:06.433 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:06 GMT /qskrps-49/ 2023-05-24T21:27:06.434 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:M5Ht46Haz/rqlYXANpRLYN5U11w= 2023-05-24T21:27:06.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:06 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:M5Ht46Haz/rqlYXANpRLYN5U11w=', 'Content-Length': '0'} 2023-05-24T21:27:06.436 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ef8d0874ea1c18ac-00646e812a-1625-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:06.436 DEBUG:boto:path=/qskrps-49/obj2 2023-05-24T21:27:06.437 DEBUG:boto:auth_path=/qskrps-49/obj2 2023-05-24T21:27:06.437 DEBUG:boto:Method: PUT 2023-05-24T21:27:06.438 DEBUG:boto:Path: /qskrps-49/obj2 2023-05-24T21:27:06.438 DEBUG:boto:Data: 2023-05-24T21:27:06.438 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-24T21:27:06.439 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:06.439 DEBUG:boto:Port: 8001 2023-05-24T21:27:06.439 DEBUG:boto:Params: {} 2023-05-24T21:27:06.440 DEBUG:boto:Token: None 2023-05-24T21:27:06.440 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 24 May 2023 21:27:06 GMT /qskrps-49/obj2 2023-05-24T21:27:06.440 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:pTPZe7tEiWi3QXRiKR+pK8tzB3U= 2023-05-24T21:27:06.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:27:06 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:pTPZe7tEiWi3QXRiKR+pK8tzB3U='} 2023-05-24T21:27:06.484 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005a570840a663090d-00646e812a-1625-a2'), ('Date', 'Wed, 24 May 2023 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:06.485 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:27:06.486 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:27:06.486 DEBUG:boto:Method: HEAD 2023-05-24T21:27:06.487 DEBUG:boto:Path: /qskrps-49/ 2023-05-24T21:27:06.487 DEBUG:boto:Data: 2023-05-24T21:27:06.487 DEBUG:boto:Headers: {} 2023-05-24T21:27:06.488 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:06.488 DEBUG:boto:Port: 8001 2023-05-24T21:27:06.489 DEBUG:boto:Params: {} 2023-05-24T21:27:06.489 DEBUG:boto:Token: None 2023-05-24T21:27:06.490 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:06 GMT /qskrps-49/ 2023-05-24T21:27:06.490 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:M5Ht46Haz/rqlYXANpRLYN5U11w= 2023-05-24T21:27:06.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:06 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:M5Ht46Haz/rqlYXANpRLYN5U11w=', 'Content-Length': '0'} 2023-05-24T21:27:06.493 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000014594a1f92bde220-00646e812a-1625-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:06.494 DEBUG:boto:path=/qskrps-49/obj3 2023-05-24T21:27:06.494 DEBUG:boto:auth_path=/qskrps-49/obj3 2023-05-24T21:27:06.495 DEBUG:boto:Method: PUT 2023-05-24T21:27:06.495 DEBUG:boto:Path: /qskrps-49/obj3 2023-05-24T21:27:06.495 DEBUG:boto:Data: 2023-05-24T21:27:06.496 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-24T21:27:06.496 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:06.496 DEBUG:boto:Port: 8001 2023-05-24T21:27:06.497 DEBUG:boto:Params: {} 2023-05-24T21:27:06.497 DEBUG:boto:Token: None 2023-05-24T21:27:06.497 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 24 May 2023 21:27:06 GMT /qskrps-49/obj3 2023-05-24T21:27:06.498 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:weUuLA2ePIsyE5+ysB/QzXzSiQM= 2023-05-24T21:27:06.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:27:06 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:weUuLA2ePIsyE5+ysB/QzXzSiQM='} 2023-05-24T21:27:06.504 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000001f09b7b2f1d2631-00646e812a-1625-a2'), ('Date', 'Wed, 24 May 2023 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:06.505 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:27:06.505 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:27:06.506 DEBUG:boto:Method: HEAD 2023-05-24T21:27:06.506 DEBUG:boto:Path: /qskrps-49/ 2023-05-24T21:27:06.507 DEBUG:boto:Data: 2023-05-24T21:27:06.507 DEBUG:boto:Headers: {} 2023-05-24T21:27:06.507 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:06.508 DEBUG:boto:Port: 8001 2023-05-24T21:27:06.508 DEBUG:boto:Params: {} 2023-05-24T21:27:06.509 DEBUG:boto:Token: None 2023-05-24T21:27:06.509 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:06 GMT /qskrps-49/ 2023-05-24T21:27:06.510 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:M5Ht46Haz/rqlYXANpRLYN5U11w= 2023-05-24T21:27:06.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:06 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:M5Ht46Haz/rqlYXANpRLYN5U11w=', 'Content-Length': '0'} 2023-05-24T21:27:06.512 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cdf20c7e645fbbcd-00646e812a-1625-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:06.513 DEBUG:boto:path=/qskrps-49/obj4 2023-05-24T21:27:06.513 DEBUG:boto:auth_path=/qskrps-49/obj4 2023-05-24T21:27:06.513 DEBUG:boto:Method: PUT 2023-05-24T21:27:06.514 DEBUG:boto:Path: /qskrps-49/obj4 2023-05-24T21:27:06.514 DEBUG:boto:Data: 2023-05-24T21:27:06.514 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-24T21:27:06.515 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:06.515 DEBUG:boto:Port: 8001 2023-05-24T21:27:06.516 DEBUG:boto:Params: {} 2023-05-24T21:27:06.516 DEBUG:boto:Token: None 2023-05-24T21:27:06.517 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 24 May 2023 21:27:06 GMT /qskrps-49/obj4 2023-05-24T21:27:06.517 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Gq61E+ibxoZAr0ouDCdU9z5NuTQ= 2023-05-24T21:27:06.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:27:06 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Gq61E+ibxoZAr0ouDCdU9z5NuTQ='} 2023-05-24T21:27:06.560 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000085ac2b12bdd13f80-00646e812a-1625-a2'), ('Date', 'Wed, 24 May 2023 21:27:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:06.561 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:27:06.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:27:06.562 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:27:06.611 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:27:06.611 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:27:06.654 INFO:teuthology.orchestra.run.smithi071.stderr:No --period given, using current period=5c04936b-70ea-4415-b4eb-3aad3dc612e5 2023-05-24T21:27:06.670 INFO:teuthology.orchestra.run.smithi071.stdout:[ 2023-05-24T21:27:06.671 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:27:06.671 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963626.276651_526.1", 2023-05-24T21:27:06.671 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:27:06.276651Z" 2023-05-24T21:27:06.671 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:27:06.671 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:27:06.671 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:27:06.672 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:06:32.333844Z" 2023-05-24T21:27:06.672 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:27:06.672 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:27:06.672 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:27:06.672 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:27:06.672 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:27:06.672 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:27:06.673 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:27:06.673 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:27:06.673 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:27:06.673 INFO:teuthology.orchestra.run.smithi071.stdout:] 2023-05-24T21:27:06.674 DEBUG:rgw_multi.tests:master meta markers={0: '1_1684963626.276651_526.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:27:06.674 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-24T21:27:06.675 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:27:06.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:27:06.676 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:27:06.724 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:27:06.724 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:27:06.768 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:27:06.762+0000 7f552e794780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:27:06.769 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:27:06.763+0000 7f552e794780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:27:06.785 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:27:06.786 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:27:06.786 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:27:06.786 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:27:06.786 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:27:06.787 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:27:06.787 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:27:06.787 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:27:06.787 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:27:06.787 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:27:06.787 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:27:06.787 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:27:06.788 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:27:06.788 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963582.036316_514.1", 2023-05-24T21:27:06.788 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:27:06.788 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:27:06.788 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:27:06.788 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:26:22.036316Z", 2023-05-24T21:27:06.789 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:27:06.789 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:27:06.789 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:27:06.789 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:27:06.789 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:27:06.789 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:27:06.790 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:27:06.790 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:27:06.790 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:27:06.790 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:27:06.790 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:27:06.790 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:27:06.790 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:27:06.791 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:27:06.791 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:27:06.791 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:27:06.791 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:27:06.791 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:27:06.791 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:27:06.791 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:27:06.792 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:27:06.792 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:27:06.792 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:27:06.792 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:27:06.792 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:27:06.792 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:27:06.792 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:27:06.793 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:27:06.793 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:27:06.793 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:27:06.793 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:27:06.793 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:27:06.793 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:27:06.793 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:27:06.794 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:27:06.794 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:27:06.794 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:27:06.794 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:27:06.794 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:27:06.795 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:27:06.795 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:27:06.795 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:27:06.795 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:27:06.795 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:27:06.795 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:27:06.795 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:27:06Z" 2023-05-24T21:27:06.796 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:27:06.797 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684963582.036316_514.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:26:22.036316Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:27:06Z" } 2023-05-24T21:27:06.797 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963582.036316_514.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:26:22.036316Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:27:06.798 DEBUG:rgw_multi.tests:log_status={0: '1_1684963626.276651_526.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:27:06.798 DEBUG:rgw_multi.tests:sync_status={0: '1_1684963582.036316_514.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:27:06.798 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963626.276651_526.1 target=1_1684963582.036316_514.1 2023-05-24T21:27:11.800 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:27:11.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:27:11.801 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:27:11.858 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:27:11.859 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:27:11.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:27:11.891+0000 7f087c35b780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:27:11.898 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:27:11.892+0000 7f087c35b780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:27:11.913 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:27:11.913 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:27:11.913 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:27:11.914 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:27:11.914 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:27:11.914 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:27:11.914 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:27:11.914 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:27:11.915 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:27:11.915 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:27:11.915 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:27:11.915 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:27:11.915 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:27:11.915 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963626.276651_526.1", 2023-05-24T21:27:11.916 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:27:11.916 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:27:11.916 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:27:11.916 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:27:06.276651Z", 2023-05-24T21:27:11.916 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:27:11.917 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:27:11.917 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:27:11.917 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:27:11.917 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:27:11.917 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:27:11.917 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:27:11.917 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:27:11.918 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:27:11.918 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:27:11.918 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:27:11.918 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:27:11.918 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:27:11.918 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:27:11.918 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:27:11.919 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:27:11.919 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:27:11.919 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:27:11.919 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:27:11.919 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:27:11.919 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:27:11.919 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:27:11.920 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:27:11.920 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:27:11.920 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:27:11.920 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:27:11.920 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:27:11.921 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:27:11.921 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:27:11.921 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:27:11.921 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:27:11.921 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:27:11.921 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:27:11.921 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:27:11.922 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:27:11.922 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:27:11.922 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:27:11.922 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:27:11.922 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:27:11.922 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:27:11.922 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:27:11.923 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:27:11.923 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:27:11.923 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:27:11.923 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:27:11.923 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:27:11Z" 2023-05-24T21:27:11.923 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:27:11.924 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684963626.276651_526.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:27:06.276651Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:27:11Z" } 2023-05-24T21:27:11.925 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963626.276651_526.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:27:06.276651Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:27:11.925 DEBUG:rgw_multi.tests:log_status={0: '1_1684963626.276651_526.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:27:11.925 DEBUG:rgw_multi.tests:sync_status={0: '1_1684963626.276651_526.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:27:11.926 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-24T21:27:11.926 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=qskrps-48 2023-05-24T21:27:11.926 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:27:11.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:27:11.927 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-48 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:27:11.984 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:27:11.984 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:27:12.029 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:27:12.024+0000 7f7772c3b780 1 bucket sync caught up with source: 2023-05-24T21:27:12.029 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [00000000001.18.6, , 00000000001.19.6, , 00000000001.30.6, , , 00000000001.20.6, , , ] 2023-05-24T21:27:12.029 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [00000000001.18.6, , 00000000001.19.6, , 00000000001.30.6, , , 00000000001.20.6, , , ] 2023-05-24T21:27:12.030 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:27:12.024+0000 7f7772c3b780 0 bucket checkpoint complete 2023-05-24T21:27:12.044 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=qskrps-48 2023-05-24T21:27:12.045 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:27:12.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:27:12.045 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-48 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:27:12.101 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:27:12.102 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:27:12.145 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:27:12.139+0000 7f897596d780 1 bucket sync caught up with source: 2023-05-24T21:27:12.145 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [00000000001.661.6, , 00000000001.343.6, , 00000000001.399.6, , , 00000000001.830.6, , , ] 2023-05-24T21:27:12.145 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [00000000001.661.6, , 00000000001.343.6, , 00000000001.399.6, , , 00000000001.830.6, , , ] 2023-05-24T21:27:12.145 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:27:12.139+0000 7f897596d780 0 bucket checkpoint complete 2023-05-24T21:27:12.161 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qskrps-48 zones={a2, a1} 2023-05-24T21:27:12.161 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:27:12.161 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:27:12.162 DEBUG:boto:Method: HEAD 2023-05-24T21:27:12.162 DEBUG:boto:Path: /qskrps-48/ 2023-05-24T21:27:12.162 DEBUG:boto:Data: 2023-05-24T21:27:12.163 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.163 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:12.163 DEBUG:boto:Port: 8001 2023-05-24T21:27:12.164 DEBUG:boto:Params: {} 2023-05-24T21:27:12.164 DEBUG:boto:Token: None 2023-05-24T21:27:12.165 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:12 GMT /qskrps-48/ 2023-05-24T21:27:12.165 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:blJKXeMR7pFZtE2SZtCm7AIDv4A= 2023-05-24T21:27:12.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:blJKXeMR7pFZtE2SZtCm7AIDv4A=', 'Content-Length': '0'} 2023-05-24T21:27:12.168 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006d1636b4666be5d0-00646e8130-1625-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.169 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:27:12.169 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:27:12.169 DEBUG:boto:Method: HEAD 2023-05-24T21:27:12.170 DEBUG:boto:Path: /qskrps-48/ 2023-05-24T21:27:12.170 DEBUG:boto:Data: 2023-05-24T21:27:12.170 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.171 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:12.171 DEBUG:boto:Port: 8000 2023-05-24T21:27:12.171 DEBUG:boto:Params: {} 2023-05-24T21:27:12.172 DEBUG:boto:Token: None 2023-05-24T21:27:12.172 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:12 GMT /qskrps-48/ 2023-05-24T21:27:12.173 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:blJKXeMR7pFZtE2SZtCm7AIDv4A= 2023-05-24T21:27:12.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:blJKXeMR7pFZtE2SZtCm7AIDv4A=', 'Content-Length': '0'} 2023-05-24T21:27:12.175 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c038d37fa4aab703-00646e8130-1627-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.175 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-24T21:27:12.176 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:27:12.176 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:27:12.177 DEBUG:boto:path=/qskrps-48/?versions&max-keys=999 2023-05-24T21:27:12.177 DEBUG:boto:auth_path=/qskrps-48/?versions&max-keys=999 2023-05-24T21:27:12.178 DEBUG:boto:Method: GET 2023-05-24T21:27:12.178 DEBUG:boto:Path: /qskrps-48/?versions&max-keys=999 2023-05-24T21:27:12.178 DEBUG:boto:Data: 2023-05-24T21:27:12.179 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.179 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:12.179 DEBUG:boto:Port: 8001 2023-05-24T21:27:12.180 DEBUG:boto:Params: {} 2023-05-24T21:27:12.180 DEBUG:boto:Token: None 2023-05-24T21:27:12.180 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:12 GMT /qskrps-48/?versions 2023-05-24T21:27:12.181 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:LSes17wod/k4akdDkksUoPiVQ/U= 2023-05-24T21:27:12.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:LSes17wod/k4akdDkksUoPiVQ/U=', 'Content-Length': '0'} 2023-05-24T21:27:12.183 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ab1bfadac77cf54-00646e8130-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.184 DEBUG:boto:b'qskrps-48999falseobj1nulltrue2023-05-24T21:27:06.310Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-24T21:27:06.330Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-24T21:27:06.348Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-24T21:27:06.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:12.184 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-05-24T21:27:12.185 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-05-24T21:27:12.185 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-05-24T21:27:12.185 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-05-24T21:27:12.186 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-24T21:27:12.186 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:27:12.186 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:27:12.187 DEBUG:boto:path=/qskrps-48/?versions&max-keys=999 2023-05-24T21:27:12.187 DEBUG:boto:auth_path=/qskrps-48/?versions&max-keys=999 2023-05-24T21:27:12.187 DEBUG:boto:Method: GET 2023-05-24T21:27:12.188 DEBUG:boto:Path: /qskrps-48/?versions&max-keys=999 2023-05-24T21:27:12.188 DEBUG:boto:Data: 2023-05-24T21:27:12.188 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.189 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:12.189 DEBUG:boto:Port: 8000 2023-05-24T21:27:12.189 DEBUG:boto:Params: {} 2023-05-24T21:27:12.190 DEBUG:boto:Token: None 2023-05-24T21:27:12.190 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:12 GMT /qskrps-48/?versions 2023-05-24T21:27:12.191 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:LSes17wod/k4akdDkksUoPiVQ/U= 2023-05-24T21:27:12.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:LSes17wod/k4akdDkksUoPiVQ/U=', 'Content-Length': '0'} 2023-05-24T21:27:12.193 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0492e5de7e549c7-00646e8130-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.194 DEBUG:boto:b'qskrps-48999falseobj1nulltrue2023-05-24T21:27:06.310Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-24T21:27:06.330Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-24T21:27:06.348Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-24T21:27:06.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:12.194 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-05-24T21:27:12.195 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-05-24T21:27:12.195 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-05-24T21:27:12.195 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-05-24T21:27:12.196 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:27:12.196 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:27:12.197 DEBUG:boto:path=/qskrps-48/?versions&max-keys=999 2023-05-24T21:27:12.197 DEBUG:boto:auth_path=/qskrps-48/?versions&max-keys=999 2023-05-24T21:27:12.197 DEBUG:boto:Method: GET 2023-05-24T21:27:12.198 DEBUG:boto:Path: /qskrps-48/?versions&max-keys=999 2023-05-24T21:27:12.198 DEBUG:boto:Data: 2023-05-24T21:27:12.199 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.199 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:12.199 DEBUG:boto:Port: 8001 2023-05-24T21:27:12.200 DEBUG:boto:Params: {} 2023-05-24T21:27:12.200 DEBUG:boto:Token: None 2023-05-24T21:27:12.200 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:12 GMT /qskrps-48/?versions 2023-05-24T21:27:12.201 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:LSes17wod/k4akdDkksUoPiVQ/U= 2023-05-24T21:27:12.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:LSes17wod/k4akdDkksUoPiVQ/U=', 'Content-Length': '0'} 2023-05-24T21:27:12.203 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a2cc01cf73ebbe1-00646e8130-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.204 DEBUG:boto:b'qskrps-48999falseobj1nulltrue2023-05-24T21:27:06.310Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-24T21:27:06.330Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-24T21:27:06.348Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-24T21:27:06.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:12.204 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:27:12.205 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:27:12.205 DEBUG:boto:path=/qskrps-48/?versions&max-keys=999 2023-05-24T21:27:12.205 DEBUG:boto:auth_path=/qskrps-48/?versions&max-keys=999 2023-05-24T21:27:12.206 DEBUG:boto:Method: GET 2023-05-24T21:27:12.206 DEBUG:boto:Path: /qskrps-48/?versions&max-keys=999 2023-05-24T21:27:12.206 DEBUG:boto:Data: 2023-05-24T21:27:12.207 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.207 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:12.207 DEBUG:boto:Port: 8000 2023-05-24T21:27:12.208 DEBUG:boto:Params: {} 2023-05-24T21:27:12.208 DEBUG:boto:Token: None 2023-05-24T21:27:12.209 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:12 GMT /qskrps-48/?versions 2023-05-24T21:27:12.209 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:LSes17wod/k4akdDkksUoPiVQ/U= 2023-05-24T21:27:12.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:LSes17wod/k4akdDkksUoPiVQ/U=', 'Content-Length': '0'} 2023-05-24T21:27:12.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ed58ce290d2f1d6-00646e8130-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.212 DEBUG:boto:b'qskrps-48999falseobj1nulltrue2023-05-24T21:27:06.310Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-24T21:27:06.330Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-24T21:27:06.348Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-24T21:27:06.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:12.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-24T21:27:12.213 DEBUG:boto:path=/qskrps-48/obj1 2023-05-24T21:27:12.213 DEBUG:boto:auth_path=/qskrps-48/obj1 2023-05-24T21:27:12.214 DEBUG:boto:path=/qskrps-48/obj1?versionId=null 2023-05-24T21:27:12.214 DEBUG:boto:auth_path=/qskrps-48/obj1?versionId=null 2023-05-24T21:27:12.214 DEBUG:boto:Method: GET 2023-05-24T21:27:12.215 DEBUG:boto:Path: /qskrps-48/obj1?versionId=null 2023-05-24T21:27:12.215 DEBUG:boto:Data: 2023-05-24T21:27:12.215 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.216 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:12.216 DEBUG:boto:Port: 8001 2023-05-24T21:27:12.216 DEBUG:boto:Params: {} 2023-05-24T21:27:12.217 DEBUG:boto:Token: None 2023-05-24T21:27:12.217 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:12 GMT /qskrps-48/obj1?versionId=null 2023-05-24T21:27:12.218 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:zOA0v8gu8NBOiN0GF/xqlLUfjlQ= 2023-05-24T21:27:12.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:zOA0v8gu8NBOiN0GF/xqlLUfjlQ=', 'Content-Length': '0'} 2023-05-24T21:27:12.220 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cd190d9d1b38a93f-00646e8130-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.220 DEBUG:boto:path=/qskrps-48/obj1 2023-05-24T21:27:12.221 DEBUG:boto:auth_path=/qskrps-48/obj1 2023-05-24T21:27:12.221 DEBUG:boto:path=/qskrps-48/obj1?versionId=null 2023-05-24T21:27:12.221 DEBUG:boto:auth_path=/qskrps-48/obj1?versionId=null 2023-05-24T21:27:12.222 DEBUG:boto:Method: GET 2023-05-24T21:27:12.222 DEBUG:boto:Path: /qskrps-48/obj1?versionId=null 2023-05-24T21:27:12.223 DEBUG:boto:Data: 2023-05-24T21:27:12.223 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.223 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:12.224 DEBUG:boto:Port: 8000 2023-05-24T21:27:12.224 DEBUG:boto:Params: {} 2023-05-24T21:27:12.224 DEBUG:boto:Token: None 2023-05-24T21:27:12.225 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:12 GMT /qskrps-48/obj1?versionId=null 2023-05-24T21:27:12.225 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:zOA0v8gu8NBOiN0GF/xqlLUfjlQ= 2023-05-24T21:27:12.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:zOA0v8gu8NBOiN0GF/xqlLUfjlQ=', 'Content-Length': '0'} 2023-05-24T21:27:12.227 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c77b5572c663a7b3-00646e8130-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.228 DEBUG:boto:path=/qskrps-48/obj1 2023-05-24T21:27:12.228 DEBUG:boto:auth_path=/qskrps-48/obj1 2023-05-24T21:27:12.228 DEBUG:boto:path=/qskrps-48/obj1?versionId=null 2023-05-24T21:27:12.229 DEBUG:boto:auth_path=/qskrps-48/obj1?versionId=null 2023-05-24T21:27:12.229 DEBUG:boto:Method: HEAD 2023-05-24T21:27:12.229 DEBUG:boto:Path: /qskrps-48/obj1?versionId=null 2023-05-24T21:27:12.230 DEBUG:boto:Data: 2023-05-24T21:27:12.230 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.230 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:12.231 DEBUG:boto:Port: 8001 2023-05-24T21:27:12.231 DEBUG:boto:Params: {} 2023-05-24T21:27:12.232 DEBUG:boto:Token: None 2023-05-24T21:27:12.232 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:12 GMT /qskrps-48/obj1?versionId=null 2023-05-24T21:27:12.232 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:RRRE1LLAo78563iLg87sVVUwpaM= 2023-05-24T21:27:12.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:RRRE1LLAo78563iLg87sVVUwpaM=', 'Content-Length': '0'} 2023-05-24T21:27:12.234 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000041c04418e46d829e-00646e8130-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.235 DEBUG:boto:path=/qskrps-48/obj1 2023-05-24T21:27:12.235 DEBUG:boto:auth_path=/qskrps-48/obj1 2023-05-24T21:27:12.235 DEBUG:boto:path=/qskrps-48/obj1?versionId=null 2023-05-24T21:27:12.236 DEBUG:boto:auth_path=/qskrps-48/obj1?versionId=null 2023-05-24T21:27:12.236 DEBUG:boto:Method: HEAD 2023-05-24T21:27:12.236 DEBUG:boto:Path: /qskrps-48/obj1?versionId=null 2023-05-24T21:27:12.237 DEBUG:boto:Data: 2023-05-24T21:27:12.237 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.237 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:12.238 DEBUG:boto:Port: 8000 2023-05-24T21:27:12.238 DEBUG:boto:Params: {} 2023-05-24T21:27:12.238 DEBUG:boto:Token: None 2023-05-24T21:27:12.239 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:12 GMT /qskrps-48/obj1?versionId=null 2023-05-24T21:27:12.239 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:RRRE1LLAo78563iLg87sVVUwpaM= 2023-05-24T21:27:12.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:RRRE1LLAo78563iLg87sVVUwpaM=', 'Content-Length': '0'} 2023-05-24T21:27:12.241 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000631b7292b191c3bf-00646e8130-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-24T21:27:12.242 DEBUG:boto:path=/qskrps-48/obj1 2023-05-24T21:27:12.242 DEBUG:boto:auth_path=/qskrps-48/obj1 2023-05-24T21:27:12.243 DEBUG:boto:path=/qskrps-48/obj1?versionId=null 2023-05-24T21:27:12.243 DEBUG:boto:auth_path=/qskrps-48/obj1?versionId=null 2023-05-24T21:27:12.243 DEBUG:boto:Method: GET 2023-05-24T21:27:12.244 DEBUG:boto:Path: /qskrps-48/obj1?versionId=null 2023-05-24T21:27:12.244 DEBUG:boto:Data: 2023-05-24T21:27:12.244 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.245 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:12.245 DEBUG:boto:Port: 8001 2023-05-24T21:27:12.246 DEBUG:boto:Params: {} 2023-05-24T21:27:12.246 DEBUG:boto:Token: None 2023-05-24T21:27:12.246 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:12 GMT /qskrps-48/obj1?versionId=null 2023-05-24T21:27:12.247 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:zOA0v8gu8NBOiN0GF/xqlLUfjlQ= 2023-05-24T21:27:12.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:zOA0v8gu8NBOiN0GF/xqlLUfjlQ=', 'Content-Length': '0'} 2023-05-24T21:27:12.249 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000601ac7b2fbefbbea-00646e8130-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.249 DEBUG:boto:path=/qskrps-48/obj1 2023-05-24T21:27:12.250 DEBUG:boto:auth_path=/qskrps-48/obj1 2023-05-24T21:27:12.250 DEBUG:boto:path=/qskrps-48/obj1?versionId=null 2023-05-24T21:27:12.251 DEBUG:boto:auth_path=/qskrps-48/obj1?versionId=null 2023-05-24T21:27:12.251 DEBUG:boto:Method: GET 2023-05-24T21:27:12.251 DEBUG:boto:Path: /qskrps-48/obj1?versionId=null 2023-05-24T21:27:12.252 DEBUG:boto:Data: 2023-05-24T21:27:12.252 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.252 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:12.253 DEBUG:boto:Port: 8000 2023-05-24T21:27:12.253 DEBUG:boto:Params: {} 2023-05-24T21:27:12.254 DEBUG:boto:Token: None 2023-05-24T21:27:12.254 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:12 GMT /qskrps-48/obj1?versionId=null 2023-05-24T21:27:12.254 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:zOA0v8gu8NBOiN0GF/xqlLUfjlQ= 2023-05-24T21:27:12.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:zOA0v8gu8NBOiN0GF/xqlLUfjlQ=', 'Content-Length': '0'} 2023-05-24T21:27:12.257 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ceda7cd9190e8664-00646e8130-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.257 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:27:12.258 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:27:12.258 DEBUG:boto:path=/qskrps-48/?prefix=obj1 2023-05-24T21:27:12.258 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj1 2023-05-24T21:27:12.259 DEBUG:boto:Method: GET 2023-05-24T21:27:12.259 DEBUG:boto:Path: /qskrps-48/?prefix=obj1 2023-05-24T21:27:12.259 DEBUG:boto:Data: 2023-05-24T21:27:12.260 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.260 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:12.260 DEBUG:boto:Port: 8001 2023-05-24T21:27:12.261 DEBUG:boto:Params: {} 2023-05-24T21:27:12.261 DEBUG:boto:Token: None 2023-05-24T21:27:12.262 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:12 GMT /qskrps-48/ 2023-05-24T21:27:12.262 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:8TAMkXi8EVUV3ltARiym9EgqjRs= 2023-05-24T21:27:12.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:8TAMkXi8EVUV3ltARiym9EgqjRs=', 'Content-Length': '0'} 2023-05-24T21:27:12.264 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000624a2e605f3106f-00646e8130-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.265 DEBUG:boto:b'qskrps-48obj11000falseobj12023-05-24T21:27:06.310Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:12.266 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:27:12.266 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:27:12.266 DEBUG:boto:path=/qskrps-48/?prefix=obj1 2023-05-24T21:27:12.267 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj1 2023-05-24T21:27:12.267 DEBUG:boto:Method: GET 2023-05-24T21:27:12.267 DEBUG:boto:Path: /qskrps-48/?prefix=obj1 2023-05-24T21:27:12.268 DEBUG:boto:Data: 2023-05-24T21:27:12.268 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.268 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:12.269 DEBUG:boto:Port: 8000 2023-05-24T21:27:12.269 DEBUG:boto:Params: {} 2023-05-24T21:27:12.270 DEBUG:boto:Token: None 2023-05-24T21:27:12.270 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:12 GMT /qskrps-48/ 2023-05-24T21:27:12.270 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:8TAMkXi8EVUV3ltARiym9EgqjRs= 2023-05-24T21:27:12.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:8TAMkXi8EVUV3ltARiym9EgqjRs=', 'Content-Length': '0'} 2023-05-24T21:27:12.273 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000436e14d4527bcd0a-00646e8130-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.316 DEBUG:boto:b'qskrps-48obj11000falseobj12023-05-24T21:27:06.310Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:12.317 DEBUG:boto:path=/qskrps-48/obj1 2023-05-24T21:27:12.317 DEBUG:boto:auth_path=/qskrps-48/obj1 2023-05-24T21:27:12.318 DEBUG:boto:Method: HEAD 2023-05-24T21:27:12.318 DEBUG:boto:Path: /qskrps-48/obj1 2023-05-24T21:27:12.318 DEBUG:boto:Data: 2023-05-24T21:27:12.319 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.319 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:12.319 DEBUG:boto:Port: 8001 2023-05-24T21:27:12.320 DEBUG:boto:Params: {} 2023-05-24T21:27:12.320 DEBUG:boto:Token: None 2023-05-24T21:27:12.320 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:12 GMT /qskrps-48/obj1 2023-05-24T21:27:12.321 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5aOdrCsTPzqnRU0F+jqvXkV+VMM= 2023-05-24T21:27:12.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5aOdrCsTPzqnRU0F+jqvXkV+VMM=', 'Content-Length': '0'} 2023-05-24T21:27:12.323 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000571a0d8382739273-00646e8130-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.323 DEBUG:boto:path=/qskrps-48/obj1 2023-05-24T21:27:12.324 DEBUG:boto:auth_path=/qskrps-48/obj1 2023-05-24T21:27:12.324 DEBUG:boto:Method: HEAD 2023-05-24T21:27:12.324 DEBUG:boto:Path: /qskrps-48/obj1 2023-05-24T21:27:12.325 DEBUG:boto:Data: 2023-05-24T21:27:12.325 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.325 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:12.326 DEBUG:boto:Port: 8000 2023-05-24T21:27:12.326 DEBUG:boto:Params: {} 2023-05-24T21:27:12.326 DEBUG:boto:Token: None 2023-05-24T21:27:12.327 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:12 GMT /qskrps-48/obj1 2023-05-24T21:27:12.327 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5aOdrCsTPzqnRU0F+jqvXkV+VMM= 2023-05-24T21:27:12.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5aOdrCsTPzqnRU0F+jqvXkV+VMM=', 'Content-Length': '0'} 2023-05-24T21:27:12.329 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c1c859bf89da471d-00646e8130-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-24T21:27:12.330 DEBUG:boto:path=/qskrps-48/obj1 2023-05-24T21:27:12.330 DEBUG:boto:auth_path=/qskrps-48/obj1 2023-05-24T21:27:12.331 DEBUG:boto:Method: GET 2023-05-24T21:27:12.331 DEBUG:boto:Path: /qskrps-48/obj1 2023-05-24T21:27:12.332 DEBUG:boto:Data: 2023-05-24T21:27:12.332 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.332 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:12.333 DEBUG:boto:Port: 8001 2023-05-24T21:27:12.333 DEBUG:boto:Params: {} 2023-05-24T21:27:12.333 DEBUG:boto:Token: None 2023-05-24T21:27:12.334 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:12 GMT /qskrps-48/obj1 2023-05-24T21:27:12.334 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:EcY6kyF8FXGlRe7jQ0mp0T7tc0k= 2023-05-24T21:27:12.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:EcY6kyF8FXGlRe7jQ0mp0T7tc0k=', 'Content-Length': '0'} 2023-05-24T21:27:12.336 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009bbd4208dd821ce4-00646e8130-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.337 DEBUG:boto:path=/qskrps-48/obj1 2023-05-24T21:27:12.337 DEBUG:boto:auth_path=/qskrps-48/obj1 2023-05-24T21:27:12.338 DEBUG:boto:Method: GET 2023-05-24T21:27:12.338 DEBUG:boto:Path: /qskrps-48/obj1 2023-05-24T21:27:12.338 DEBUG:boto:Data: 2023-05-24T21:27:12.339 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.339 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:12.339 DEBUG:boto:Port: 8000 2023-05-24T21:27:12.340 DEBUG:boto:Params: {} 2023-05-24T21:27:12.340 DEBUG:boto:Token: None 2023-05-24T21:27:12.341 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:12 GMT /qskrps-48/obj1 2023-05-24T21:27:12.341 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:EcY6kyF8FXGlRe7jQ0mp0T7tc0k= 2023-05-24T21:27:12.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:EcY6kyF8FXGlRe7jQ0mp0T7tc0k=', 'Content-Length': '0'} 2023-05-24T21:27:12.343 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005cff63d691eb4723-00646e8130-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.344 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:27:12.344 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:27:12.344 DEBUG:boto:path=/qskrps-48/?prefix=obj1 2023-05-24T21:27:12.345 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj1 2023-05-24T21:27:12.345 DEBUG:boto:Method: GET 2023-05-24T21:27:12.346 DEBUG:boto:Path: /qskrps-48/?prefix=obj1 2023-05-24T21:27:12.346 DEBUG:boto:Data: 2023-05-24T21:27:12.346 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.347 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:12.347 DEBUG:boto:Port: 8001 2023-05-24T21:27:12.347 DEBUG:boto:Params: {} 2023-05-24T21:27:12.348 DEBUG:boto:Token: None 2023-05-24T21:27:12.348 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:12 GMT /qskrps-48/ 2023-05-24T21:27:12.349 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:8TAMkXi8EVUV3ltARiym9EgqjRs= 2023-05-24T21:27:12.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:8TAMkXi8EVUV3ltARiym9EgqjRs=', 'Content-Length': '0'} 2023-05-24T21:27:12.351 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064cb337d9e17ca68-00646e8130-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.396 DEBUG:boto:b'qskrps-48obj11000falseobj12023-05-24T21:27:06.310Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:12.397 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:27:12.397 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:27:12.398 DEBUG:boto:path=/qskrps-48/?prefix=obj1 2023-05-24T21:27:12.398 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj1 2023-05-24T21:27:12.398 DEBUG:boto:Method: GET 2023-05-24T21:27:12.399 DEBUG:boto:Path: /qskrps-48/?prefix=obj1 2023-05-24T21:27:12.399 DEBUG:boto:Data: 2023-05-24T21:27:12.400 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.400 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:12.400 DEBUG:boto:Port: 8000 2023-05-24T21:27:12.401 DEBUG:boto:Params: {} 2023-05-24T21:27:12.401 DEBUG:boto:Token: None 2023-05-24T21:27:12.401 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:12 GMT /qskrps-48/ 2023-05-24T21:27:12.402 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:8TAMkXi8EVUV3ltARiym9EgqjRs= 2023-05-24T21:27:12.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:8TAMkXi8EVUV3ltARiym9EgqjRs=', 'Content-Length': '0'} 2023-05-24T21:27:12.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a5022683cc5d8c2-00646e8130-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.405 DEBUG:boto:b'qskrps-48obj11000falseobj12023-05-24T21:27:06.310Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:12.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-24T21:27:12.406 DEBUG:boto:path=/qskrps-48/obj2 2023-05-24T21:27:12.406 DEBUG:boto:auth_path=/qskrps-48/obj2 2023-05-24T21:27:12.407 DEBUG:boto:path=/qskrps-48/obj2?versionId=null 2023-05-24T21:27:12.407 DEBUG:boto:auth_path=/qskrps-48/obj2?versionId=null 2023-05-24T21:27:12.407 DEBUG:boto:Method: GET 2023-05-24T21:27:12.408 DEBUG:boto:Path: /qskrps-48/obj2?versionId=null 2023-05-24T21:27:12.408 DEBUG:boto:Data: 2023-05-24T21:27:12.408 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.409 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:12.409 DEBUG:boto:Port: 8001 2023-05-24T21:27:12.409 DEBUG:boto:Params: {} 2023-05-24T21:27:12.410 DEBUG:boto:Token: None 2023-05-24T21:27:12.410 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:12 GMT /qskrps-48/obj2?versionId=null 2023-05-24T21:27:12.411 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:WqtUGpR/hb9VUED4hjvxc8rVDo0= 2023-05-24T21:27:12.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WqtUGpR/hb9VUED4hjvxc8rVDo0=', 'Content-Length': '0'} 2023-05-24T21:27:12.413 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000744fc407f06081dc-00646e8130-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.413 DEBUG:boto:path=/qskrps-48/obj2 2023-05-24T21:27:12.414 DEBUG:boto:auth_path=/qskrps-48/obj2 2023-05-24T21:27:12.414 DEBUG:boto:path=/qskrps-48/obj2?versionId=null 2023-05-24T21:27:12.414 DEBUG:boto:auth_path=/qskrps-48/obj2?versionId=null 2023-05-24T21:27:12.415 DEBUG:boto:Method: GET 2023-05-24T21:27:12.415 DEBUG:boto:Path: /qskrps-48/obj2?versionId=null 2023-05-24T21:27:12.415 DEBUG:boto:Data: 2023-05-24T21:27:12.416 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.416 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:12.416 DEBUG:boto:Port: 8000 2023-05-24T21:27:12.417 DEBUG:boto:Params: {} 2023-05-24T21:27:12.417 DEBUG:boto:Token: None 2023-05-24T21:27:12.418 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:12 GMT /qskrps-48/obj2?versionId=null 2023-05-24T21:27:12.418 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:WqtUGpR/hb9VUED4hjvxc8rVDo0= 2023-05-24T21:27:12.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WqtUGpR/hb9VUED4hjvxc8rVDo0=', 'Content-Length': '0'} 2023-05-24T21:27:12.420 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f8dd807348c60d4b-00646e8130-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.421 DEBUG:boto:path=/qskrps-48/obj2 2023-05-24T21:27:12.421 DEBUG:boto:auth_path=/qskrps-48/obj2 2023-05-24T21:27:12.422 DEBUG:boto:path=/qskrps-48/obj2?versionId=null 2023-05-24T21:27:12.422 DEBUG:boto:auth_path=/qskrps-48/obj2?versionId=null 2023-05-24T21:27:12.422 DEBUG:boto:Method: HEAD 2023-05-24T21:27:12.423 DEBUG:boto:Path: /qskrps-48/obj2?versionId=null 2023-05-24T21:27:12.423 DEBUG:boto:Data: 2023-05-24T21:27:12.423 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.424 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:12.424 DEBUG:boto:Port: 8001 2023-05-24T21:27:12.424 DEBUG:boto:Params: {} 2023-05-24T21:27:12.425 DEBUG:boto:Token: None 2023-05-24T21:27:12.425 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:12 GMT /qskrps-48/obj2?versionId=null 2023-05-24T21:27:12.425 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ikGLJI8TEIlpYEUg8zrNJe/7h8U= 2023-05-24T21:27:12.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ikGLJI8TEIlpYEUg8zrNJe/7h8U=', 'Content-Length': '0'} 2023-05-24T21:27:12.427 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d500490f377b77f4-00646e8130-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.428 DEBUG:boto:path=/qskrps-48/obj2 2023-05-24T21:27:12.428 DEBUG:boto:auth_path=/qskrps-48/obj2 2023-05-24T21:27:12.429 DEBUG:boto:path=/qskrps-48/obj2?versionId=null 2023-05-24T21:27:12.429 DEBUG:boto:auth_path=/qskrps-48/obj2?versionId=null 2023-05-24T21:27:12.429 DEBUG:boto:Method: HEAD 2023-05-24T21:27:12.430 DEBUG:boto:Path: /qskrps-48/obj2?versionId=null 2023-05-24T21:27:12.430 DEBUG:boto:Data: 2023-05-24T21:27:12.430 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.431 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:12.431 DEBUG:boto:Port: 8000 2023-05-24T21:27:12.431 DEBUG:boto:Params: {} 2023-05-24T21:27:12.432 DEBUG:boto:Token: None 2023-05-24T21:27:12.432 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:12 GMT /qskrps-48/obj2?versionId=null 2023-05-24T21:27:12.432 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ikGLJI8TEIlpYEUg8zrNJe/7h8U= 2023-05-24T21:27:12.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ikGLJI8TEIlpYEUg8zrNJe/7h8U=', 'Content-Length': '0'} 2023-05-24T21:27:12.434 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e384e2e8c48a4c37-00646e8130-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-24T21:27:12.435 DEBUG:boto:path=/qskrps-48/obj2 2023-05-24T21:27:12.435 DEBUG:boto:auth_path=/qskrps-48/obj2 2023-05-24T21:27:12.436 DEBUG:boto:path=/qskrps-48/obj2?versionId=null 2023-05-24T21:27:12.436 DEBUG:boto:auth_path=/qskrps-48/obj2?versionId=null 2023-05-24T21:27:12.437 DEBUG:boto:Method: GET 2023-05-24T21:27:12.437 DEBUG:boto:Path: /qskrps-48/obj2?versionId=null 2023-05-24T21:27:12.437 DEBUG:boto:Data: 2023-05-24T21:27:12.438 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.438 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:12.438 DEBUG:boto:Port: 8001 2023-05-24T21:27:12.439 DEBUG:boto:Params: {} 2023-05-24T21:27:12.439 DEBUG:boto:Token: None 2023-05-24T21:27:12.439 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:12 GMT /qskrps-48/obj2?versionId=null 2023-05-24T21:27:12.440 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:WqtUGpR/hb9VUED4hjvxc8rVDo0= 2023-05-24T21:27:12.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WqtUGpR/hb9VUED4hjvxc8rVDo0=', 'Content-Length': '0'} 2023-05-24T21:27:12.442 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000038ea73d86d11593-00646e8130-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.442 DEBUG:boto:path=/qskrps-48/obj2 2023-05-24T21:27:12.443 DEBUG:boto:auth_path=/qskrps-48/obj2 2023-05-24T21:27:12.443 DEBUG:boto:path=/qskrps-48/obj2?versionId=null 2023-05-24T21:27:12.444 DEBUG:boto:auth_path=/qskrps-48/obj2?versionId=null 2023-05-24T21:27:12.444 DEBUG:boto:Method: GET 2023-05-24T21:27:12.444 DEBUG:boto:Path: /qskrps-48/obj2?versionId=null 2023-05-24T21:27:12.445 DEBUG:boto:Data: 2023-05-24T21:27:12.445 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.445 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:12.446 DEBUG:boto:Port: 8000 2023-05-24T21:27:12.446 DEBUG:boto:Params: {} 2023-05-24T21:27:12.447 DEBUG:boto:Token: None 2023-05-24T21:27:12.447 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:12 GMT /qskrps-48/obj2?versionId=null 2023-05-24T21:27:12.447 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:WqtUGpR/hb9VUED4hjvxc8rVDo0= 2023-05-24T21:27:12.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WqtUGpR/hb9VUED4hjvxc8rVDo0=', 'Content-Length': '0'} 2023-05-24T21:27:12.449 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004025d0a064b176a4-00646e8130-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.492 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:27:12.493 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:27:12.493 DEBUG:boto:path=/qskrps-48/?prefix=obj2 2023-05-24T21:27:12.493 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj2 2023-05-24T21:27:12.494 DEBUG:boto:Method: GET 2023-05-24T21:27:12.494 DEBUG:boto:Path: /qskrps-48/?prefix=obj2 2023-05-24T21:27:12.494 DEBUG:boto:Data: 2023-05-24T21:27:12.495 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.495 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:12.496 DEBUG:boto:Port: 8001 2023-05-24T21:27:12.496 DEBUG:boto:Params: {} 2023-05-24T21:27:12.496 DEBUG:boto:Token: None 2023-05-24T21:27:12.497 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:12 GMT /qskrps-48/ 2023-05-24T21:27:12.497 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:8TAMkXi8EVUV3ltARiym9EgqjRs= 2023-05-24T21:27:12.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:8TAMkXi8EVUV3ltARiym9EgqjRs=', 'Content-Length': '0'} 2023-05-24T21:27:12.500 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000465408c854ca308b-00646e8130-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.501 DEBUG:boto:b'qskrps-48obj21000falseobj22023-05-24T21:27:06.330Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:12.501 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:27:12.502 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:27:12.502 DEBUG:boto:path=/qskrps-48/?prefix=obj2 2023-05-24T21:27:12.502 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj2 2023-05-24T21:27:12.503 DEBUG:boto:Method: GET 2023-05-24T21:27:12.503 DEBUG:boto:Path: /qskrps-48/?prefix=obj2 2023-05-24T21:27:12.503 DEBUG:boto:Data: 2023-05-24T21:27:12.504 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.504 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:12.504 DEBUG:boto:Port: 8000 2023-05-24T21:27:12.505 DEBUG:boto:Params: {} 2023-05-24T21:27:12.505 DEBUG:boto:Token: None 2023-05-24T21:27:12.506 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:12 GMT /qskrps-48/ 2023-05-24T21:27:12.506 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:8TAMkXi8EVUV3ltARiym9EgqjRs= 2023-05-24T21:27:12.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:8TAMkXi8EVUV3ltARiym9EgqjRs=', 'Content-Length': '0'} 2023-05-24T21:27:12.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000529d83a8fbe8e530-00646e8130-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.509 DEBUG:boto:b'qskrps-48obj21000falseobj22023-05-24T21:27:06.330Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:12.510 DEBUG:boto:path=/qskrps-48/obj2 2023-05-24T21:27:12.510 DEBUG:boto:auth_path=/qskrps-48/obj2 2023-05-24T21:27:12.510 DEBUG:boto:Method: HEAD 2023-05-24T21:27:12.511 DEBUG:boto:Path: /qskrps-48/obj2 2023-05-24T21:27:12.511 DEBUG:boto:Data: 2023-05-24T21:27:12.512 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.512 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:12.512 DEBUG:boto:Port: 8001 2023-05-24T21:27:12.513 DEBUG:boto:Params: {} 2023-05-24T21:27:12.513 DEBUG:boto:Token: None 2023-05-24T21:27:12.513 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:12 GMT /qskrps-48/obj2 2023-05-24T21:27:12.514 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:GqnG1zMSU3dhLHpKCc/7P45UlEQ= 2023-05-24T21:27:12.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:GqnG1zMSU3dhLHpKCc/7P45UlEQ=', 'Content-Length': '0'} 2023-05-24T21:27:12.516 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000021419631acd16437-00646e8130-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.516 DEBUG:boto:path=/qskrps-48/obj2 2023-05-24T21:27:12.517 DEBUG:boto:auth_path=/qskrps-48/obj2 2023-05-24T21:27:12.517 DEBUG:boto:Method: HEAD 2023-05-24T21:27:12.517 DEBUG:boto:Path: /qskrps-48/obj2 2023-05-24T21:27:12.518 DEBUG:boto:Data: 2023-05-24T21:27:12.518 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.518 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:12.519 DEBUG:boto:Port: 8000 2023-05-24T21:27:12.519 DEBUG:boto:Params: {} 2023-05-24T21:27:12.519 DEBUG:boto:Token: None 2023-05-24T21:27:12.520 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:12 GMT /qskrps-48/obj2 2023-05-24T21:27:12.520 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:GqnG1zMSU3dhLHpKCc/7P45UlEQ= 2023-05-24T21:27:12.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:GqnG1zMSU3dhLHpKCc/7P45UlEQ=', 'Content-Length': '0'} 2023-05-24T21:27:12.522 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ef119ba69dc0f8fc-00646e8130-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-24T21:27:12.523 DEBUG:boto:path=/qskrps-48/obj2 2023-05-24T21:27:12.523 DEBUG:boto:auth_path=/qskrps-48/obj2 2023-05-24T21:27:12.524 DEBUG:boto:Method: GET 2023-05-24T21:27:12.524 DEBUG:boto:Path: /qskrps-48/obj2 2023-05-24T21:27:12.524 DEBUG:boto:Data: 2023-05-24T21:27:12.525 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.525 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:12.525 DEBUG:boto:Port: 8001 2023-05-24T21:27:12.526 DEBUG:boto:Params: {} 2023-05-24T21:27:12.526 DEBUG:boto:Token: None 2023-05-24T21:27:12.527 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:12 GMT /qskrps-48/obj2 2023-05-24T21:27:12.527 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Yv/Cm9BzKeeiEOXr5VNvXnV38vg= 2023-05-24T21:27:12.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Yv/Cm9BzKeeiEOXr5VNvXnV38vg=', 'Content-Length': '0'} 2023-05-24T21:27:12.529 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009d3acff10b6b1d21-00646e8130-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.572 DEBUG:boto:path=/qskrps-48/obj2 2023-05-24T21:27:12.573 DEBUG:boto:auth_path=/qskrps-48/obj2 2023-05-24T21:27:12.573 DEBUG:boto:Method: GET 2023-05-24T21:27:12.573 DEBUG:boto:Path: /qskrps-48/obj2 2023-05-24T21:27:12.574 DEBUG:boto:Data: 2023-05-24T21:27:12.574 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.575 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:12.575 DEBUG:boto:Port: 8000 2023-05-24T21:27:12.575 DEBUG:boto:Params: {} 2023-05-24T21:27:12.576 DEBUG:boto:Token: None 2023-05-24T21:27:12.576 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:12 GMT /qskrps-48/obj2 2023-05-24T21:27:12.577 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Yv/Cm9BzKeeiEOXr5VNvXnV38vg= 2023-05-24T21:27:12.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Yv/Cm9BzKeeiEOXr5VNvXnV38vg=', 'Content-Length': '0'} 2023-05-24T21:27:12.579 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006db7ad0934a55862-00646e8130-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.579 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:27:12.580 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:27:12.580 DEBUG:boto:path=/qskrps-48/?prefix=obj2 2023-05-24T21:27:12.581 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj2 2023-05-24T21:27:12.581 DEBUG:boto:Method: GET 2023-05-24T21:27:12.581 DEBUG:boto:Path: /qskrps-48/?prefix=obj2 2023-05-24T21:27:12.582 DEBUG:boto:Data: 2023-05-24T21:27:12.582 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.582 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:12.583 DEBUG:boto:Port: 8001 2023-05-24T21:27:12.583 DEBUG:boto:Params: {} 2023-05-24T21:27:12.583 DEBUG:boto:Token: None 2023-05-24T21:27:12.584 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:12 GMT /qskrps-48/ 2023-05-24T21:27:12.584 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:8TAMkXi8EVUV3ltARiym9EgqjRs= 2023-05-24T21:27:12.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:8TAMkXi8EVUV3ltARiym9EgqjRs=', 'Content-Length': '0'} 2023-05-24T21:27:12.587 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008058105229d0fce4-00646e8130-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.587 DEBUG:boto:b'qskrps-48obj21000falseobj22023-05-24T21:27:06.330Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:12.588 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:27:12.588 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:27:12.589 DEBUG:boto:path=/qskrps-48/?prefix=obj2 2023-05-24T21:27:12.589 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj2 2023-05-24T21:27:12.589 DEBUG:boto:Method: GET 2023-05-24T21:27:12.590 DEBUG:boto:Path: /qskrps-48/?prefix=obj2 2023-05-24T21:27:12.590 DEBUG:boto:Data: 2023-05-24T21:27:12.590 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.591 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:12.591 DEBUG:boto:Port: 8000 2023-05-24T21:27:12.592 DEBUG:boto:Params: {} 2023-05-24T21:27:12.592 DEBUG:boto:Token: None 2023-05-24T21:27:12.592 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:12 GMT /qskrps-48/ 2023-05-24T21:27:12.593 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:8TAMkXi8EVUV3ltARiym9EgqjRs= 2023-05-24T21:27:12.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:8TAMkXi8EVUV3ltARiym9EgqjRs=', 'Content-Length': '0'} 2023-05-24T21:27:12.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4c3dd753af48f32-00646e8130-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.596 DEBUG:boto:b'qskrps-48obj21000falseobj22023-05-24T21:27:06.330Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:12.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-24T21:27:12.597 DEBUG:boto:path=/qskrps-48/obj3 2023-05-24T21:27:12.597 DEBUG:boto:auth_path=/qskrps-48/obj3 2023-05-24T21:27:12.597 DEBUG:boto:path=/qskrps-48/obj3?versionId=null 2023-05-24T21:27:12.598 DEBUG:boto:auth_path=/qskrps-48/obj3?versionId=null 2023-05-24T21:27:12.598 DEBUG:boto:Method: GET 2023-05-24T21:27:12.599 DEBUG:boto:Path: /qskrps-48/obj3?versionId=null 2023-05-24T21:27:12.599 DEBUG:boto:Data: 2023-05-24T21:27:12.599 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.600 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:12.600 DEBUG:boto:Port: 8001 2023-05-24T21:27:12.600 DEBUG:boto:Params: {} 2023-05-24T21:27:12.601 DEBUG:boto:Token: None 2023-05-24T21:27:12.601 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:12 GMT /qskrps-48/obj3?versionId=null 2023-05-24T21:27:12.602 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:WyDpsUStotKuTsxH3/FMMg1jCJg= 2023-05-24T21:27:12.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WyDpsUStotKuTsxH3/FMMg1jCJg=', 'Content-Length': '0'} 2023-05-24T21:27:12.603 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003aa80cabf3106d2e-00646e8130-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.604 DEBUG:boto:path=/qskrps-48/obj3 2023-05-24T21:27:12.605 DEBUG:boto:auth_path=/qskrps-48/obj3 2023-05-24T21:27:12.605 DEBUG:boto:path=/qskrps-48/obj3?versionId=null 2023-05-24T21:27:12.605 DEBUG:boto:auth_path=/qskrps-48/obj3?versionId=null 2023-05-24T21:27:12.606 DEBUG:boto:Method: GET 2023-05-24T21:27:12.606 DEBUG:boto:Path: /qskrps-48/obj3?versionId=null 2023-05-24T21:27:12.606 DEBUG:boto:Data: 2023-05-24T21:27:12.607 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.607 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:12.607 DEBUG:boto:Port: 8000 2023-05-24T21:27:12.608 DEBUG:boto:Params: {} 2023-05-24T21:27:12.608 DEBUG:boto:Token: None 2023-05-24T21:27:12.609 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:12 GMT /qskrps-48/obj3?versionId=null 2023-05-24T21:27:12.609 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:WyDpsUStotKuTsxH3/FMMg1jCJg= 2023-05-24T21:27:12.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WyDpsUStotKuTsxH3/FMMg1jCJg=', 'Content-Length': '0'} 2023-05-24T21:27:12.611 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000341c7678a0daa114-00646e8130-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.612 DEBUG:boto:path=/qskrps-48/obj3 2023-05-24T21:27:12.612 DEBUG:boto:auth_path=/qskrps-48/obj3 2023-05-24T21:27:12.612 DEBUG:boto:path=/qskrps-48/obj3?versionId=null 2023-05-24T21:27:12.613 DEBUG:boto:auth_path=/qskrps-48/obj3?versionId=null 2023-05-24T21:27:12.613 DEBUG:boto:Method: HEAD 2023-05-24T21:27:12.613 DEBUG:boto:Path: /qskrps-48/obj3?versionId=null 2023-05-24T21:27:12.614 DEBUG:boto:Data: 2023-05-24T21:27:12.614 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.614 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:12.615 DEBUG:boto:Port: 8001 2023-05-24T21:27:12.615 DEBUG:boto:Params: {} 2023-05-24T21:27:12.615 DEBUG:boto:Token: None 2023-05-24T21:27:12.616 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:12 GMT /qskrps-48/obj3?versionId=null 2023-05-24T21:27:12.616 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:E3yvXdQRVMbzU6qHS6wH/dq7TCo= 2023-05-24T21:27:12.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:E3yvXdQRVMbzU6qHS6wH/dq7TCo=', 'Content-Length': '0'} 2023-05-24T21:27:12.618 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000260f4c94549c891d-00646e8130-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.618 DEBUG:boto:path=/qskrps-48/obj3 2023-05-24T21:27:12.619 DEBUG:boto:auth_path=/qskrps-48/obj3 2023-05-24T21:27:12.619 DEBUG:boto:path=/qskrps-48/obj3?versionId=null 2023-05-24T21:27:12.620 DEBUG:boto:auth_path=/qskrps-48/obj3?versionId=null 2023-05-24T21:27:12.620 DEBUG:boto:Method: HEAD 2023-05-24T21:27:12.620 DEBUG:boto:Path: /qskrps-48/obj3?versionId=null 2023-05-24T21:27:12.621 DEBUG:boto:Data: 2023-05-24T21:27:12.621 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.621 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:12.622 DEBUG:boto:Port: 8000 2023-05-24T21:27:12.622 DEBUG:boto:Params: {} 2023-05-24T21:27:12.622 DEBUG:boto:Token: None 2023-05-24T21:27:12.623 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:12 GMT /qskrps-48/obj3?versionId=null 2023-05-24T21:27:12.623 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:E3yvXdQRVMbzU6qHS6wH/dq7TCo= 2023-05-24T21:27:12.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:E3yvXdQRVMbzU6qHS6wH/dq7TCo=', 'Content-Length': '0'} 2023-05-24T21:27:12.625 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001845e31e476c9cdd-00646e8130-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-24T21:27:12.626 DEBUG:boto:path=/qskrps-48/obj3 2023-05-24T21:27:12.626 DEBUG:boto:auth_path=/qskrps-48/obj3 2023-05-24T21:27:12.627 DEBUG:boto:path=/qskrps-48/obj3?versionId=null 2023-05-24T21:27:12.627 DEBUG:boto:auth_path=/qskrps-48/obj3?versionId=null 2023-05-24T21:27:12.627 DEBUG:boto:Method: GET 2023-05-24T21:27:12.628 DEBUG:boto:Path: /qskrps-48/obj3?versionId=null 2023-05-24T21:27:12.628 DEBUG:boto:Data: 2023-05-24T21:27:12.628 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.629 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:12.629 DEBUG:boto:Port: 8001 2023-05-24T21:27:12.629 DEBUG:boto:Params: {} 2023-05-24T21:27:12.630 DEBUG:boto:Token: None 2023-05-24T21:27:12.630 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:12 GMT /qskrps-48/obj3?versionId=null 2023-05-24T21:27:12.631 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:WyDpsUStotKuTsxH3/FMMg1jCJg= 2023-05-24T21:27:12.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WyDpsUStotKuTsxH3/FMMg1jCJg=', 'Content-Length': '0'} 2023-05-24T21:27:12.633 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000032f33beeb1fb15ff-00646e8130-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.633 DEBUG:boto:path=/qskrps-48/obj3 2023-05-24T21:27:12.634 DEBUG:boto:auth_path=/qskrps-48/obj3 2023-05-24T21:27:12.634 DEBUG:boto:path=/qskrps-48/obj3?versionId=null 2023-05-24T21:27:12.634 DEBUG:boto:auth_path=/qskrps-48/obj3?versionId=null 2023-05-24T21:27:12.635 DEBUG:boto:Method: GET 2023-05-24T21:27:12.635 DEBUG:boto:Path: /qskrps-48/obj3?versionId=null 2023-05-24T21:27:12.635 DEBUG:boto:Data: 2023-05-24T21:27:12.636 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.636 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:12.637 DEBUG:boto:Port: 8000 2023-05-24T21:27:12.637 DEBUG:boto:Params: {} 2023-05-24T21:27:12.637 DEBUG:boto:Token: None 2023-05-24T21:27:12.638 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:12 GMT /qskrps-48/obj3?versionId=null 2023-05-24T21:27:12.638 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:WyDpsUStotKuTsxH3/FMMg1jCJg= 2023-05-24T21:27:12.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WyDpsUStotKuTsxH3/FMMg1jCJg=', 'Content-Length': '0'} 2023-05-24T21:27:12.640 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000069a666feb0d34cc-00646e8130-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.684 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:27:12.685 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:27:12.685 DEBUG:boto:path=/qskrps-48/?prefix=obj3 2023-05-24T21:27:12.685 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj3 2023-05-24T21:27:12.686 DEBUG:boto:Method: GET 2023-05-24T21:27:12.686 DEBUG:boto:Path: /qskrps-48/?prefix=obj3 2023-05-24T21:27:12.686 DEBUG:boto:Data: 2023-05-24T21:27:12.687 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.687 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:12.688 DEBUG:boto:Port: 8001 2023-05-24T21:27:12.688 DEBUG:boto:Params: {} 2023-05-24T21:27:12.688 DEBUG:boto:Token: None 2023-05-24T21:27:12.689 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:12 GMT /qskrps-48/ 2023-05-24T21:27:12.689 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:8TAMkXi8EVUV3ltARiym9EgqjRs= 2023-05-24T21:27:12.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:8TAMkXi8EVUV3ltARiym9EgqjRs=', 'Content-Length': '0'} 2023-05-24T21:27:12.692 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afe319300d8b997a-00646e8130-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.692 DEBUG:boto:b'qskrps-48obj31000falseobj32023-05-24T21:27:06.348Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:12.693 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:27:12.693 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:27:12.694 DEBUG:boto:path=/qskrps-48/?prefix=obj3 2023-05-24T21:27:12.694 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj3 2023-05-24T21:27:12.694 DEBUG:boto:Method: GET 2023-05-24T21:27:12.695 DEBUG:boto:Path: /qskrps-48/?prefix=obj3 2023-05-24T21:27:12.695 DEBUG:boto:Data: 2023-05-24T21:27:12.695 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.696 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:12.696 DEBUG:boto:Port: 8000 2023-05-24T21:27:12.696 DEBUG:boto:Params: {} 2023-05-24T21:27:12.697 DEBUG:boto:Token: None 2023-05-24T21:27:12.697 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:12 GMT /qskrps-48/ 2023-05-24T21:27:12.698 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:8TAMkXi8EVUV3ltARiym9EgqjRs= 2023-05-24T21:27:12.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:8TAMkXi8EVUV3ltARiym9EgqjRs=', 'Content-Length': '0'} 2023-05-24T21:27:12.700 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cc25639624d0a54-00646e8130-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.701 DEBUG:boto:b'qskrps-48obj31000falseobj32023-05-24T21:27:06.348Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:12.701 DEBUG:boto:path=/qskrps-48/obj3 2023-05-24T21:27:12.702 DEBUG:boto:auth_path=/qskrps-48/obj3 2023-05-24T21:27:12.702 DEBUG:boto:Method: HEAD 2023-05-24T21:27:12.703 DEBUG:boto:Path: /qskrps-48/obj3 2023-05-24T21:27:12.703 DEBUG:boto:Data: 2023-05-24T21:27:12.703 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.704 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:12.704 DEBUG:boto:Port: 8001 2023-05-24T21:27:12.704 DEBUG:boto:Params: {} 2023-05-24T21:27:12.705 DEBUG:boto:Token: None 2023-05-24T21:27:12.705 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:12 GMT /qskrps-48/obj3 2023-05-24T21:27:12.705 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Vl2aQEjnOCdGiLVPyFynQu0vGog= 2023-05-24T21:27:12.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Vl2aQEjnOCdGiLVPyFynQu0vGog=', 'Content-Length': '0'} 2023-05-24T21:27:12.707 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000076552abd497f14e6-00646e8130-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.708 DEBUG:boto:path=/qskrps-48/obj3 2023-05-24T21:27:12.708 DEBUG:boto:auth_path=/qskrps-48/obj3 2023-05-24T21:27:12.709 DEBUG:boto:Method: HEAD 2023-05-24T21:27:12.709 DEBUG:boto:Path: /qskrps-48/obj3 2023-05-24T21:27:12.709 DEBUG:boto:Data: 2023-05-24T21:27:12.710 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.710 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:12.710 DEBUG:boto:Port: 8000 2023-05-24T21:27:12.711 DEBUG:boto:Params: {} 2023-05-24T21:27:12.711 DEBUG:boto:Token: None 2023-05-24T21:27:12.712 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:12 GMT /qskrps-48/obj3 2023-05-24T21:27:12.712 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Vl2aQEjnOCdGiLVPyFynQu0vGog= 2023-05-24T21:27:12.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Vl2aQEjnOCdGiLVPyFynQu0vGog=', 'Content-Length': '0'} 2023-05-24T21:27:12.714 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bcd0f53981e61dfc-00646e8130-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-24T21:27:12.715 DEBUG:boto:path=/qskrps-48/obj3 2023-05-24T21:27:12.715 DEBUG:boto:auth_path=/qskrps-48/obj3 2023-05-24T21:27:12.716 DEBUG:boto:Method: GET 2023-05-24T21:27:12.716 DEBUG:boto:Path: /qskrps-48/obj3 2023-05-24T21:27:12.716 DEBUG:boto:Data: 2023-05-24T21:27:12.717 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.717 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:12.718 DEBUG:boto:Port: 8001 2023-05-24T21:27:12.718 DEBUG:boto:Params: {} 2023-05-24T21:27:12.718 DEBUG:boto:Token: None 2023-05-24T21:27:12.719 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:12 GMT /qskrps-48/obj3 2023-05-24T21:27:12.719 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:jsvvueriBIXVIw1/qilmCL6SDi0= 2023-05-24T21:27:12.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:jsvvueriBIXVIw1/qilmCL6SDi0=', 'Content-Length': '0'} 2023-05-24T21:27:12.721 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ffce96b128d83c56-00646e8130-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.722 DEBUG:boto:path=/qskrps-48/obj3 2023-05-24T21:27:12.722 DEBUG:boto:auth_path=/qskrps-48/obj3 2023-05-24T21:27:12.722 DEBUG:boto:Method: GET 2023-05-24T21:27:12.723 DEBUG:boto:Path: /qskrps-48/obj3 2023-05-24T21:27:12.723 DEBUG:boto:Data: 2023-05-24T21:27:12.723 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.724 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:12.724 DEBUG:boto:Port: 8000 2023-05-24T21:27:12.725 DEBUG:boto:Params: {} 2023-05-24T21:27:12.725 DEBUG:boto:Token: None 2023-05-24T21:27:12.725 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:12 GMT /qskrps-48/obj3 2023-05-24T21:27:12.726 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:jsvvueriBIXVIw1/qilmCL6SDi0= 2023-05-24T21:27:12.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:jsvvueriBIXVIw1/qilmCL6SDi0=', 'Content-Length': '0'} 2023-05-24T21:27:12.728 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009fa55ef24565f8e5-00646e8130-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.728 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:27:12.729 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:27:12.729 DEBUG:boto:path=/qskrps-48/?prefix=obj3 2023-05-24T21:27:12.730 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj3 2023-05-24T21:27:12.730 DEBUG:boto:Method: GET 2023-05-24T21:27:12.730 DEBUG:boto:Path: /qskrps-48/?prefix=obj3 2023-05-24T21:27:12.731 DEBUG:boto:Data: 2023-05-24T21:27:12.731 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.731 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:12.732 DEBUG:boto:Port: 8001 2023-05-24T21:27:12.732 DEBUG:boto:Params: {} 2023-05-24T21:27:12.732 DEBUG:boto:Token: None 2023-05-24T21:27:12.733 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:12 GMT /qskrps-48/ 2023-05-24T21:27:12.733 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:8TAMkXi8EVUV3ltARiym9EgqjRs= 2023-05-24T21:27:12.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:8TAMkXi8EVUV3ltARiym9EgqjRs=', 'Content-Length': '0'} 2023-05-24T21:27:12.736 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000129cedcc2f0c75b5-00646e8130-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.736 DEBUG:boto:b'qskrps-48obj31000falseobj32023-05-24T21:27:06.348Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:12.737 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:27:12.737 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:27:12.738 DEBUG:boto:path=/qskrps-48/?prefix=obj3 2023-05-24T21:27:12.738 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj3 2023-05-24T21:27:12.738 DEBUG:boto:Method: GET 2023-05-24T21:27:12.739 DEBUG:boto:Path: /qskrps-48/?prefix=obj3 2023-05-24T21:27:12.739 DEBUG:boto:Data: 2023-05-24T21:27:12.739 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.740 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:12.740 DEBUG:boto:Port: 8000 2023-05-24T21:27:12.740 DEBUG:boto:Params: {} 2023-05-24T21:27:12.741 DEBUG:boto:Token: None 2023-05-24T21:27:12.741 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:12 GMT /qskrps-48/ 2023-05-24T21:27:12.742 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:8TAMkXi8EVUV3ltARiym9EgqjRs= 2023-05-24T21:27:12.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:8TAMkXi8EVUV3ltARiym9EgqjRs=', 'Content-Length': '0'} 2023-05-24T21:27:12.744 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b628813be1d9d761-00646e8130-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.745 DEBUG:boto:b'qskrps-48obj31000falseobj32023-05-24T21:27:06.348Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:12.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-24T21:27:12.746 DEBUG:boto:path=/qskrps-48/obj4 2023-05-24T21:27:12.746 DEBUG:boto:auth_path=/qskrps-48/obj4 2023-05-24T21:27:12.746 DEBUG:boto:path=/qskrps-48/obj4?versionId=null 2023-05-24T21:27:12.747 DEBUG:boto:auth_path=/qskrps-48/obj4?versionId=null 2023-05-24T21:27:12.747 DEBUG:boto:Method: GET 2023-05-24T21:27:12.747 DEBUG:boto:Path: /qskrps-48/obj4?versionId=null 2023-05-24T21:27:12.748 DEBUG:boto:Data: 2023-05-24T21:27:12.748 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.748 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:12.749 DEBUG:boto:Port: 8001 2023-05-24T21:27:12.749 DEBUG:boto:Params: {} 2023-05-24T21:27:12.750 DEBUG:boto:Token: None 2023-05-24T21:27:12.750 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:12 GMT /qskrps-48/obj4?versionId=null 2023-05-24T21:27:12.750 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Q2XOus2nA9ifibka5FgwzaQP9uE= 2023-05-24T21:27:12.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Q2XOus2nA9ifibka5FgwzaQP9uE=', 'Content-Length': '0'} 2023-05-24T21:27:12.753 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000047c28e0bccee4206-00646e8130-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.753 DEBUG:boto:path=/qskrps-48/obj4 2023-05-24T21:27:12.754 DEBUG:boto:auth_path=/qskrps-48/obj4 2023-05-24T21:27:12.754 DEBUG:boto:path=/qskrps-48/obj4?versionId=null 2023-05-24T21:27:12.754 DEBUG:boto:auth_path=/qskrps-48/obj4?versionId=null 2023-05-24T21:27:12.755 DEBUG:boto:Method: GET 2023-05-24T21:27:12.755 DEBUG:boto:Path: /qskrps-48/obj4?versionId=null 2023-05-24T21:27:12.755 DEBUG:boto:Data: 2023-05-24T21:27:12.756 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.756 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:12.756 DEBUG:boto:Port: 8000 2023-05-24T21:27:12.757 DEBUG:boto:Params: {} 2023-05-24T21:27:12.757 DEBUG:boto:Token: None 2023-05-24T21:27:12.758 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:12 GMT /qskrps-48/obj4?versionId=null 2023-05-24T21:27:12.758 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Q2XOus2nA9ifibka5FgwzaQP9uE= 2023-05-24T21:27:12.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Q2XOus2nA9ifibka5FgwzaQP9uE=', 'Content-Length': '0'} 2023-05-24T21:27:12.760 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d5b72123fce86c1c-00646e8130-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.804 DEBUG:boto:path=/qskrps-48/obj4 2023-05-24T21:27:12.805 DEBUG:boto:auth_path=/qskrps-48/obj4 2023-05-24T21:27:12.805 DEBUG:boto:path=/qskrps-48/obj4?versionId=null 2023-05-24T21:27:12.805 DEBUG:boto:auth_path=/qskrps-48/obj4?versionId=null 2023-05-24T21:27:12.806 DEBUG:boto:Method: HEAD 2023-05-24T21:27:12.806 DEBUG:boto:Path: /qskrps-48/obj4?versionId=null 2023-05-24T21:27:12.807 DEBUG:boto:Data: 2023-05-24T21:27:12.807 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.807 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:12.807 DEBUG:boto:Port: 8001 2023-05-24T21:27:12.808 DEBUG:boto:Params: {} 2023-05-24T21:27:12.808 DEBUG:boto:Token: None 2023-05-24T21:27:12.809 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:12 GMT /qskrps-48/obj4?versionId=null 2023-05-24T21:27:12.809 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5ZR3lLHevQ7ty5axboKaw2qZpc4= 2023-05-24T21:27:12.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5ZR3lLHevQ7ty5axboKaw2qZpc4=', 'Content-Length': '0'} 2023-05-24T21:27:12.811 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000033032ec65fdbd080-00646e8130-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.812 DEBUG:boto:path=/qskrps-48/obj4 2023-05-24T21:27:12.812 DEBUG:boto:auth_path=/qskrps-48/obj4 2023-05-24T21:27:12.812 DEBUG:boto:path=/qskrps-48/obj4?versionId=null 2023-05-24T21:27:12.813 DEBUG:boto:auth_path=/qskrps-48/obj4?versionId=null 2023-05-24T21:27:12.813 DEBUG:boto:Method: HEAD 2023-05-24T21:27:12.813 DEBUG:boto:Path: /qskrps-48/obj4?versionId=null 2023-05-24T21:27:12.814 DEBUG:boto:Data: 2023-05-24T21:27:12.814 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.814 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:12.815 DEBUG:boto:Port: 8000 2023-05-24T21:27:12.815 DEBUG:boto:Params: {} 2023-05-24T21:27:12.815 DEBUG:boto:Token: None 2023-05-24T21:27:12.816 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:12 GMT /qskrps-48/obj4?versionId=null 2023-05-24T21:27:12.816 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:5ZR3lLHevQ7ty5axboKaw2qZpc4= 2023-05-24T21:27:12.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5ZR3lLHevQ7ty5axboKaw2qZpc4=', 'Content-Length': '0'} 2023-05-24T21:27:12.818 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000032490df9eb45af24-00646e8130-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-24T21:27:12.819 DEBUG:boto:path=/qskrps-48/obj4 2023-05-24T21:27:12.820 DEBUG:boto:auth_path=/qskrps-48/obj4 2023-05-24T21:27:12.820 DEBUG:boto:path=/qskrps-48/obj4?versionId=null 2023-05-24T21:27:12.820 DEBUG:boto:auth_path=/qskrps-48/obj4?versionId=null 2023-05-24T21:27:12.821 DEBUG:boto:Method: GET 2023-05-24T21:27:12.821 DEBUG:boto:Path: /qskrps-48/obj4?versionId=null 2023-05-24T21:27:12.821 DEBUG:boto:Data: 2023-05-24T21:27:12.822 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.822 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:12.823 DEBUG:boto:Port: 8001 2023-05-24T21:27:12.823 DEBUG:boto:Params: {} 2023-05-24T21:27:12.823 DEBUG:boto:Token: None 2023-05-24T21:27:12.824 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:12 GMT /qskrps-48/obj4?versionId=null 2023-05-24T21:27:12.824 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Q2XOus2nA9ifibka5FgwzaQP9uE= 2023-05-24T21:27:12.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Q2XOus2nA9ifibka5FgwzaQP9uE=', 'Content-Length': '0'} 2023-05-24T21:27:12.826 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000057e5ff731a3510ff-00646e8130-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.827 DEBUG:boto:path=/qskrps-48/obj4 2023-05-24T21:27:12.827 DEBUG:boto:auth_path=/qskrps-48/obj4 2023-05-24T21:27:12.827 DEBUG:boto:path=/qskrps-48/obj4?versionId=null 2023-05-24T21:27:12.828 DEBUG:boto:auth_path=/qskrps-48/obj4?versionId=null 2023-05-24T21:27:12.828 DEBUG:boto:Method: GET 2023-05-24T21:27:12.829 DEBUG:boto:Path: /qskrps-48/obj4?versionId=null 2023-05-24T21:27:12.829 DEBUG:boto:Data: 2023-05-24T21:27:12.829 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.830 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:12.830 DEBUG:boto:Port: 8000 2023-05-24T21:27:12.830 DEBUG:boto:Params: {} 2023-05-24T21:27:12.831 DEBUG:boto:Token: None 2023-05-24T21:27:12.831 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:12 GMT /qskrps-48/obj4?versionId=null 2023-05-24T21:27:12.832 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Q2XOus2nA9ifibka5FgwzaQP9uE= 2023-05-24T21:27:12.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Q2XOus2nA9ifibka5FgwzaQP9uE=', 'Content-Length': '0'} 2023-05-24T21:27:12.834 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001d439ddc0f53d6e8-00646e8130-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.835 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:27:12.835 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:27:12.836 DEBUG:boto:path=/qskrps-48/?prefix=obj4 2023-05-24T21:27:12.836 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj4 2023-05-24T21:27:12.836 DEBUG:boto:Method: GET 2023-05-24T21:27:12.837 DEBUG:boto:Path: /qskrps-48/?prefix=obj4 2023-05-24T21:27:12.837 DEBUG:boto:Data: 2023-05-24T21:27:12.837 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.838 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:12.838 DEBUG:boto:Port: 8001 2023-05-24T21:27:12.838 DEBUG:boto:Params: {} 2023-05-24T21:27:12.839 DEBUG:boto:Token: None 2023-05-24T21:27:12.839 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:12 GMT /qskrps-48/ 2023-05-24T21:27:12.840 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:8TAMkXi8EVUV3ltARiym9EgqjRs= 2023-05-24T21:27:12.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:8TAMkXi8EVUV3ltARiym9EgqjRs=', 'Content-Length': '0'} 2023-05-24T21:27:12.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065eb3e33d6d6062b-00646e8130-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.843 DEBUG:boto:b'qskrps-48obj41000falseobj42023-05-24T21:27:06.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:12.844 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:27:12.844 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:27:12.844 DEBUG:boto:path=/qskrps-48/?prefix=obj4 2023-05-24T21:27:12.845 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj4 2023-05-24T21:27:12.845 DEBUG:boto:Method: GET 2023-05-24T21:27:12.845 DEBUG:boto:Path: /qskrps-48/?prefix=obj4 2023-05-24T21:27:12.846 DEBUG:boto:Data: 2023-05-24T21:27:12.846 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.847 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:12.847 DEBUG:boto:Port: 8000 2023-05-24T21:27:12.847 DEBUG:boto:Params: {} 2023-05-24T21:27:12.848 DEBUG:boto:Token: None 2023-05-24T21:27:12.848 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:12 GMT /qskrps-48/ 2023-05-24T21:27:12.848 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:8TAMkXi8EVUV3ltARiym9EgqjRs= 2023-05-24T21:27:12.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:8TAMkXi8EVUV3ltARiym9EgqjRs=', 'Content-Length': '0'} 2023-05-24T21:27:12.851 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc0318afee8a00ea-00646e8130-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.851 DEBUG:boto:b'qskrps-48obj41000falseobj42023-05-24T21:27:06.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:12.852 DEBUG:boto:path=/qskrps-48/obj4 2023-05-24T21:27:12.853 DEBUG:boto:auth_path=/qskrps-48/obj4 2023-05-24T21:27:12.853 DEBUG:boto:Method: HEAD 2023-05-24T21:27:12.853 DEBUG:boto:Path: /qskrps-48/obj4 2023-05-24T21:27:12.854 DEBUG:boto:Data: 2023-05-24T21:27:12.854 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.854 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:12.855 DEBUG:boto:Port: 8001 2023-05-24T21:27:12.855 DEBUG:boto:Params: {} 2023-05-24T21:27:12.855 DEBUG:boto:Token: None 2023-05-24T21:27:12.856 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:12 GMT /qskrps-48/obj4 2023-05-24T21:27:12.856 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:d//diSA+oucOqBLwLi1g1PT3sdU= 2023-05-24T21:27:12.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:d//diSA+oucOqBLwLi1g1PT3sdU=', 'Content-Length': '0'} 2023-05-24T21:27:12.858 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003182be8d374df1d6-00646e8130-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.858 DEBUG:boto:path=/qskrps-48/obj4 2023-05-24T21:27:12.859 DEBUG:boto:auth_path=/qskrps-48/obj4 2023-05-24T21:27:12.859 DEBUG:boto:Method: HEAD 2023-05-24T21:27:12.859 DEBUG:boto:Path: /qskrps-48/obj4 2023-05-24T21:27:12.860 DEBUG:boto:Data: 2023-05-24T21:27:12.860 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.860 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:12.861 DEBUG:boto:Port: 8000 2023-05-24T21:27:12.861 DEBUG:boto:Params: {} 2023-05-24T21:27:12.862 DEBUG:boto:Token: None 2023-05-24T21:27:12.862 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:27:12 GMT /qskrps-48/obj4 2023-05-24T21:27:12.862 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:d//diSA+oucOqBLwLi1g1PT3sdU= 2023-05-24T21:27:12.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:d//diSA+oucOqBLwLi1g1PT3sdU=', 'Content-Length': '0'} 2023-05-24T21:27:12.864 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002cc0b36a8b840d7c-00646e8130-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-24T21:27:12.865 DEBUG:boto:path=/qskrps-48/obj4 2023-05-24T21:27:12.865 DEBUG:boto:auth_path=/qskrps-48/obj4 2023-05-24T21:27:12.866 DEBUG:boto:Method: GET 2023-05-24T21:27:12.866 DEBUG:boto:Path: /qskrps-48/obj4 2023-05-24T21:27:12.867 DEBUG:boto:Data: 2023-05-24T21:27:12.867 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.867 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:12.868 DEBUG:boto:Port: 8001 2023-05-24T21:27:12.868 DEBUG:boto:Params: {} 2023-05-24T21:27:12.868 DEBUG:boto:Token: None 2023-05-24T21:27:12.869 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:12 GMT /qskrps-48/obj4 2023-05-24T21:27:12.869 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:iQW7JQ8jpKlkuLdHFx2S736J4FY= 2023-05-24T21:27:12.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:iQW7JQ8jpKlkuLdHFx2S736J4FY=', 'Content-Length': '0'} 2023-05-24T21:27:12.871 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005a5cce321c58d562-00646e8130-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.872 DEBUG:boto:path=/qskrps-48/obj4 2023-05-24T21:27:12.872 DEBUG:boto:auth_path=/qskrps-48/obj4 2023-05-24T21:27:12.873 DEBUG:boto:Method: GET 2023-05-24T21:27:12.873 DEBUG:boto:Path: /qskrps-48/obj4 2023-05-24T21:27:12.873 DEBUG:boto:Data: 2023-05-24T21:27:12.874 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.874 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:12.874 DEBUG:boto:Port: 8000 2023-05-24T21:27:12.875 DEBUG:boto:Params: {} 2023-05-24T21:27:12.875 DEBUG:boto:Token: None 2023-05-24T21:27:12.875 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:12 GMT /qskrps-48/obj4 2023-05-24T21:27:12.876 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:iQW7JQ8jpKlkuLdHFx2S736J4FY= 2023-05-24T21:27:12.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:iQW7JQ8jpKlkuLdHFx2S736J4FY=', 'Content-Length': '0'} 2023-05-24T21:27:12.878 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001fa2c8fe71e8b72a-00646e8130-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.878 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:27:12.879 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:27:12.879 DEBUG:boto:path=/qskrps-48/?prefix=obj4 2023-05-24T21:27:12.880 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj4 2023-05-24T21:27:12.880 DEBUG:boto:Method: GET 2023-05-24T21:27:12.880 DEBUG:boto:Path: /qskrps-48/?prefix=obj4 2023-05-24T21:27:12.881 DEBUG:boto:Data: 2023-05-24T21:27:12.881 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.881 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:27:12.882 DEBUG:boto:Port: 8001 2023-05-24T21:27:12.882 DEBUG:boto:Params: {} 2023-05-24T21:27:12.883 DEBUG:boto:Token: None 2023-05-24T21:27:12.883 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:12 GMT /qskrps-48/ 2023-05-24T21:27:12.883 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:8TAMkXi8EVUV3ltARiym9EgqjRs= 2023-05-24T21:27:12.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:8TAMkXi8EVUV3ltARiym9EgqjRs=', 'Content-Length': '0'} 2023-05-24T21:27:12.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d8bae62e49e67c1-00646e8130-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.886 DEBUG:boto:b'qskrps-48obj41000falseobj42023-05-24T21:27:06.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:12.887 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:27:12.887 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:27:12.888 DEBUG:boto:path=/qskrps-48/?prefix=obj4 2023-05-24T21:27:12.888 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj4 2023-05-24T21:27:12.889 DEBUG:boto:Method: GET 2023-05-24T21:27:12.889 DEBUG:boto:Path: /qskrps-48/?prefix=obj4 2023-05-24T21:27:12.889 DEBUG:boto:Data: 2023-05-24T21:27:12.890 DEBUG:boto:Headers: {} 2023-05-24T21:27:12.890 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:27:12.890 DEBUG:boto:Port: 8000 2023-05-24T21:27:12.891 DEBUG:boto:Params: {} 2023-05-24T21:27:12.891 DEBUG:boto:Token: None 2023-05-24T21:27:12.891 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:27:12 GMT /qskrps-48/ 2023-05-24T21:27:12.892 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:8TAMkXi8EVUV3ltARiym9EgqjRs= 2023-05-24T21:27:12.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:27:12 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:8TAMkXi8EVUV3ltARiym9EgqjRs=', 'Content-Length': '0'} 2023-05-24T21:27:12.894 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011d4f4d520d81882-00646e8130-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:27:12.895 DEBUG:boto:b'qskrps-48obj41000falseobj42023-05-24T21:27:06.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:27:12.895 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qskrps-48 zones={a2, a1} 2023-05-24T21:27:12.896 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=qskrps-49 2023-05-24T21:27:12.896 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:27:12.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:27:12.897 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-49 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:27:12.955 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:27:13.080 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:27:13.082 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:27:12.994+0000 7f463962b780 1 waiting to reach incremental sync.. 2023-05-24T21:27:18.000 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:27:17.994+0000 7f463962b780 1 waiting to reach incremental sync.. 2023-05-24T21:27:23.000 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:27:22.995+0000 7f463962b780 1 waiting to reach incremental sync.. 2023-05-24T21:27:28.001 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:27:27.994+0000 7f463962b780 1 waiting to reach incremental sync.. 2023-05-24T21:27:33.003 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:27:32.996+0000 7f463962b780 1 bucket sync caught up with source: 2023-05-24T21:27:33.003 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [00000000001.22.6, , 00000000001.28.6, , 00000000001.57.6, , , 00000000001.23.6, , , ] 2023-05-24T21:27:33.003 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [00000000001.22.6, , 00000000001.28.6, , 00000000001.57.6, , , 00000000001.23.6, , , ] 2023-05-24T21:27:33.003 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:27:32.996+0000 7f463962b780 0 bucket checkpoint complete 2023-05-24T21:27:33.026 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=qskrps-49 2023-05-24T21:27:33.027 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:27:33.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:27:33.028 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-49 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:27:33.090 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:27:33.090 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:27:33.142 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:27:33.136+0000 7f3487ddc780 1 waiting to reach incremental sync.. 2023-05-24T21:27:38.143 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:27:38.137+0000 7f3487ddc780 1 waiting to reach incremental sync.. 2023-05-24T21:27:43.144 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:27:43.138+0000 7f3487ddc780 1 waiting to reach incremental sync.. 2023-05-24T21:27:48.145 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:27:48.139+0000 7f3487ddc780 1 waiting to reach incremental sync.. 2023-05-24T21:27:53.145 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:27:53.139+0000 7f3487ddc780 1 waiting to reach incremental sync.. 2023-05-24T21:27:58.146 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:27:58.140+0000 7f3487ddc780 1 waiting to reach incremental sync.. 2023-05-24T21:28:03.148 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:28:03.142+0000 7f3487ddc780 1 bucket sync caught up with source: 2023-05-24T21:28:03.148 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [00000000001.361.6, , 00000000001.532.6, , 00000000001.553.6, , , 00000000001.1283.6, , , ] 2023-05-24T21:28:03.148 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [00000000001.361.6, , 00000000001.532.6, , 00000000001.553.6, , , 00000000001.1283.6, , , ] 2023-05-24T21:28:03.149 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:28:03.142+0000 7f3487ddc780 0 bucket checkpoint complete 2023-05-24T21:28:03.165 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qskrps-49 zones={a2, a1} 2023-05-24T21:28:03.166 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:03.166 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:03.167 DEBUG:boto:Method: HEAD 2023-05-24T21:28:03.167 DEBUG:boto:Path: /qskrps-49/ 2023-05-24T21:28:03.167 DEBUG:boto:Data: 2023-05-24T21:28:03.168 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.168 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:03.168 DEBUG:boto:Port: 8001 2023-05-24T21:28:03.169 DEBUG:boto:Params: {} 2023-05-24T21:28:03.169 DEBUG:boto:Token: None 2023-05-24T21:28:03.170 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:03 GMT /qskrps-49/ 2023-05-24T21:28:03.170 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:584i8H2YJi0VrhJfUFIUuSkProw= 2023-05-24T21:28:03.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:584i8H2YJi0VrhJfUFIUuSkProw=', 'Content-Length': '0'} 2023-05-24T21:28:03.173 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a8bca81cd091e801-00646e8163-1625-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.174 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:03.174 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:03.174 DEBUG:boto:Method: HEAD 2023-05-24T21:28:03.175 DEBUG:boto:Path: /qskrps-49/ 2023-05-24T21:28:03.175 DEBUG:boto:Data: 2023-05-24T21:28:03.176 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.176 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:03.176 DEBUG:boto:Port: 8000 2023-05-24T21:28:03.177 DEBUG:boto:Params: {} 2023-05-24T21:28:03.177 DEBUG:boto:Token: None 2023-05-24T21:28:03.178 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:03 GMT /qskrps-49/ 2023-05-24T21:28:03.178 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:584i8H2YJi0VrhJfUFIUuSkProw= 2023-05-24T21:28:03.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:584i8H2YJi0VrhJfUFIUuSkProw=', 'Content-Length': '0'} 2023-05-24T21:28:03.181 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003af8dfdfe6811689-00646e8163-1627-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.181 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-24T21:28:03.182 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:03.182 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:03.183 DEBUG:boto:path=/qskrps-49/?versions&max-keys=999 2023-05-24T21:28:03.183 DEBUG:boto:auth_path=/qskrps-49/?versions&max-keys=999 2023-05-24T21:28:03.183 DEBUG:boto:Method: GET 2023-05-24T21:28:03.184 DEBUG:boto:Path: /qskrps-49/?versions&max-keys=999 2023-05-24T21:28:03.184 DEBUG:boto:Data: 2023-05-24T21:28:03.184 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.185 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:03.185 DEBUG:boto:Port: 8001 2023-05-24T21:28:03.185 DEBUG:boto:Params: {} 2023-05-24T21:28:03.186 DEBUG:boto:Token: None 2023-05-24T21:28:03.186 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:03 GMT /qskrps-49/?versions 2023-05-24T21:28:03.187 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:jqyhtt2J71wjMyfWOe2aDhw05eM= 2023-05-24T21:28:03.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:jqyhtt2J71wjMyfWOe2aDhw05eM=', 'Content-Length': '0'} 2023-05-24T21:28:03.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008d3e699fc41c138-00646e8163-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.190 DEBUG:boto:b'qskrps-49999falseobj1nulltrue2023-05-24T21:27:06.420Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-24T21:27:06.438Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-24T21:27:06.495Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-24T21:27:06.514Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:03.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-05-24T21:28:03.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-05-24T21:28:03.192 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-05-24T21:28:03.192 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-05-24T21:28:03.192 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-24T21:28:03.193 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:03.193 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:03.193 DEBUG:boto:path=/qskrps-49/?versions&max-keys=999 2023-05-24T21:28:03.194 DEBUG:boto:auth_path=/qskrps-49/?versions&max-keys=999 2023-05-24T21:28:03.194 DEBUG:boto:Method: GET 2023-05-24T21:28:03.194 DEBUG:boto:Path: /qskrps-49/?versions&max-keys=999 2023-05-24T21:28:03.195 DEBUG:boto:Data: 2023-05-24T21:28:03.195 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.195 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:03.196 DEBUG:boto:Port: 8000 2023-05-24T21:28:03.196 DEBUG:boto:Params: {} 2023-05-24T21:28:03.196 DEBUG:boto:Token: None 2023-05-24T21:28:03.197 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:03 GMT /qskrps-49/?versions 2023-05-24T21:28:03.197 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:jqyhtt2J71wjMyfWOe2aDhw05eM= 2023-05-24T21:28:03.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:jqyhtt2J71wjMyfWOe2aDhw05eM=', 'Content-Length': '0'} 2023-05-24T21:28:03.200 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db7149d9e78d5898-00646e8163-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.200 DEBUG:boto:b'qskrps-49999falseobj1nulltrue2023-05-24T21:27:06.420Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-24T21:27:06.438Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-24T21:27:06.495Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-24T21:27:06.514Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:03.201 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-05-24T21:28:03.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-05-24T21:28:03.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-05-24T21:28:03.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-05-24T21:28:03.203 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:03.203 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:03.203 DEBUG:boto:path=/qskrps-49/?versions&max-keys=999 2023-05-24T21:28:03.204 DEBUG:boto:auth_path=/qskrps-49/?versions&max-keys=999 2023-05-24T21:28:03.204 DEBUG:boto:Method: GET 2023-05-24T21:28:03.204 DEBUG:boto:Path: /qskrps-49/?versions&max-keys=999 2023-05-24T21:28:03.205 DEBUG:boto:Data: 2023-05-24T21:28:03.205 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.206 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:03.206 DEBUG:boto:Port: 8001 2023-05-24T21:28:03.207 DEBUG:boto:Params: {} 2023-05-24T21:28:03.207 DEBUG:boto:Token: None 2023-05-24T21:28:03.208 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:03 GMT /qskrps-49/?versions 2023-05-24T21:28:03.208 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:jqyhtt2J71wjMyfWOe2aDhw05eM= 2023-05-24T21:28:03.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:jqyhtt2J71wjMyfWOe2aDhw05eM=', 'Content-Length': '0'} 2023-05-24T21:28:03.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d2d28c7eaed7792-00646e8163-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.211 DEBUG:boto:b'qskrps-49999falseobj1nulltrue2023-05-24T21:27:06.420Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-24T21:27:06.438Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-24T21:27:06.495Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-24T21:27:06.514Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:03.212 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:03.213 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:03.213 DEBUG:boto:path=/qskrps-49/?versions&max-keys=999 2023-05-24T21:28:03.214 DEBUG:boto:auth_path=/qskrps-49/?versions&max-keys=999 2023-05-24T21:28:03.214 DEBUG:boto:Method: GET 2023-05-24T21:28:03.214 DEBUG:boto:Path: /qskrps-49/?versions&max-keys=999 2023-05-24T21:28:03.215 DEBUG:boto:Data: 2023-05-24T21:28:03.215 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.215 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:03.216 DEBUG:boto:Port: 8000 2023-05-24T21:28:03.216 DEBUG:boto:Params: {} 2023-05-24T21:28:03.217 DEBUG:boto:Token: None 2023-05-24T21:28:03.217 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:03 GMT /qskrps-49/?versions 2023-05-24T21:28:03.217 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:jqyhtt2J71wjMyfWOe2aDhw05eM= 2023-05-24T21:28:03.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:jqyhtt2J71wjMyfWOe2aDhw05eM=', 'Content-Length': '0'} 2023-05-24T21:28:03.220 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064c4c467d1636c05-00646e8163-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.221 DEBUG:boto:b'qskrps-49999falseobj1nulltrue2023-05-24T21:27:06.420Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-24T21:27:06.438Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-24T21:27:06.495Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-24T21:27:06.514Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:03.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-24T21:28:03.222 DEBUG:boto:path=/qskrps-49/obj1 2023-05-24T21:28:03.222 DEBUG:boto:auth_path=/qskrps-49/obj1 2023-05-24T21:28:03.223 DEBUG:boto:path=/qskrps-49/obj1?versionId=null 2023-05-24T21:28:03.223 DEBUG:boto:auth_path=/qskrps-49/obj1?versionId=null 2023-05-24T21:28:03.223 DEBUG:boto:Method: GET 2023-05-24T21:28:03.224 DEBUG:boto:Path: /qskrps-49/obj1?versionId=null 2023-05-24T21:28:03.224 DEBUG:boto:Data: 2023-05-24T21:28:03.225 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.225 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:03.225 DEBUG:boto:Port: 8001 2023-05-24T21:28:03.226 DEBUG:boto:Params: {} 2023-05-24T21:28:03.226 DEBUG:boto:Token: None 2023-05-24T21:28:03.227 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:03 GMT /qskrps-49/obj1?versionId=null 2023-05-24T21:28:03.227 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:UjfSlSRT4geE6XGcitrjwT3jO9Y= 2023-05-24T21:28:03.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:UjfSlSRT4geE6XGcitrjwT3jO9Y=', 'Content-Length': '0'} 2023-05-24T21:28:03.229 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a129f1878cbada2b-00646e8163-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.272 DEBUG:boto:path=/qskrps-49/obj1 2023-05-24T21:28:03.273 DEBUG:boto:auth_path=/qskrps-49/obj1 2023-05-24T21:28:03.273 DEBUG:boto:path=/qskrps-49/obj1?versionId=null 2023-05-24T21:28:03.274 DEBUG:boto:auth_path=/qskrps-49/obj1?versionId=null 2023-05-24T21:28:03.274 DEBUG:boto:Method: GET 2023-05-24T21:28:03.275 DEBUG:boto:Path: /qskrps-49/obj1?versionId=null 2023-05-24T21:28:03.275 DEBUG:boto:Data: 2023-05-24T21:28:03.276 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.276 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:03.276 DEBUG:boto:Port: 8000 2023-05-24T21:28:03.277 DEBUG:boto:Params: {} 2023-05-24T21:28:03.277 DEBUG:boto:Token: None 2023-05-24T21:28:03.278 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:03 GMT /qskrps-49/obj1?versionId=null 2023-05-24T21:28:03.278 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:UjfSlSRT4geE6XGcitrjwT3jO9Y= 2023-05-24T21:28:03.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:UjfSlSRT4geE6XGcitrjwT3jO9Y=', 'Content-Length': '0'} 2023-05-24T21:28:03.281 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003572580bf61c5196-00646e8163-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.282 DEBUG:boto:path=/qskrps-49/obj1 2023-05-24T21:28:03.282 DEBUG:boto:auth_path=/qskrps-49/obj1 2023-05-24T21:28:03.282 DEBUG:boto:path=/qskrps-49/obj1?versionId=null 2023-05-24T21:28:03.283 DEBUG:boto:auth_path=/qskrps-49/obj1?versionId=null 2023-05-24T21:28:03.283 DEBUG:boto:Method: HEAD 2023-05-24T21:28:03.284 DEBUG:boto:Path: /qskrps-49/obj1?versionId=null 2023-05-24T21:28:03.284 DEBUG:boto:Data: 2023-05-24T21:28:03.284 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.285 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:03.285 DEBUG:boto:Port: 8001 2023-05-24T21:28:03.285 DEBUG:boto:Params: {} 2023-05-24T21:28:03.286 DEBUG:boto:Token: None 2023-05-24T21:28:03.286 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:03 GMT /qskrps-49/obj1?versionId=null 2023-05-24T21:28:03.287 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:LKsDeJpEOYj0w695r7U3c+hLroU= 2023-05-24T21:28:03.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:LKsDeJpEOYj0w695r7U3c+hLroU=', 'Content-Length': '0'} 2023-05-24T21:28:03.289 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000085b11cfd470a2cce-00646e8163-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.290 DEBUG:boto:path=/qskrps-49/obj1 2023-05-24T21:28:03.290 DEBUG:boto:auth_path=/qskrps-49/obj1 2023-05-24T21:28:03.291 DEBUG:boto:path=/qskrps-49/obj1?versionId=null 2023-05-24T21:28:03.291 DEBUG:boto:auth_path=/qskrps-49/obj1?versionId=null 2023-05-24T21:28:03.291 DEBUG:boto:Method: HEAD 2023-05-24T21:28:03.292 DEBUG:boto:Path: /qskrps-49/obj1?versionId=null 2023-05-24T21:28:03.292 DEBUG:boto:Data: 2023-05-24T21:28:03.292 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.293 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:03.293 DEBUG:boto:Port: 8000 2023-05-24T21:28:03.293 DEBUG:boto:Params: {} 2023-05-24T21:28:03.294 DEBUG:boto:Token: None 2023-05-24T21:28:03.294 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:03 GMT /qskrps-49/obj1?versionId=null 2023-05-24T21:28:03.295 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:LKsDeJpEOYj0w695r7U3c+hLroU= 2023-05-24T21:28:03.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:LKsDeJpEOYj0w695r7U3c+hLroU=', 'Content-Length': '0'} 2023-05-24T21:28:03.297 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009927125643aff924-00646e8163-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-24T21:28:03.298 DEBUG:boto:path=/qskrps-49/obj1 2023-05-24T21:28:03.298 DEBUG:boto:auth_path=/qskrps-49/obj1 2023-05-24T21:28:03.298 DEBUG:boto:path=/qskrps-49/obj1?versionId=null 2023-05-24T21:28:03.299 DEBUG:boto:auth_path=/qskrps-49/obj1?versionId=null 2023-05-24T21:28:03.299 DEBUG:boto:Method: GET 2023-05-24T21:28:03.300 DEBUG:boto:Path: /qskrps-49/obj1?versionId=null 2023-05-24T21:28:03.300 DEBUG:boto:Data: 2023-05-24T21:28:03.300 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.301 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:03.301 DEBUG:boto:Port: 8001 2023-05-24T21:28:03.301 DEBUG:boto:Params: {} 2023-05-24T21:28:03.302 DEBUG:boto:Token: None 2023-05-24T21:28:03.302 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:03 GMT /qskrps-49/obj1?versionId=null 2023-05-24T21:28:03.303 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:UjfSlSRT4geE6XGcitrjwT3jO9Y= 2023-05-24T21:28:03.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:UjfSlSRT4geE6XGcitrjwT3jO9Y=', 'Content-Length': '0'} 2023-05-24T21:28:03.305 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002ac32a01bae99817-00646e8163-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.305 DEBUG:boto:path=/qskrps-49/obj1 2023-05-24T21:28:03.306 DEBUG:boto:auth_path=/qskrps-49/obj1 2023-05-24T21:28:03.306 DEBUG:boto:path=/qskrps-49/obj1?versionId=null 2023-05-24T21:28:03.307 DEBUG:boto:auth_path=/qskrps-49/obj1?versionId=null 2023-05-24T21:28:03.307 DEBUG:boto:Method: GET 2023-05-24T21:28:03.307 DEBUG:boto:Path: /qskrps-49/obj1?versionId=null 2023-05-24T21:28:03.308 DEBUG:boto:Data: 2023-05-24T21:28:03.308 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.308 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:03.309 DEBUG:boto:Port: 8000 2023-05-24T21:28:03.309 DEBUG:boto:Params: {} 2023-05-24T21:28:03.310 DEBUG:boto:Token: None 2023-05-24T21:28:03.310 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:03 GMT /qskrps-49/obj1?versionId=null 2023-05-24T21:28:03.310 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:UjfSlSRT4geE6XGcitrjwT3jO9Y= 2023-05-24T21:28:03.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:UjfSlSRT4geE6XGcitrjwT3jO9Y=', 'Content-Length': '0'} 2023-05-24T21:28:03.313 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b288a843099db0af-00646e8163-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.313 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:03.314 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:03.314 DEBUG:boto:path=/qskrps-49/?prefix=obj1 2023-05-24T21:28:03.315 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj1 2023-05-24T21:28:03.315 DEBUG:boto:Method: GET 2023-05-24T21:28:03.316 DEBUG:boto:Path: /qskrps-49/?prefix=obj1 2023-05-24T21:28:03.316 DEBUG:boto:Data: 2023-05-24T21:28:03.316 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.317 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:03.317 DEBUG:boto:Port: 8001 2023-05-24T21:28:03.317 DEBUG:boto:Params: {} 2023-05-24T21:28:03.318 DEBUG:boto:Token: None 2023-05-24T21:28:03.318 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:03 GMT /qskrps-49/ 2023-05-24T21:28:03.319 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:/UdLQCNOgXdcB4jQAdZKvD0AqBI= 2023-05-24T21:28:03.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:/UdLQCNOgXdcB4jQAdZKvD0AqBI=', 'Content-Length': '0'} 2023-05-24T21:28:03.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000340d7b1a71e4ddd9-00646e8163-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.364 DEBUG:boto:b'qskrps-49obj11000falseobj12023-05-24T21:27:06.420Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:03.366 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:03.366 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:03.367 DEBUG:boto:path=/qskrps-49/?prefix=obj1 2023-05-24T21:28:03.367 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj1 2023-05-24T21:28:03.367 DEBUG:boto:Method: GET 2023-05-24T21:28:03.368 DEBUG:boto:Path: /qskrps-49/?prefix=obj1 2023-05-24T21:28:03.368 DEBUG:boto:Data: 2023-05-24T21:28:03.368 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.369 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:03.369 DEBUG:boto:Port: 8000 2023-05-24T21:28:03.370 DEBUG:boto:Params: {} 2023-05-24T21:28:03.370 DEBUG:boto:Token: None 2023-05-24T21:28:03.371 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:03 GMT /qskrps-49/ 2023-05-24T21:28:03.371 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:/UdLQCNOgXdcB4jQAdZKvD0AqBI= 2023-05-24T21:28:03.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:/UdLQCNOgXdcB4jQAdZKvD0AqBI=', 'Content-Length': '0'} 2023-05-24T21:28:03.374 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd4abc0275d3762b-00646e8163-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.375 DEBUG:boto:b'qskrps-49obj11000falseobj12023-05-24T21:27:06.420Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:03.376 DEBUG:boto:path=/qskrps-49/obj1 2023-05-24T21:28:03.376 DEBUG:boto:auth_path=/qskrps-49/obj1 2023-05-24T21:28:03.376 DEBUG:boto:Method: HEAD 2023-05-24T21:28:03.377 DEBUG:boto:Path: /qskrps-49/obj1 2023-05-24T21:28:03.377 DEBUG:boto:Data: 2023-05-24T21:28:03.377 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.378 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:03.378 DEBUG:boto:Port: 8001 2023-05-24T21:28:03.378 DEBUG:boto:Params: {} 2023-05-24T21:28:03.379 DEBUG:boto:Token: None 2023-05-24T21:28:03.379 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:03 GMT /qskrps-49/obj1 2023-05-24T21:28:03.380 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:04h3q6hz4EtjVRLLKyZuJt+NArU= 2023-05-24T21:28:03.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:04h3q6hz4EtjVRLLKyZuJt+NArU=', 'Content-Length': '0'} 2023-05-24T21:28:03.382 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f932320ab82cf0c-00646e8163-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.382 DEBUG:boto:path=/qskrps-49/obj1 2023-05-24T21:28:03.383 DEBUG:boto:auth_path=/qskrps-49/obj1 2023-05-24T21:28:03.383 DEBUG:boto:Method: HEAD 2023-05-24T21:28:03.384 DEBUG:boto:Path: /qskrps-49/obj1 2023-05-24T21:28:03.384 DEBUG:boto:Data: 2023-05-24T21:28:03.384 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.385 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:03.385 DEBUG:boto:Port: 8000 2023-05-24T21:28:03.385 DEBUG:boto:Params: {} 2023-05-24T21:28:03.386 DEBUG:boto:Token: None 2023-05-24T21:28:03.386 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:03 GMT /qskrps-49/obj1 2023-05-24T21:28:03.386 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:04h3q6hz4EtjVRLLKyZuJt+NArU= 2023-05-24T21:28:03.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:04h3q6hz4EtjVRLLKyZuJt+NArU=', 'Content-Length': '0'} 2023-05-24T21:28:03.389 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009944f2de96387892-00646e8163-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-24T21:28:03.390 DEBUG:boto:path=/qskrps-49/obj1 2023-05-24T21:28:03.390 DEBUG:boto:auth_path=/qskrps-49/obj1 2023-05-24T21:28:03.390 DEBUG:boto:Method: GET 2023-05-24T21:28:03.391 DEBUG:boto:Path: /qskrps-49/obj1 2023-05-24T21:28:03.391 DEBUG:boto:Data: 2023-05-24T21:28:03.391 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.392 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:03.393 DEBUG:boto:Port: 8001 2023-05-24T21:28:03.393 DEBUG:boto:Params: {} 2023-05-24T21:28:03.394 DEBUG:boto:Token: None 2023-05-24T21:28:03.394 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:03 GMT /qskrps-49/obj1 2023-05-24T21:28:03.395 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:s6ZQKJxSbU7FPQ/pHYlQ98ZBs9g= 2023-05-24T21:28:03.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:s6ZQKJxSbU7FPQ/pHYlQ98ZBs9g=', 'Content-Length': '0'} 2023-05-24T21:28:03.398 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000740c6679e407cc8b-00646e8163-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.399 DEBUG:boto:path=/qskrps-49/obj1 2023-05-24T21:28:03.399 DEBUG:boto:auth_path=/qskrps-49/obj1 2023-05-24T21:28:03.400 DEBUG:boto:Method: GET 2023-05-24T21:28:03.400 DEBUG:boto:Path: /qskrps-49/obj1 2023-05-24T21:28:03.400 DEBUG:boto:Data: 2023-05-24T21:28:03.401 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.401 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:03.401 DEBUG:boto:Port: 8000 2023-05-24T21:28:03.402 DEBUG:boto:Params: {} 2023-05-24T21:28:03.402 DEBUG:boto:Token: None 2023-05-24T21:28:03.403 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:03 GMT /qskrps-49/obj1 2023-05-24T21:28:03.403 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:s6ZQKJxSbU7FPQ/pHYlQ98ZBs9g= 2023-05-24T21:28:03.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:s6ZQKJxSbU7FPQ/pHYlQ98ZBs9g=', 'Content-Length': '0'} 2023-05-24T21:28:03.406 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000032a2d6f577127ab3-00646e8163-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.448 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:03.449 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:03.450 DEBUG:boto:path=/qskrps-49/?prefix=obj1 2023-05-24T21:28:03.450 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj1 2023-05-24T21:28:03.450 DEBUG:boto:Method: GET 2023-05-24T21:28:03.451 DEBUG:boto:Path: /qskrps-49/?prefix=obj1 2023-05-24T21:28:03.451 DEBUG:boto:Data: 2023-05-24T21:28:03.451 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.452 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:03.452 DEBUG:boto:Port: 8001 2023-05-24T21:28:03.452 DEBUG:boto:Params: {} 2023-05-24T21:28:03.453 DEBUG:boto:Token: None 2023-05-24T21:28:03.453 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:03 GMT /qskrps-49/ 2023-05-24T21:28:03.454 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:/UdLQCNOgXdcB4jQAdZKvD0AqBI= 2023-05-24T21:28:03.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:/UdLQCNOgXdcB4jQAdZKvD0AqBI=', 'Content-Length': '0'} 2023-05-24T21:28:03.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096bcad211f2454b2-00646e8163-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.457 DEBUG:boto:b'qskrps-49obj11000falseobj12023-05-24T21:27:06.420Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:03.458 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:03.459 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:03.459 DEBUG:boto:path=/qskrps-49/?prefix=obj1 2023-05-24T21:28:03.459 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj1 2023-05-24T21:28:03.460 DEBUG:boto:Method: GET 2023-05-24T21:28:03.460 DEBUG:boto:Path: /qskrps-49/?prefix=obj1 2023-05-24T21:28:03.460 DEBUG:boto:Data: 2023-05-24T21:28:03.461 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.461 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:03.462 DEBUG:boto:Port: 8000 2023-05-24T21:28:03.462 DEBUG:boto:Params: {} 2023-05-24T21:28:03.462 DEBUG:boto:Token: None 2023-05-24T21:28:03.463 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:03 GMT /qskrps-49/ 2023-05-24T21:28:03.463 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:/UdLQCNOgXdcB4jQAdZKvD0AqBI= 2023-05-24T21:28:03.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:/UdLQCNOgXdcB4jQAdZKvD0AqBI=', 'Content-Length': '0'} 2023-05-24T21:28:03.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d11517b9bb402f12-00646e8163-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.467 DEBUG:boto:b'qskrps-49obj11000falseobj12023-05-24T21:27:06.420Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:03.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-24T21:28:03.468 DEBUG:boto:path=/qskrps-49/obj2 2023-05-24T21:28:03.468 DEBUG:boto:auth_path=/qskrps-49/obj2 2023-05-24T21:28:03.469 DEBUG:boto:path=/qskrps-49/obj2?versionId=null 2023-05-24T21:28:03.469 DEBUG:boto:auth_path=/qskrps-49/obj2?versionId=null 2023-05-24T21:28:03.469 DEBUG:boto:Method: GET 2023-05-24T21:28:03.470 DEBUG:boto:Path: /qskrps-49/obj2?versionId=null 2023-05-24T21:28:03.470 DEBUG:boto:Data: 2023-05-24T21:28:03.471 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.471 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:03.471 DEBUG:boto:Port: 8001 2023-05-24T21:28:03.472 DEBUG:boto:Params: {} 2023-05-24T21:28:03.472 DEBUG:boto:Token: None 2023-05-24T21:28:03.472 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:03 GMT /qskrps-49/obj2?versionId=null 2023-05-24T21:28:03.473 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:9o5LL4Ww3jp9mc7CBUtmz44V8YE= 2023-05-24T21:28:03.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:9o5LL4Ww3jp9mc7CBUtmz44V8YE=', 'Content-Length': '0'} 2023-05-24T21:28:03.475 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f0a717a9118891c-00646e8163-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.476 DEBUG:boto:path=/qskrps-49/obj2 2023-05-24T21:28:03.476 DEBUG:boto:auth_path=/qskrps-49/obj2 2023-05-24T21:28:03.477 DEBUG:boto:path=/qskrps-49/obj2?versionId=null 2023-05-24T21:28:03.477 DEBUG:boto:auth_path=/qskrps-49/obj2?versionId=null 2023-05-24T21:28:03.478 DEBUG:boto:Method: GET 2023-05-24T21:28:03.478 DEBUG:boto:Path: /qskrps-49/obj2?versionId=null 2023-05-24T21:28:03.479 DEBUG:boto:Data: 2023-05-24T21:28:03.479 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.479 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:03.480 DEBUG:boto:Port: 8000 2023-05-24T21:28:03.480 DEBUG:boto:Params: {} 2023-05-24T21:28:03.481 DEBUG:boto:Token: None 2023-05-24T21:28:03.481 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:03 GMT /qskrps-49/obj2?versionId=null 2023-05-24T21:28:03.481 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:9o5LL4Ww3jp9mc7CBUtmz44V8YE= 2023-05-24T21:28:03.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:9o5LL4Ww3jp9mc7CBUtmz44V8YE=', 'Content-Length': '0'} 2023-05-24T21:28:03.484 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e638f5d7b250cd1-00646e8163-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.485 DEBUG:boto:path=/qskrps-49/obj2 2023-05-24T21:28:03.485 DEBUG:boto:auth_path=/qskrps-49/obj2 2023-05-24T21:28:03.486 DEBUG:boto:path=/qskrps-49/obj2?versionId=null 2023-05-24T21:28:03.486 DEBUG:boto:auth_path=/qskrps-49/obj2?versionId=null 2023-05-24T21:28:03.486 DEBUG:boto:Method: HEAD 2023-05-24T21:28:03.487 DEBUG:boto:Path: /qskrps-49/obj2?versionId=null 2023-05-24T21:28:03.487 DEBUG:boto:Data: 2023-05-24T21:28:03.487 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.487 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:03.488 DEBUG:boto:Port: 8001 2023-05-24T21:28:03.488 DEBUG:boto:Params: {} 2023-05-24T21:28:03.489 DEBUG:boto:Token: None 2023-05-24T21:28:03.489 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:03 GMT /qskrps-49/obj2?versionId=null 2023-05-24T21:28:03.489 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:dWEJ/6SK5O8C8MVpJI9j+/Oyo+I= 2023-05-24T21:28:03.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:dWEJ/6SK5O8C8MVpJI9j+/Oyo+I=', 'Content-Length': '0'} 2023-05-24T21:28:03.492 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001032850308ab4e21-00646e8163-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.492 DEBUG:boto:path=/qskrps-49/obj2 2023-05-24T21:28:03.492 DEBUG:boto:auth_path=/qskrps-49/obj2 2023-05-24T21:28:03.493 DEBUG:boto:path=/qskrps-49/obj2?versionId=null 2023-05-24T21:28:03.493 DEBUG:boto:auth_path=/qskrps-49/obj2?versionId=null 2023-05-24T21:28:03.493 DEBUG:boto:Method: HEAD 2023-05-24T21:28:03.494 DEBUG:boto:Path: /qskrps-49/obj2?versionId=null 2023-05-24T21:28:03.494 DEBUG:boto:Data: 2023-05-24T21:28:03.495 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.495 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:03.495 DEBUG:boto:Port: 8000 2023-05-24T21:28:03.496 DEBUG:boto:Params: {} 2023-05-24T21:28:03.496 DEBUG:boto:Token: None 2023-05-24T21:28:03.496 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:03 GMT /qskrps-49/obj2?versionId=null 2023-05-24T21:28:03.497 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:dWEJ/6SK5O8C8MVpJI9j+/Oyo+I= 2023-05-24T21:28:03.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:dWEJ/6SK5O8C8MVpJI9j+/Oyo+I=', 'Content-Length': '0'} 2023-05-24T21:28:03.499 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e0ca0bc05e6e5e35-00646e8163-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-24T21:28:03.500 DEBUG:boto:path=/qskrps-49/obj2 2023-05-24T21:28:03.500 DEBUG:boto:auth_path=/qskrps-49/obj2 2023-05-24T21:28:03.501 DEBUG:boto:path=/qskrps-49/obj2?versionId=null 2023-05-24T21:28:03.501 DEBUG:boto:auth_path=/qskrps-49/obj2?versionId=null 2023-05-24T21:28:03.501 DEBUG:boto:Method: GET 2023-05-24T21:28:03.502 DEBUG:boto:Path: /qskrps-49/obj2?versionId=null 2023-05-24T21:28:03.502 DEBUG:boto:Data: 2023-05-24T21:28:03.502 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.503 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:03.503 DEBUG:boto:Port: 8001 2023-05-24T21:28:03.503 DEBUG:boto:Params: {} 2023-05-24T21:28:03.504 DEBUG:boto:Token: None 2023-05-24T21:28:03.504 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:03 GMT /qskrps-49/obj2?versionId=null 2023-05-24T21:28:03.505 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:9o5LL4Ww3jp9mc7CBUtmz44V8YE= 2023-05-24T21:28:03.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:9o5LL4Ww3jp9mc7CBUtmz44V8YE=', 'Content-Length': '0'} 2023-05-24T21:28:03.507 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000033cb2cc7f0a6023c-00646e8163-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.508 DEBUG:boto:path=/qskrps-49/obj2 2023-05-24T21:28:03.508 DEBUG:boto:auth_path=/qskrps-49/obj2 2023-05-24T21:28:03.509 DEBUG:boto:path=/qskrps-49/obj2?versionId=null 2023-05-24T21:28:03.509 DEBUG:boto:auth_path=/qskrps-49/obj2?versionId=null 2023-05-24T21:28:03.509 DEBUG:boto:Method: GET 2023-05-24T21:28:03.510 DEBUG:boto:Path: /qskrps-49/obj2?versionId=null 2023-05-24T21:28:03.510 DEBUG:boto:Data: 2023-05-24T21:28:03.510 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.511 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:03.511 DEBUG:boto:Port: 8000 2023-05-24T21:28:03.511 DEBUG:boto:Params: {} 2023-05-24T21:28:03.512 DEBUG:boto:Token: None 2023-05-24T21:28:03.512 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:03 GMT /qskrps-49/obj2?versionId=null 2023-05-24T21:28:03.513 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:9o5LL4Ww3jp9mc7CBUtmz44V8YE= 2023-05-24T21:28:03.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:9o5LL4Ww3jp9mc7CBUtmz44V8YE=', 'Content-Length': '0'} 2023-05-24T21:28:03.515 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008063183bd20dc2ee-00646e8163-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.516 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:03.516 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:03.516 DEBUG:boto:path=/qskrps-49/?prefix=obj2 2023-05-24T21:28:03.517 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj2 2023-05-24T21:28:03.517 DEBUG:boto:Method: GET 2023-05-24T21:28:03.518 DEBUG:boto:Path: /qskrps-49/?prefix=obj2 2023-05-24T21:28:03.518 DEBUG:boto:Data: 2023-05-24T21:28:03.518 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.519 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:03.519 DEBUG:boto:Port: 8001 2023-05-24T21:28:03.519 DEBUG:boto:Params: {} 2023-05-24T21:28:03.520 DEBUG:boto:Token: None 2023-05-24T21:28:03.520 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:03 GMT /qskrps-49/ 2023-05-24T21:28:03.521 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:/UdLQCNOgXdcB4jQAdZKvD0AqBI= 2023-05-24T21:28:03.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:/UdLQCNOgXdcB4jQAdZKvD0AqBI=', 'Content-Length': '0'} 2023-05-24T21:28:03.524 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005af9ac348f4c0896-00646e8163-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.525 DEBUG:boto:b'qskrps-49obj21000falseobj22023-05-24T21:27:06.438Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:03.526 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:03.526 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:03.526 DEBUG:boto:path=/qskrps-49/?prefix=obj2 2023-05-24T21:28:03.527 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj2 2023-05-24T21:28:03.527 DEBUG:boto:Method: GET 2023-05-24T21:28:03.528 DEBUG:boto:Path: /qskrps-49/?prefix=obj2 2023-05-24T21:28:03.528 DEBUG:boto:Data: 2023-05-24T21:28:03.528 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.529 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:03.529 DEBUG:boto:Port: 8000 2023-05-24T21:28:03.529 DEBUG:boto:Params: {} 2023-05-24T21:28:03.530 DEBUG:boto:Token: None 2023-05-24T21:28:03.530 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:03 GMT /qskrps-49/ 2023-05-24T21:28:03.531 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:/UdLQCNOgXdcB4jQAdZKvD0AqBI= 2023-05-24T21:28:03.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:/UdLQCNOgXdcB4jQAdZKvD0AqBI=', 'Content-Length': '0'} 2023-05-24T21:28:03.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c29538981b1b7c04-00646e8163-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.534 DEBUG:boto:b'qskrps-49obj21000falseobj22023-05-24T21:27:06.438Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:03.534 DEBUG:boto:path=/qskrps-49/obj2 2023-05-24T21:28:03.535 DEBUG:boto:auth_path=/qskrps-49/obj2 2023-05-24T21:28:03.535 DEBUG:boto:Method: HEAD 2023-05-24T21:28:03.535 DEBUG:boto:Path: /qskrps-49/obj2 2023-05-24T21:28:03.536 DEBUG:boto:Data: 2023-05-24T21:28:03.536 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.537 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:03.537 DEBUG:boto:Port: 8001 2023-05-24T21:28:03.537 DEBUG:boto:Params: {} 2023-05-24T21:28:03.538 DEBUG:boto:Token: None 2023-05-24T21:28:03.538 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:03 GMT /qskrps-49/obj2 2023-05-24T21:28:03.538 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:i0CY6BUgGcT2SoO24y6g+gY814E= 2023-05-24T21:28:03.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:i0CY6BUgGcT2SoO24y6g+gY814E=', 'Content-Length': '0'} 2023-05-24T21:28:03.540 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000002981fc4983b009c-00646e8163-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.541 DEBUG:boto:path=/qskrps-49/obj2 2023-05-24T21:28:03.541 DEBUG:boto:auth_path=/qskrps-49/obj2 2023-05-24T21:28:03.542 DEBUG:boto:Method: HEAD 2023-05-24T21:28:03.542 DEBUG:boto:Path: /qskrps-49/obj2 2023-05-24T21:28:03.542 DEBUG:boto:Data: 2023-05-24T21:28:03.543 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.543 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:03.543 DEBUG:boto:Port: 8000 2023-05-24T21:28:03.544 DEBUG:boto:Params: {} 2023-05-24T21:28:03.544 DEBUG:boto:Token: None 2023-05-24T21:28:03.545 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:03 GMT /qskrps-49/obj2 2023-05-24T21:28:03.545 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:i0CY6BUgGcT2SoO24y6g+gY814E= 2023-05-24T21:28:03.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:i0CY6BUgGcT2SoO24y6g+gY814E=', 'Content-Length': '0'} 2023-05-24T21:28:03.547 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f82ea9831e3ed992-00646e8163-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-24T21:28:03.548 DEBUG:boto:path=/qskrps-49/obj2 2023-05-24T21:28:03.549 DEBUG:boto:auth_path=/qskrps-49/obj2 2023-05-24T21:28:03.549 DEBUG:boto:Method: GET 2023-05-24T21:28:03.549 DEBUG:boto:Path: /qskrps-49/obj2 2023-05-24T21:28:03.550 DEBUG:boto:Data: 2023-05-24T21:28:03.550 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.550 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:03.551 DEBUG:boto:Port: 8001 2023-05-24T21:28:03.551 DEBUG:boto:Params: {} 2023-05-24T21:28:03.552 DEBUG:boto:Token: None 2023-05-24T21:28:03.552 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:03 GMT /qskrps-49/obj2 2023-05-24T21:28:03.552 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:IB6ShspVnCvSHR7Dk/99z5z7e8c= 2023-05-24T21:28:03.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:IB6ShspVnCvSHR7Dk/99z5z7e8c=', 'Content-Length': '0'} 2023-05-24T21:28:03.555 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000008552fde001f7f5-00646e8163-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.596 DEBUG:boto:path=/qskrps-49/obj2 2023-05-24T21:28:03.597 DEBUG:boto:auth_path=/qskrps-49/obj2 2023-05-24T21:28:03.598 DEBUG:boto:Method: GET 2023-05-24T21:28:03.598 DEBUG:boto:Path: /qskrps-49/obj2 2023-05-24T21:28:03.598 DEBUG:boto:Data: 2023-05-24T21:28:03.599 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.599 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:03.599 DEBUG:boto:Port: 8000 2023-05-24T21:28:03.600 DEBUG:boto:Params: {} 2023-05-24T21:28:03.600 DEBUG:boto:Token: None 2023-05-24T21:28:03.601 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:03 GMT /qskrps-49/obj2 2023-05-24T21:28:03.601 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:IB6ShspVnCvSHR7Dk/99z5z7e8c= 2023-05-24T21:28:03.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:IB6ShspVnCvSHR7Dk/99z5z7e8c=', 'Content-Length': '0'} 2023-05-24T21:28:03.604 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007a2d7f4f994e82b1-00646e8163-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.604 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:03.605 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:03.605 DEBUG:boto:path=/qskrps-49/?prefix=obj2 2023-05-24T21:28:03.606 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj2 2023-05-24T21:28:03.606 DEBUG:boto:Method: GET 2023-05-24T21:28:03.607 DEBUG:boto:Path: /qskrps-49/?prefix=obj2 2023-05-24T21:28:03.607 DEBUG:boto:Data: 2023-05-24T21:28:03.608 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.609 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:03.609 DEBUG:boto:Port: 8001 2023-05-24T21:28:03.610 DEBUG:boto:Params: {} 2023-05-24T21:28:03.610 DEBUG:boto:Token: None 2023-05-24T21:28:03.611 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:03 GMT /qskrps-49/ 2023-05-24T21:28:03.611 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:/UdLQCNOgXdcB4jQAdZKvD0AqBI= 2023-05-24T21:28:03.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:/UdLQCNOgXdcB4jQAdZKvD0AqBI=', 'Content-Length': '0'} 2023-05-24T21:28:03.614 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000574cd1920b73aa58-00646e8163-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.615 DEBUG:boto:b'qskrps-49obj21000falseobj22023-05-24T21:27:06.438Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:03.616 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:03.616 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:03.617 DEBUG:boto:path=/qskrps-49/?prefix=obj2 2023-05-24T21:28:03.617 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj2 2023-05-24T21:28:03.617 DEBUG:boto:Method: GET 2023-05-24T21:28:03.618 DEBUG:boto:Path: /qskrps-49/?prefix=obj2 2023-05-24T21:28:03.618 DEBUG:boto:Data: 2023-05-24T21:28:03.619 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.619 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:03.619 DEBUG:boto:Port: 8000 2023-05-24T21:28:03.620 DEBUG:boto:Params: {} 2023-05-24T21:28:03.620 DEBUG:boto:Token: None 2023-05-24T21:28:03.621 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:03 GMT /qskrps-49/ 2023-05-24T21:28:03.621 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:/UdLQCNOgXdcB4jQAdZKvD0AqBI= 2023-05-24T21:28:03.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:/UdLQCNOgXdcB4jQAdZKvD0AqBI=', 'Content-Length': '0'} 2023-05-24T21:28:03.624 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004cd25545d670cda-00646e8163-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.624 DEBUG:boto:b'qskrps-49obj21000falseobj22023-05-24T21:27:06.438Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:03.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-24T21:28:03.625 DEBUG:boto:path=/qskrps-49/obj3 2023-05-24T21:28:03.626 DEBUG:boto:auth_path=/qskrps-49/obj3 2023-05-24T21:28:03.626 DEBUG:boto:path=/qskrps-49/obj3?versionId=null 2023-05-24T21:28:03.626 DEBUG:boto:auth_path=/qskrps-49/obj3?versionId=null 2023-05-24T21:28:03.627 DEBUG:boto:Method: GET 2023-05-24T21:28:03.627 DEBUG:boto:Path: /qskrps-49/obj3?versionId=null 2023-05-24T21:28:03.628 DEBUG:boto:Data: 2023-05-24T21:28:03.628 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.628 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:03.629 DEBUG:boto:Port: 8001 2023-05-24T21:28:03.629 DEBUG:boto:Params: {} 2023-05-24T21:28:03.629 DEBUG:boto:Token: None 2023-05-24T21:28:03.630 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:03 GMT /qskrps-49/obj3?versionId=null 2023-05-24T21:28:03.630 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:VBnBlBDxf4H+IkKMTO3xNLFoFJE= 2023-05-24T21:28:03.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:VBnBlBDxf4H+IkKMTO3xNLFoFJE=', 'Content-Length': '0'} 2023-05-24T21:28:03.632 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ef6958639cb23ef8-00646e8163-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.633 DEBUG:boto:path=/qskrps-49/obj3 2023-05-24T21:28:03.633 DEBUG:boto:auth_path=/qskrps-49/obj3 2023-05-24T21:28:03.634 DEBUG:boto:path=/qskrps-49/obj3?versionId=null 2023-05-24T21:28:03.634 DEBUG:boto:auth_path=/qskrps-49/obj3?versionId=null 2023-05-24T21:28:03.634 DEBUG:boto:Method: GET 2023-05-24T21:28:03.635 DEBUG:boto:Path: /qskrps-49/obj3?versionId=null 2023-05-24T21:28:03.635 DEBUG:boto:Data: 2023-05-24T21:28:03.635 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.636 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:03.636 DEBUG:boto:Port: 8000 2023-05-24T21:28:03.637 DEBUG:boto:Params: {} 2023-05-24T21:28:03.637 DEBUG:boto:Token: None 2023-05-24T21:28:03.637 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:03 GMT /qskrps-49/obj3?versionId=null 2023-05-24T21:28:03.638 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:VBnBlBDxf4H+IkKMTO3xNLFoFJE= 2023-05-24T21:28:03.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:VBnBlBDxf4H+IkKMTO3xNLFoFJE=', 'Content-Length': '0'} 2023-05-24T21:28:03.640 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000143179e4a7ad7fe9-00646e8163-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.641 DEBUG:boto:path=/qskrps-49/obj3 2023-05-24T21:28:03.641 DEBUG:boto:auth_path=/qskrps-49/obj3 2023-05-24T21:28:03.641 DEBUG:boto:path=/qskrps-49/obj3?versionId=null 2023-05-24T21:28:03.642 DEBUG:boto:auth_path=/qskrps-49/obj3?versionId=null 2023-05-24T21:28:03.642 DEBUG:boto:Method: HEAD 2023-05-24T21:28:03.642 DEBUG:boto:Path: /qskrps-49/obj3?versionId=null 2023-05-24T21:28:03.643 DEBUG:boto:Data: 2023-05-24T21:28:03.643 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.643 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:03.644 DEBUG:boto:Port: 8001 2023-05-24T21:28:03.644 DEBUG:boto:Params: {} 2023-05-24T21:28:03.645 DEBUG:boto:Token: None 2023-05-24T21:28:03.645 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:03 GMT /qskrps-49/obj3?versionId=null 2023-05-24T21:28:03.645 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:hLO7Rt7q6R16fAipA5KWInScu9o= 2023-05-24T21:28:03.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:hLO7Rt7q6R16fAipA5KWInScu9o=', 'Content-Length': '0'} 2023-05-24T21:28:03.647 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e97b85ec33403bd8-00646e8163-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.648 DEBUG:boto:path=/qskrps-49/obj3 2023-05-24T21:28:03.648 DEBUG:boto:auth_path=/qskrps-49/obj3 2023-05-24T21:28:03.649 DEBUG:boto:path=/qskrps-49/obj3?versionId=null 2023-05-24T21:28:03.649 DEBUG:boto:auth_path=/qskrps-49/obj3?versionId=null 2023-05-24T21:28:03.649 DEBUG:boto:Method: HEAD 2023-05-24T21:28:03.650 DEBUG:boto:Path: /qskrps-49/obj3?versionId=null 2023-05-24T21:28:03.650 DEBUG:boto:Data: 2023-05-24T21:28:03.650 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.651 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:03.651 DEBUG:boto:Port: 8000 2023-05-24T21:28:03.651 DEBUG:boto:Params: {} 2023-05-24T21:28:03.652 DEBUG:boto:Token: None 2023-05-24T21:28:03.652 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:03 GMT /qskrps-49/obj3?versionId=null 2023-05-24T21:28:03.652 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:hLO7Rt7q6R16fAipA5KWInScu9o= 2023-05-24T21:28:03.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:hLO7Rt7q6R16fAipA5KWInScu9o=', 'Content-Length': '0'} 2023-05-24T21:28:03.654 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000049751d48a9d7e266-00646e8163-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-24T21:28:03.655 DEBUG:boto:path=/qskrps-49/obj3 2023-05-24T21:28:03.656 DEBUG:boto:auth_path=/qskrps-49/obj3 2023-05-24T21:28:03.656 DEBUG:boto:path=/qskrps-49/obj3?versionId=null 2023-05-24T21:28:03.657 DEBUG:boto:auth_path=/qskrps-49/obj3?versionId=null 2023-05-24T21:28:03.657 DEBUG:boto:Method: GET 2023-05-24T21:28:03.657 DEBUG:boto:Path: /qskrps-49/obj3?versionId=null 2023-05-24T21:28:03.658 DEBUG:boto:Data: 2023-05-24T21:28:03.658 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.658 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:03.659 DEBUG:boto:Port: 8001 2023-05-24T21:28:03.659 DEBUG:boto:Params: {} 2023-05-24T21:28:03.659 DEBUG:boto:Token: None 2023-05-24T21:28:03.660 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:03 GMT /qskrps-49/obj3?versionId=null 2023-05-24T21:28:03.660 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:VBnBlBDxf4H+IkKMTO3xNLFoFJE= 2023-05-24T21:28:03.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:VBnBlBDxf4H+IkKMTO3xNLFoFJE=', 'Content-Length': '0'} 2023-05-24T21:28:03.662 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000595a9820b00a3eca-00646e8163-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.663 DEBUG:boto:path=/qskrps-49/obj3 2023-05-24T21:28:03.663 DEBUG:boto:auth_path=/qskrps-49/obj3 2023-05-24T21:28:03.664 DEBUG:boto:path=/qskrps-49/obj3?versionId=null 2023-05-24T21:28:03.664 DEBUG:boto:auth_path=/qskrps-49/obj3?versionId=null 2023-05-24T21:28:03.665 DEBUG:boto:Method: GET 2023-05-24T21:28:03.665 DEBUG:boto:Path: /qskrps-49/obj3?versionId=null 2023-05-24T21:28:03.665 DEBUG:boto:Data: 2023-05-24T21:28:03.666 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.666 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:03.666 DEBUG:boto:Port: 8000 2023-05-24T21:28:03.667 DEBUG:boto:Params: {} 2023-05-24T21:28:03.667 DEBUG:boto:Token: None 2023-05-24T21:28:03.668 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:03 GMT /qskrps-49/obj3?versionId=null 2023-05-24T21:28:03.668 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:VBnBlBDxf4H+IkKMTO3xNLFoFJE= 2023-05-24T21:28:03.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:VBnBlBDxf4H+IkKMTO3xNLFoFJE=', 'Content-Length': '0'} 2023-05-24T21:28:03.670 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ebbbab1e2cd3949b-00646e8163-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.671 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:03.671 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:03.672 DEBUG:boto:path=/qskrps-49/?prefix=obj3 2023-05-24T21:28:03.672 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj3 2023-05-24T21:28:03.672 DEBUG:boto:Method: GET 2023-05-24T21:28:03.673 DEBUG:boto:Path: /qskrps-49/?prefix=obj3 2023-05-24T21:28:03.673 DEBUG:boto:Data: 2023-05-24T21:28:03.673 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.674 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:03.674 DEBUG:boto:Port: 8001 2023-05-24T21:28:03.675 DEBUG:boto:Params: {} 2023-05-24T21:28:03.675 DEBUG:boto:Token: None 2023-05-24T21:28:03.675 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:03 GMT /qskrps-49/ 2023-05-24T21:28:03.676 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:/UdLQCNOgXdcB4jQAdZKvD0AqBI= 2023-05-24T21:28:03.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:/UdLQCNOgXdcB4jQAdZKvD0AqBI=', 'Content-Length': '0'} 2023-05-24T21:28:03.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020b26e56420a8bb1-00646e8163-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.679 DEBUG:boto:b'qskrps-49obj31000falseobj32023-05-24T21:27:06.495Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:03.680 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:03.680 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:03.681 DEBUG:boto:path=/qskrps-49/?prefix=obj3 2023-05-24T21:28:03.681 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj3 2023-05-24T21:28:03.681 DEBUG:boto:Method: GET 2023-05-24T21:28:03.682 DEBUG:boto:Path: /qskrps-49/?prefix=obj3 2023-05-24T21:28:03.682 DEBUG:boto:Data: 2023-05-24T21:28:03.682 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.683 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:03.683 DEBUG:boto:Port: 8000 2023-05-24T21:28:03.683 DEBUG:boto:Params: {} 2023-05-24T21:28:03.684 DEBUG:boto:Token: None 2023-05-24T21:28:03.684 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:03 GMT /qskrps-49/ 2023-05-24T21:28:03.685 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:/UdLQCNOgXdcB4jQAdZKvD0AqBI= 2023-05-24T21:28:03.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:/UdLQCNOgXdcB4jQAdZKvD0AqBI=', 'Content-Length': '0'} 2023-05-24T21:28:03.687 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082b332177a70d2e6-00646e8163-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.688 DEBUG:boto:b'qskrps-49obj31000falseobj32023-05-24T21:27:06.495Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:03.689 DEBUG:boto:path=/qskrps-49/obj3 2023-05-24T21:28:03.689 DEBUG:boto:auth_path=/qskrps-49/obj3 2023-05-24T21:28:03.689 DEBUG:boto:Method: HEAD 2023-05-24T21:28:03.690 DEBUG:boto:Path: /qskrps-49/obj3 2023-05-24T21:28:03.690 DEBUG:boto:Data: 2023-05-24T21:28:03.690 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.691 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:03.691 DEBUG:boto:Port: 8001 2023-05-24T21:28:03.691 DEBUG:boto:Params: {} 2023-05-24T21:28:03.692 DEBUG:boto:Token: None 2023-05-24T21:28:03.692 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:03 GMT /qskrps-49/obj3 2023-05-24T21:28:03.692 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:qKbTy1Td0LNTF0ClQz+r7zgJx/8= 2023-05-24T21:28:03.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qKbTy1Td0LNTF0ClQz+r7zgJx/8=', 'Content-Length': '0'} 2023-05-24T21:28:03.695 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000072dcd67cb599b532-00646e8163-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.695 DEBUG:boto:path=/qskrps-49/obj3 2023-05-24T21:28:03.696 DEBUG:boto:auth_path=/qskrps-49/obj3 2023-05-24T21:28:03.696 DEBUG:boto:Method: HEAD 2023-05-24T21:28:03.696 DEBUG:boto:Path: /qskrps-49/obj3 2023-05-24T21:28:03.697 DEBUG:boto:Data: 2023-05-24T21:28:03.697 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.697 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:03.698 DEBUG:boto:Port: 8000 2023-05-24T21:28:03.698 DEBUG:boto:Params: {} 2023-05-24T21:28:03.698 DEBUG:boto:Token: None 2023-05-24T21:28:03.699 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:03 GMT /qskrps-49/obj3 2023-05-24T21:28:03.699 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:qKbTy1Td0LNTF0ClQz+r7zgJx/8= 2023-05-24T21:28:03.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qKbTy1Td0LNTF0ClQz+r7zgJx/8=', 'Content-Length': '0'} 2023-05-24T21:28:03.701 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005ea0bd2a1b44d572-00646e8163-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-24T21:28:03.702 DEBUG:boto:path=/qskrps-49/obj3 2023-05-24T21:28:03.703 DEBUG:boto:auth_path=/qskrps-49/obj3 2023-05-24T21:28:03.703 DEBUG:boto:Method: GET 2023-05-24T21:28:03.704 DEBUG:boto:Path: /qskrps-49/obj3 2023-05-24T21:28:03.704 DEBUG:boto:Data: 2023-05-24T21:28:03.704 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.705 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:03.705 DEBUG:boto:Port: 8001 2023-05-24T21:28:03.705 DEBUG:boto:Params: {} 2023-05-24T21:28:03.706 DEBUG:boto:Token: None 2023-05-24T21:28:03.706 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:03 GMT /qskrps-49/obj3 2023-05-24T21:28:03.707 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:8FDAwMtIFFduqrljnV94Hr3pukI= 2023-05-24T21:28:03.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:8FDAwMtIFFduqrljnV94Hr3pukI=', 'Content-Length': '0'} 2023-05-24T21:28:03.709 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ddeaec985cfde1b3-00646e8163-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.709 DEBUG:boto:path=/qskrps-49/obj3 2023-05-24T21:28:03.710 DEBUG:boto:auth_path=/qskrps-49/obj3 2023-05-24T21:28:03.710 DEBUG:boto:Method: GET 2023-05-24T21:28:03.711 DEBUG:boto:Path: /qskrps-49/obj3 2023-05-24T21:28:03.711 DEBUG:boto:Data: 2023-05-24T21:28:03.711 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.712 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:03.712 DEBUG:boto:Port: 8000 2023-05-24T21:28:03.713 DEBUG:boto:Params: {} 2023-05-24T21:28:03.713 DEBUG:boto:Token: None 2023-05-24T21:28:03.714 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:03 GMT /qskrps-49/obj3 2023-05-24T21:28:03.714 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:8FDAwMtIFFduqrljnV94Hr3pukI= 2023-05-24T21:28:03.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:8FDAwMtIFFduqrljnV94Hr3pukI=', 'Content-Length': '0'} 2023-05-24T21:28:03.717 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000086ec5e0c7475eb0d-00646e8163-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.718 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:03.718 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:03.719 DEBUG:boto:path=/qskrps-49/?prefix=obj3 2023-05-24T21:28:03.719 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj3 2023-05-24T21:28:03.719 DEBUG:boto:Method: GET 2023-05-24T21:28:03.720 DEBUG:boto:Path: /qskrps-49/?prefix=obj3 2023-05-24T21:28:03.720 DEBUG:boto:Data: 2023-05-24T21:28:03.721 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.721 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:03.721 DEBUG:boto:Port: 8001 2023-05-24T21:28:03.722 DEBUG:boto:Params: {} 2023-05-24T21:28:03.722 DEBUG:boto:Token: None 2023-05-24T21:28:03.723 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:03 GMT /qskrps-49/ 2023-05-24T21:28:03.723 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:/UdLQCNOgXdcB4jQAdZKvD0AqBI= 2023-05-24T21:28:03.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:/UdLQCNOgXdcB4jQAdZKvD0AqBI=', 'Content-Length': '0'} 2023-05-24T21:28:03.726 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7cb5127cb500957-00646e8163-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.772 DEBUG:boto:b'qskrps-49obj31000falseobj32023-05-24T21:27:06.495Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:03.773 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:03.774 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:03.774 DEBUG:boto:path=/qskrps-49/?prefix=obj3 2023-05-24T21:28:03.775 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj3 2023-05-24T21:28:03.775 DEBUG:boto:Method: GET 2023-05-24T21:28:03.775 DEBUG:boto:Path: /qskrps-49/?prefix=obj3 2023-05-24T21:28:03.776 DEBUG:boto:Data: 2023-05-24T21:28:03.776 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.776 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:03.777 DEBUG:boto:Port: 8000 2023-05-24T21:28:03.777 DEBUG:boto:Params: {} 2023-05-24T21:28:03.778 DEBUG:boto:Token: None 2023-05-24T21:28:03.778 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:03 GMT /qskrps-49/ 2023-05-24T21:28:03.779 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:/UdLQCNOgXdcB4jQAdZKvD0AqBI= 2023-05-24T21:28:03.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:/UdLQCNOgXdcB4jQAdZKvD0AqBI=', 'Content-Length': '0'} 2023-05-24T21:28:03.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070595c3fb40ae456-00646e8163-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.782 DEBUG:boto:b'qskrps-49obj31000falseobj32023-05-24T21:27:06.495Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:03.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-24T21:28:03.784 DEBUG:boto:path=/qskrps-49/obj4 2023-05-24T21:28:03.784 DEBUG:boto:auth_path=/qskrps-49/obj4 2023-05-24T21:28:03.784 DEBUG:boto:path=/qskrps-49/obj4?versionId=null 2023-05-24T21:28:03.785 DEBUG:boto:auth_path=/qskrps-49/obj4?versionId=null 2023-05-24T21:28:03.785 DEBUG:boto:Method: GET 2023-05-24T21:28:03.785 DEBUG:boto:Path: /qskrps-49/obj4?versionId=null 2023-05-24T21:28:03.786 DEBUG:boto:Data: 2023-05-24T21:28:03.786 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.787 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:03.787 DEBUG:boto:Port: 8001 2023-05-24T21:28:03.787 DEBUG:boto:Params: {} 2023-05-24T21:28:03.788 DEBUG:boto:Token: None 2023-05-24T21:28:03.788 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:03 GMT /qskrps-49/obj4?versionId=null 2023-05-24T21:28:03.789 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:rbR21sDnRpfc87nMV/ixp3VYZAg= 2023-05-24T21:28:03.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:rbR21sDnRpfc87nMV/ixp3VYZAg=', 'Content-Length': '0'} 2023-05-24T21:28:03.791 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000025cd99ada312deb9-00646e8163-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.792 DEBUG:boto:path=/qskrps-49/obj4 2023-05-24T21:28:03.792 DEBUG:boto:auth_path=/qskrps-49/obj4 2023-05-24T21:28:03.792 DEBUG:boto:path=/qskrps-49/obj4?versionId=null 2023-05-24T21:28:03.793 DEBUG:boto:auth_path=/qskrps-49/obj4?versionId=null 2023-05-24T21:28:03.793 DEBUG:boto:Method: GET 2023-05-24T21:28:03.793 DEBUG:boto:Path: /qskrps-49/obj4?versionId=null 2023-05-24T21:28:03.794 DEBUG:boto:Data: 2023-05-24T21:28:03.794 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.794 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:03.795 DEBUG:boto:Port: 8000 2023-05-24T21:28:03.795 DEBUG:boto:Params: {} 2023-05-24T21:28:03.796 DEBUG:boto:Token: None 2023-05-24T21:28:03.796 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:03 GMT /qskrps-49/obj4?versionId=null 2023-05-24T21:28:03.796 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:rbR21sDnRpfc87nMV/ixp3VYZAg= 2023-05-24T21:28:03.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:rbR21sDnRpfc87nMV/ixp3VYZAg=', 'Content-Length': '0'} 2023-05-24T21:28:03.799 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005b661da4acb1650b-00646e8163-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.799 DEBUG:boto:path=/qskrps-49/obj4 2023-05-24T21:28:03.800 DEBUG:boto:auth_path=/qskrps-49/obj4 2023-05-24T21:28:03.800 DEBUG:boto:path=/qskrps-49/obj4?versionId=null 2023-05-24T21:28:03.801 DEBUG:boto:auth_path=/qskrps-49/obj4?versionId=null 2023-05-24T21:28:03.801 DEBUG:boto:Method: HEAD 2023-05-24T21:28:03.801 DEBUG:boto:Path: /qskrps-49/obj4?versionId=null 2023-05-24T21:28:03.802 DEBUG:boto:Data: 2023-05-24T21:28:03.802 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.802 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:03.803 DEBUG:boto:Port: 8001 2023-05-24T21:28:03.803 DEBUG:boto:Params: {} 2023-05-24T21:28:03.803 DEBUG:boto:Token: None 2023-05-24T21:28:03.804 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:03 GMT /qskrps-49/obj4?versionId=null 2023-05-24T21:28:03.804 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ERgOy0/BjdvtzUAQhyq/3re3c4I= 2023-05-24T21:28:03.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ERgOy0/BjdvtzUAQhyq/3re3c4I=', 'Content-Length': '0'} 2023-05-24T21:28:03.806 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005994c62106658b05-00646e8163-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.807 DEBUG:boto:path=/qskrps-49/obj4 2023-05-24T21:28:03.807 DEBUG:boto:auth_path=/qskrps-49/obj4 2023-05-24T21:28:03.807 DEBUG:boto:path=/qskrps-49/obj4?versionId=null 2023-05-24T21:28:03.808 DEBUG:boto:auth_path=/qskrps-49/obj4?versionId=null 2023-05-24T21:28:03.808 DEBUG:boto:Method: HEAD 2023-05-24T21:28:03.809 DEBUG:boto:Path: /qskrps-49/obj4?versionId=null 2023-05-24T21:28:03.809 DEBUG:boto:Data: 2023-05-24T21:28:03.809 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.810 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:03.810 DEBUG:boto:Port: 8000 2023-05-24T21:28:03.810 DEBUG:boto:Params: {} 2023-05-24T21:28:03.811 DEBUG:boto:Token: None 2023-05-24T21:28:03.811 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:03 GMT /qskrps-49/obj4?versionId=null 2023-05-24T21:28:03.811 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ERgOy0/BjdvtzUAQhyq/3re3c4I= 2023-05-24T21:28:03.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ERgOy0/BjdvtzUAQhyq/3re3c4I=', 'Content-Length': '0'} 2023-05-24T21:28:03.814 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000010535eef61ae522-00646e8163-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-24T21:28:03.815 DEBUG:boto:path=/qskrps-49/obj4 2023-05-24T21:28:03.815 DEBUG:boto:auth_path=/qskrps-49/obj4 2023-05-24T21:28:03.815 DEBUG:boto:path=/qskrps-49/obj4?versionId=null 2023-05-24T21:28:03.816 DEBUG:boto:auth_path=/qskrps-49/obj4?versionId=null 2023-05-24T21:28:03.816 DEBUG:boto:Method: GET 2023-05-24T21:28:03.816 DEBUG:boto:Path: /qskrps-49/obj4?versionId=null 2023-05-24T21:28:03.817 DEBUG:boto:Data: 2023-05-24T21:28:03.817 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.818 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:03.818 DEBUG:boto:Port: 8001 2023-05-24T21:28:03.818 DEBUG:boto:Params: {} 2023-05-24T21:28:03.819 DEBUG:boto:Token: None 2023-05-24T21:28:03.819 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:03 GMT /qskrps-49/obj4?versionId=null 2023-05-24T21:28:03.819 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:rbR21sDnRpfc87nMV/ixp3VYZAg= 2023-05-24T21:28:03.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:rbR21sDnRpfc87nMV/ixp3VYZAg=', 'Content-Length': '0'} 2023-05-24T21:28:03.822 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000770a41f228b7e9f2-00646e8163-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.822 DEBUG:boto:path=/qskrps-49/obj4 2023-05-24T21:28:03.823 DEBUG:boto:auth_path=/qskrps-49/obj4 2023-05-24T21:28:03.823 DEBUG:boto:path=/qskrps-49/obj4?versionId=null 2023-05-24T21:28:03.823 DEBUG:boto:auth_path=/qskrps-49/obj4?versionId=null 2023-05-24T21:28:03.824 DEBUG:boto:Method: GET 2023-05-24T21:28:03.824 DEBUG:boto:Path: /qskrps-49/obj4?versionId=null 2023-05-24T21:28:03.825 DEBUG:boto:Data: 2023-05-24T21:28:03.825 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.825 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:03.826 DEBUG:boto:Port: 8000 2023-05-24T21:28:03.826 DEBUG:boto:Params: {} 2023-05-24T21:28:03.826 DEBUG:boto:Token: None 2023-05-24T21:28:03.827 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:03 GMT /qskrps-49/obj4?versionId=null 2023-05-24T21:28:03.827 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:rbR21sDnRpfc87nMV/ixp3VYZAg= 2023-05-24T21:28:03.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:rbR21sDnRpfc87nMV/ixp3VYZAg=', 'Content-Length': '0'} 2023-05-24T21:28:03.830 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000955f6f4303889c1-00646e8163-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.830 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:03.831 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:03.831 DEBUG:boto:path=/qskrps-49/?prefix=obj4 2023-05-24T21:28:03.831 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj4 2023-05-24T21:28:03.832 DEBUG:boto:Method: GET 2023-05-24T21:28:03.832 DEBUG:boto:Path: /qskrps-49/?prefix=obj4 2023-05-24T21:28:03.833 DEBUG:boto:Data: 2023-05-24T21:28:03.833 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.833 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:03.834 DEBUG:boto:Port: 8001 2023-05-24T21:28:03.834 DEBUG:boto:Params: {} 2023-05-24T21:28:03.834 DEBUG:boto:Token: None 2023-05-24T21:28:03.835 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:03 GMT /qskrps-49/ 2023-05-24T21:28:03.835 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:/UdLQCNOgXdcB4jQAdZKvD0AqBI= 2023-05-24T21:28:03.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:/UdLQCNOgXdcB4jQAdZKvD0AqBI=', 'Content-Length': '0'} 2023-05-24T21:28:03.838 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e45da88f7057e45c-00646e8163-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.838 DEBUG:boto:b'qskrps-49obj41000falseobj42023-05-24T21:27:06.514Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:03.839 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:03.839 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:03.840 DEBUG:boto:path=/qskrps-49/?prefix=obj4 2023-05-24T21:28:03.840 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj4 2023-05-24T21:28:03.840 DEBUG:boto:Method: GET 2023-05-24T21:28:03.841 DEBUG:boto:Path: /qskrps-49/?prefix=obj4 2023-05-24T21:28:03.841 DEBUG:boto:Data: 2023-05-24T21:28:03.842 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.842 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:03.842 DEBUG:boto:Port: 8000 2023-05-24T21:28:03.843 DEBUG:boto:Params: {} 2023-05-24T21:28:03.843 DEBUG:boto:Token: None 2023-05-24T21:28:03.843 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:03 GMT /qskrps-49/ 2023-05-24T21:28:03.844 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:/UdLQCNOgXdcB4jQAdZKvD0AqBI= 2023-05-24T21:28:03.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:/UdLQCNOgXdcB4jQAdZKvD0AqBI=', 'Content-Length': '0'} 2023-05-24T21:28:03.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098883a48cd369c1a-00646e8163-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.847 DEBUG:boto:b'qskrps-49obj41000falseobj42023-05-24T21:27:06.514Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:03.848 DEBUG:boto:path=/qskrps-49/obj4 2023-05-24T21:28:03.848 DEBUG:boto:auth_path=/qskrps-49/obj4 2023-05-24T21:28:03.849 DEBUG:boto:Method: HEAD 2023-05-24T21:28:03.849 DEBUG:boto:Path: /qskrps-49/obj4 2023-05-24T21:28:03.850 DEBUG:boto:Data: 2023-05-24T21:28:03.850 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.850 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:03.851 DEBUG:boto:Port: 8001 2023-05-24T21:28:03.851 DEBUG:boto:Params: {} 2023-05-24T21:28:03.852 DEBUG:boto:Token: None 2023-05-24T21:28:03.852 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:03 GMT /qskrps-49/obj4 2023-05-24T21:28:03.852 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:2OKpKbEBuB5mbxnOmu2ZoYrtFM4= 2023-05-24T21:28:03.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:2OKpKbEBuB5mbxnOmu2ZoYrtFM4=', 'Content-Length': '0'} 2023-05-24T21:28:03.855 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000043202236d8cd12b3-00646e8163-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.855 DEBUG:boto:path=/qskrps-49/obj4 2023-05-24T21:28:03.856 DEBUG:boto:auth_path=/qskrps-49/obj4 2023-05-24T21:28:03.856 DEBUG:boto:Method: HEAD 2023-05-24T21:28:03.856 DEBUG:boto:Path: /qskrps-49/obj4 2023-05-24T21:28:03.857 DEBUG:boto:Data: 2023-05-24T21:28:03.857 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.857 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:03.858 DEBUG:boto:Port: 8000 2023-05-24T21:28:03.858 DEBUG:boto:Params: {} 2023-05-24T21:28:03.858 DEBUG:boto:Token: None 2023-05-24T21:28:03.859 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:03 GMT /qskrps-49/obj4 2023-05-24T21:28:03.859 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:2OKpKbEBuB5mbxnOmu2ZoYrtFM4= 2023-05-24T21:28:03.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:2OKpKbEBuB5mbxnOmu2ZoYrtFM4=', 'Content-Length': '0'} 2023-05-24T21:28:03.861 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f3eea93223954acc-00646e8163-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-24T21:28:03.862 DEBUG:boto:path=/qskrps-49/obj4 2023-05-24T21:28:03.863 DEBUG:boto:auth_path=/qskrps-49/obj4 2023-05-24T21:28:03.863 DEBUG:boto:Method: GET 2023-05-24T21:28:03.863 DEBUG:boto:Path: /qskrps-49/obj4 2023-05-24T21:28:03.864 DEBUG:boto:Data: 2023-05-24T21:28:03.864 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.864 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:03.865 DEBUG:boto:Port: 8001 2023-05-24T21:28:03.865 DEBUG:boto:Params: {} 2023-05-24T21:28:03.866 DEBUG:boto:Token: None 2023-05-24T21:28:03.866 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:03 GMT /qskrps-49/obj4 2023-05-24T21:28:03.866 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Nz8Ja12XHvpOO7yB7imvF93q0Js= 2023-05-24T21:28:03.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Nz8Ja12XHvpOO7yB7imvF93q0Js=', 'Content-Length': '0'} 2023-05-24T21:28:03.868 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000029bd2d36ded2e502-00646e8163-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.869 DEBUG:boto:path=/qskrps-49/obj4 2023-05-24T21:28:03.869 DEBUG:boto:auth_path=/qskrps-49/obj4 2023-05-24T21:28:03.870 DEBUG:boto:Method: GET 2023-05-24T21:28:03.870 DEBUG:boto:Path: /qskrps-49/obj4 2023-05-24T21:28:03.871 DEBUG:boto:Data: 2023-05-24T21:28:03.871 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.871 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:03.872 DEBUG:boto:Port: 8000 2023-05-24T21:28:03.872 DEBUG:boto:Params: {} 2023-05-24T21:28:03.873 DEBUG:boto:Token: None 2023-05-24T21:28:03.873 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:03 GMT /qskrps-49/obj4 2023-05-24T21:28:03.873 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Nz8Ja12XHvpOO7yB7imvF93q0Js= 2023-05-24T21:28:03.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Nz8Ja12XHvpOO7yB7imvF93q0Js=', 'Content-Length': '0'} 2023-05-24T21:28:03.876 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f3ebfff5463669cd-00646e8163-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.876 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:03.877 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:03.877 DEBUG:boto:path=/qskrps-49/?prefix=obj4 2023-05-24T21:28:03.878 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj4 2023-05-24T21:28:03.878 DEBUG:boto:Method: GET 2023-05-24T21:28:03.878 DEBUG:boto:Path: /qskrps-49/?prefix=obj4 2023-05-24T21:28:03.879 DEBUG:boto:Data: 2023-05-24T21:28:03.879 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.879 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:03.880 DEBUG:boto:Port: 8001 2023-05-24T21:28:03.880 DEBUG:boto:Params: {} 2023-05-24T21:28:03.880 DEBUG:boto:Token: None 2023-05-24T21:28:03.881 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:03 GMT /qskrps-49/ 2023-05-24T21:28:03.881 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:/UdLQCNOgXdcB4jQAdZKvD0AqBI= 2023-05-24T21:28:03.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:/UdLQCNOgXdcB4jQAdZKvD0AqBI=', 'Content-Length': '0'} 2023-05-24T21:28:03.884 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057b322b8048d91bd-00646e8163-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.885 DEBUG:boto:b'qskrps-49obj41000falseobj42023-05-24T21:27:06.514Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:03.885 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:03.886 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:03.886 DEBUG:boto:path=/qskrps-49/?prefix=obj4 2023-05-24T21:28:03.886 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj4 2023-05-24T21:28:03.887 DEBUG:boto:Method: GET 2023-05-24T21:28:03.887 DEBUG:boto:Path: /qskrps-49/?prefix=obj4 2023-05-24T21:28:03.887 DEBUG:boto:Data: 2023-05-24T21:28:03.888 DEBUG:boto:Headers: {} 2023-05-24T21:28:03.888 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:03.889 DEBUG:boto:Port: 8000 2023-05-24T21:28:03.889 DEBUG:boto:Params: {} 2023-05-24T21:28:03.889 DEBUG:boto:Token: None 2023-05-24T21:28:03.890 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:03 GMT /qskrps-49/ 2023-05-24T21:28:03.890 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:/UdLQCNOgXdcB4jQAdZKvD0AqBI= 2023-05-24T21:28:03.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:/UdLQCNOgXdcB4jQAdZKvD0AqBI=', 'Content-Length': '0'} 2023-05-24T21:28:03.893 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000731e5878ce0d3c4d-00646e8163-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:03.894 DEBUG:boto:b'qskrps-49obj41000falseobj42023-05-24T21:27:06.514Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:03.894 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qskrps-49 zones={a2, a1} 2023-05-24T21:28:03.895 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'qskrps-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:28:03.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'qskrps-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:28:03.895 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket qskrps-48 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-24T21:28:03.951 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:28:03.951 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:28:04.033 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'qskrps-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:28:04.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'qskrps-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:28:04.034 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket qskrps-49 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-24T21:28:04.084 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:28:04.084 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:28:04.191 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:28:04.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:28:04.192 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:28:04.240 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:28:04.241 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:28:04.272 INFO:teuthology.orchestra.run.smithi071.stderr:No --period given, using current period=5c04936b-70ea-4415-b4eb-3aad3dc612e5 2023-05-24T21:28:04.285 INFO:teuthology.orchestra.run.smithi071.stdout:[ 2023-05-24T21:28:04.285 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:28:04.285 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963684.168181_538.1", 2023-05-24T21:28:04.286 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:28:04.168181Z" 2023-05-24T21:28:04.286 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:04.286 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:28:04.286 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:28:04.286 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:06:32.333844Z" 2023-05-24T21:28:04.286 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:04.286 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:28:04.287 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:28:04.287 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:28:04.287 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:04.287 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:28:04.287 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:28:04.287 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:28:04.288 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:28:04.288 INFO:teuthology.orchestra.run.smithi071.stdout:] 2023-05-24T21:28:04.288 DEBUG:rgw_multi.tests:master meta markers={0: '1_1684963684.168181_538.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:28:04.289 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-24T21:28:04.289 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:28:04.290 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:28:04.290 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:28:04.339 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:28:04.339 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:28:04.371 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:28:04.366+0000 7fb0ba399780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:28:04.372 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:28:04.367+0000 7fb0ba399780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:28:04.387 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:28:04.388 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:28:04.388 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:28:04.388 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:28:04.388 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:28:04.388 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:28:04.388 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:28:04.389 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:04.389 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:28:04.389 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:28:04.389 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:28:04.389 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:28:04.389 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:28:04.390 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963626.276651_526.1", 2023-05-24T21:28:04.390 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:28:04.390 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:28:04.390 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:28:04.390 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:27:06.276651Z", 2023-05-24T21:28:04.390 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:28:04.391 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:28:04.391 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:04.391 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:28:04.391 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:28:04.391 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:28:04.391 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:28:04.392 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:28:04.392 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:28:04.392 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:28:04.392 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:28:04.392 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:28:04.392 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:28:04.393 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:28:04.393 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:04.393 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:28:04.393 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:28:04.393 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:28:04.393 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:28:04.393 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:28:04.394 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:28:04.394 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:28:04.394 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:28:04.394 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:28:04.394 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:28:04.394 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:28:04.394 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:04.395 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:28:04.395 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:28:04.395 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:28:04.395 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:28:04.395 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:28:04.395 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:28:04.395 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:28:04.396 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:28:04.396 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:28:04.396 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:28:04.396 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:28:04.396 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:28:04.396 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:28:04.397 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:04.397 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:28:04.397 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:28:04.397 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:28:04.397 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:04.397 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:28:04Z" 2023-05-24T21:28:04.397 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:28:04.398 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684963626.276651_526.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:27:06.276651Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:28:04Z" } 2023-05-24T21:28:04.399 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963626.276651_526.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:27:06.276651Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:28:04.399 DEBUG:rgw_multi.tests:log_status={0: '1_1684963684.168181_538.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:28:04.399 DEBUG:rgw_multi.tests:sync_status={0: '1_1684963626.276651_526.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:28:04.400 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963684.168181_538.1 target=1_1684963626.276651_526.1 2023-05-24T21:28:06.716 INFO:tasks.rgw.c2.client.0.smithi157.stdout:2023-05-24T21:28:06.709+0000 7f70b8e0a700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-05-24T21:28:06.717 INFO:tasks.rgw.c2.client.0.smithi157.stdout:2023-05-24T21:28:06.710+0000 7f70bf617700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-05-24T21:28:06.717 INFO:tasks.rgw.c2.client.0.smithi157.stdout:2023-05-24T21:28:06.710+0000 7f70bde14700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-05-24T21:28:06.717 INFO:tasks.rgw.c2.client.0.smithi157.stdout:2023-05-24T21:28:06.710+0000 7f70b9e0c700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-05-24T21:28:06.718 INFO:tasks.rgw.c2.client.0.smithi157.stdout:2023-05-24T21:28:06.710+0000 7f70b960b700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-05-24T21:28:06.718 INFO:tasks.rgw.c2.client.0.smithi157.stdout:2023-05-24T21:28:06.710+0000 7f70b7e08700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-05-24T21:28:06.718 INFO:tasks.rgw.c2.client.0.smithi157.stdout:2023-05-24T21:28:06.710+0000 7f70c4621700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-05-24T21:28:06.718 INFO:tasks.rgw.c2.client.0.smithi157.stdout:2023-05-24T21:28:06.710+0000 7f70c261d700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-05-24T21:28:06.718 INFO:tasks.rgw.c2.client.0.smithi157.stdout:2023-05-24T21:28:06.711+0000 7f70c0e1a700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-05-24T21:28:06.719 INFO:tasks.rgw.c2.client.0.smithi157.stdout:2023-05-24T21:28:06.711+0000 7f70c0619700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-05-24T21:28:09.401 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:28:09.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:28:09.403 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:28:09.458 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:28:09.459 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:28:09.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:28:09.497+0000 7fc0f34fc780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:28:09.503 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:28:09.498+0000 7fc0f34fc780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:28:09.521 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:28:09.521 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:28:09.521 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:28:09.521 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:28:09.522 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:28:09.522 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:28:09.522 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:28:09.522 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:09.522 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:28:09.522 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:28:09.523 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:28:09.523 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:28:09.523 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:28:09.523 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963684.168181_538.1", 2023-05-24T21:28:09.523 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:28:09.523 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:28:09.523 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:28:09.524 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:28:04.168181Z", 2023-05-24T21:28:09.524 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:28:09.524 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:28:09.524 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:09.524 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:28:09.524 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:28:09.525 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:28:09.525 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:28:09.525 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:28:09.525 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:28:09.525 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:28:09.525 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:28:09.525 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:28:09.526 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:28:09.526 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:28:09.526 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:09.526 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:28:09.526 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:28:09.526 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:28:09.527 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:28:09.527 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:28:09.527 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:28:09.527 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:28:09.527 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:28:09.527 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:28:09.527 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:28:09.528 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:28:09.528 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:09.528 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:28:09.528 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:28:09.528 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:28:09.528 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:28:09.528 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:28:09.529 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:28:09.529 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:28:09.529 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:28:09.529 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:28:09.529 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:28:09.529 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:28:09.530 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:28:09.530 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:28:09.530 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:09.530 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:28:09.530 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:28:09.530 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:28:09.530 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:09.531 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:28:09Z" 2023-05-24T21:28:09.531 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:28:09.532 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684963684.168181_538.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:28:04.168181Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:28:09Z" } 2023-05-24T21:28:09.532 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963684.168181_538.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:28:04.168181Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:28:09.533 DEBUG:rgw_multi.tests:log_status={0: '1_1684963684.168181_538.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:28:09.533 DEBUG:rgw_multi.tests:sync_status={0: '1_1684963684.168181_538.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:28:09.533 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-24T21:28:09.534 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:28:09.534 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:28:09.535 DEBUG:boto:Method: HEAD 2023-05-24T21:28:09.535 DEBUG:boto:Path: /qskrps-48/ 2023-05-24T21:28:09.535 DEBUG:boto:Data: 2023-05-24T21:28:09.536 DEBUG:boto:Headers: {} 2023-05-24T21:28:09.536 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:09.536 DEBUG:boto:Port: 8000 2023-05-24T21:28:09.537 DEBUG:boto:Params: {} 2023-05-24T21:28:09.537 DEBUG:boto:Token: None 2023-05-24T21:28:09.537 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:09 GMT /qskrps-48/ 2023-05-24T21:28:09.538 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:G5fGSHGbXtrB5eTI/oY77ppn0V0= 2023-05-24T21:28:09.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:09 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:G5fGSHGbXtrB5eTI/oY77ppn0V0=', 'Content-Length': '0'} 2023-05-24T21:28:09.541 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000da5565a2cfb53a8f-00646e8169-1627-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:28:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:09.541 DEBUG:boto:path=/qskrps-48/obj5 2023-05-24T21:28:09.542 DEBUG:boto:auth_path=/qskrps-48/obj5 2023-05-24T21:28:09.542 DEBUG:boto:Method: PUT 2023-05-24T21:28:09.543 DEBUG:boto:Path: /qskrps-48/obj5 2023-05-24T21:28:09.543 DEBUG:boto:Data: 2023-05-24T21:28:09.543 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-24T21:28:09.544 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:09.544 DEBUG:boto:Port: 8000 2023-05-24T21:28:09.544 DEBUG:boto:Params: {} 2023-05-24T21:28:09.545 DEBUG:boto:Token: None 2023-05-24T21:28:09.545 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 24 May 2023 21:28:09 GMT /qskrps-48/obj5 2023-05-24T21:28:09.545 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:xC2UpZUvVO8f8nk9g3Di07mKihs= 2023-05-24T21:28:09.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:28:09 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:xC2UpZUvVO8f8nk9g3Di07mKihs='} 2023-05-24T21:28:09.588 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bc0784a5330fd666-00646e8169-1627-a1'), ('Date', 'Wed, 24 May 2023 21:28:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:09.589 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:28:09.590 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:28:09.590 DEBUG:boto:Method: HEAD 2023-05-24T21:28:09.590 DEBUG:boto:Path: /qskrps-48/ 2023-05-24T21:28:09.591 DEBUG:boto:Data: 2023-05-24T21:28:09.591 DEBUG:boto:Headers: {} 2023-05-24T21:28:09.591 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:09.592 DEBUG:boto:Port: 8000 2023-05-24T21:28:09.592 DEBUG:boto:Params: {} 2023-05-24T21:28:09.592 DEBUG:boto:Token: None 2023-05-24T21:28:09.593 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:09 GMT /qskrps-48/ 2023-05-24T21:28:09.593 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:G5fGSHGbXtrB5eTI/oY77ppn0V0= 2023-05-24T21:28:09.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:09 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:G5fGSHGbXtrB5eTI/oY77ppn0V0=', 'Content-Length': '0'} 2023-05-24T21:28:09.596 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002a347d2e0f3b67e2-00646e8169-1627-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:28:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:09.597 DEBUG:boto:path=/qskrps-48/obj6 2023-05-24T21:28:09.597 DEBUG:boto:auth_path=/qskrps-48/obj6 2023-05-24T21:28:09.597 DEBUG:boto:Method: PUT 2023-05-24T21:28:09.598 DEBUG:boto:Path: /qskrps-48/obj6 2023-05-24T21:28:09.598 DEBUG:boto:Data: 2023-05-24T21:28:09.598 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-24T21:28:09.599 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:09.599 DEBUG:boto:Port: 8000 2023-05-24T21:28:09.599 DEBUG:boto:Params: {} 2023-05-24T21:28:09.600 DEBUG:boto:Token: None 2023-05-24T21:28:09.600 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 24 May 2023 21:28:09 GMT /qskrps-48/obj6 2023-05-24T21:28:09.600 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:pMBgen5HxeVTBi5wHn+1NPdjFkU= 2023-05-24T21:28:09.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:28:09 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:pMBgen5HxeVTBi5wHn+1NPdjFkU='} 2023-05-24T21:28:09.606 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b7add0cc0a198890-00646e8169-1627-a1'), ('Date', 'Wed, 24 May 2023 21:28:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:09.607 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:28:09.607 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:28:09.608 DEBUG:boto:Method: HEAD 2023-05-24T21:28:09.608 DEBUG:boto:Path: /qskrps-48/ 2023-05-24T21:28:09.608 DEBUG:boto:Data: 2023-05-24T21:28:09.609 DEBUG:boto:Headers: {} 2023-05-24T21:28:09.609 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:09.609 DEBUG:boto:Port: 8000 2023-05-24T21:28:09.610 DEBUG:boto:Params: {} 2023-05-24T21:28:09.610 DEBUG:boto:Token: None 2023-05-24T21:28:09.610 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:09 GMT /qskrps-48/ 2023-05-24T21:28:09.611 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:G5fGSHGbXtrB5eTI/oY77ppn0V0= 2023-05-24T21:28:09.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:09 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:G5fGSHGbXtrB5eTI/oY77ppn0V0=', 'Content-Length': '0'} 2023-05-24T21:28:09.613 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fd06ad968ec64853-00646e8169-1627-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:28:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:09.614 DEBUG:boto:path=/qskrps-48/obj7 2023-05-24T21:28:09.614 DEBUG:boto:auth_path=/qskrps-48/obj7 2023-05-24T21:28:09.615 DEBUG:boto:Method: PUT 2023-05-24T21:28:09.615 DEBUG:boto:Path: /qskrps-48/obj7 2023-05-24T21:28:09.615 DEBUG:boto:Data: 2023-05-24T21:28:09.616 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-24T21:28:09.616 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:09.616 DEBUG:boto:Port: 8000 2023-05-24T21:28:09.617 DEBUG:boto:Params: {} 2023-05-24T21:28:09.617 DEBUG:boto:Token: None 2023-05-24T21:28:09.617 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 24 May 2023 21:28:09 GMT /qskrps-48/obj7 2023-05-24T21:28:09.618 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:CvZkUHd4qJ3hHbfm2Mu5pGoWqsM= 2023-05-24T21:28:09.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:28:09 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:CvZkUHd4qJ3hHbfm2Mu5pGoWqsM='} 2023-05-24T21:28:09.660 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d3a08a7fd34af290-00646e8169-1627-a1'), ('Date', 'Wed, 24 May 2023 21:28:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:09.661 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:28:09.662 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:28:09.662 DEBUG:boto:Method: HEAD 2023-05-24T21:28:09.662 DEBUG:boto:Path: /qskrps-48/ 2023-05-24T21:28:09.663 DEBUG:boto:Data: 2023-05-24T21:28:09.663 DEBUG:boto:Headers: {} 2023-05-24T21:28:09.663 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:09.664 DEBUG:boto:Port: 8000 2023-05-24T21:28:09.664 DEBUG:boto:Params: {} 2023-05-24T21:28:09.664 DEBUG:boto:Token: None 2023-05-24T21:28:09.665 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:09 GMT /qskrps-48/ 2023-05-24T21:28:09.665 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:G5fGSHGbXtrB5eTI/oY77ppn0V0= 2023-05-24T21:28:09.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:09 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:G5fGSHGbXtrB5eTI/oY77ppn0V0=', 'Content-Length': '0'} 2023-05-24T21:28:09.668 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008a4650a3f39994c8-00646e8169-1627-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:28:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:09.668 DEBUG:boto:path=/qskrps-48/obj8 2023-05-24T21:28:09.669 DEBUG:boto:auth_path=/qskrps-48/obj8 2023-05-24T21:28:09.669 DEBUG:boto:Method: PUT 2023-05-24T21:28:09.670 DEBUG:boto:Path: /qskrps-48/obj8 2023-05-24T21:28:09.670 DEBUG:boto:Data: 2023-05-24T21:28:09.670 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-24T21:28:09.671 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:09.671 DEBUG:boto:Port: 8000 2023-05-24T21:28:09.671 DEBUG:boto:Params: {} 2023-05-24T21:28:09.672 DEBUG:boto:Token: None 2023-05-24T21:28:09.672 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 24 May 2023 21:28:09 GMT /qskrps-48/obj8 2023-05-24T21:28:09.672 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:AwFlt0ghujJ3WpM/ZB7lSUmRt5E= 2023-05-24T21:28:09.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:28:09 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:AwFlt0ghujJ3WpM/ZB7lSUmRt5E='} 2023-05-24T21:28:09.679 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dac2d2ae6e23a2d2-00646e8169-1627-a1'), ('Date', 'Wed, 24 May 2023 21:28:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:09.679 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:09.680 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:09.680 DEBUG:boto:Method: HEAD 2023-05-24T21:28:09.680 DEBUG:boto:Path: /qskrps-49/ 2023-05-24T21:28:09.681 DEBUG:boto:Data: 2023-05-24T21:28:09.681 DEBUG:boto:Headers: {} 2023-05-24T21:28:09.681 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:09.682 DEBUG:boto:Port: 8001 2023-05-24T21:28:09.682 DEBUG:boto:Params: {} 2023-05-24T21:28:09.682 DEBUG:boto:Token: None 2023-05-24T21:28:09.683 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:09 GMT /qskrps-49/ 2023-05-24T21:28:09.683 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:zJL2cF2YNxCrlC72klLLdgQwjPs= 2023-05-24T21:28:09.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:09 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:zJL2cF2YNxCrlC72klLLdgQwjPs=', 'Content-Length': '0'} 2023-05-24T21:28:09.686 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005d379134e15f7e88-00646e8169-1625-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:28:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:09.686 DEBUG:boto:path=/qskrps-49/obj5 2023-05-24T21:28:09.687 DEBUG:boto:auth_path=/qskrps-49/obj5 2023-05-24T21:28:09.687 DEBUG:boto:Method: PUT 2023-05-24T21:28:09.687 DEBUG:boto:Path: /qskrps-49/obj5 2023-05-24T21:28:09.688 DEBUG:boto:Data: 2023-05-24T21:28:09.688 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-24T21:28:09.688 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:09.689 DEBUG:boto:Port: 8001 2023-05-24T21:28:09.689 DEBUG:boto:Params: {} 2023-05-24T21:28:09.689 DEBUG:boto:Token: None 2023-05-24T21:28:09.690 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 24 May 2023 21:28:09 GMT /qskrps-49/obj5 2023-05-24T21:28:09.690 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:6NKso6rhGYPmJ6edcVTmpHJnwFc= 2023-05-24T21:28:09.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:28:09 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:6NKso6rhGYPmJ6edcVTmpHJnwFc='} 2023-05-24T21:28:09.695 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000575dd6d6c6117f5c-00646e8169-1625-a2'), ('Date', 'Wed, 24 May 2023 21:28:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:09.695 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:09.695 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:09.696 DEBUG:boto:Method: HEAD 2023-05-24T21:28:09.696 DEBUG:boto:Path: /qskrps-49/ 2023-05-24T21:28:09.696 DEBUG:boto:Data: 2023-05-24T21:28:09.697 DEBUG:boto:Headers: {} 2023-05-24T21:28:09.697 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:09.697 DEBUG:boto:Port: 8001 2023-05-24T21:28:09.698 DEBUG:boto:Params: {} 2023-05-24T21:28:09.698 DEBUG:boto:Token: None 2023-05-24T21:28:09.699 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:09 GMT /qskrps-49/ 2023-05-24T21:28:09.699 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:zJL2cF2YNxCrlC72klLLdgQwjPs= 2023-05-24T21:28:09.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:09 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:zJL2cF2YNxCrlC72klLLdgQwjPs=', 'Content-Length': '0'} 2023-05-24T21:28:09.701 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004ea82082453d42da-00646e8169-1625-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:28:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:09.702 DEBUG:boto:path=/qskrps-49/obj6 2023-05-24T21:28:09.702 DEBUG:boto:auth_path=/qskrps-49/obj6 2023-05-24T21:28:09.703 DEBUG:boto:Method: PUT 2023-05-24T21:28:09.703 DEBUG:boto:Path: /qskrps-49/obj6 2023-05-24T21:28:09.703 DEBUG:boto:Data: 2023-05-24T21:28:09.704 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-24T21:28:09.704 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:09.705 DEBUG:boto:Port: 8001 2023-05-24T21:28:09.705 DEBUG:boto:Params: {} 2023-05-24T21:28:09.705 DEBUG:boto:Token: None 2023-05-24T21:28:09.706 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 24 May 2023 21:28:09 GMT /qskrps-49/obj6 2023-05-24T21:28:09.706 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:izQecO+y/cN5ZFMYLgQqZIqnUks= 2023-05-24T21:28:09.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:28:09 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:izQecO+y/cN5ZFMYLgQqZIqnUks='} 2023-05-24T21:28:09.752 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008f846847bda519eb-00646e8169-1625-a2'), ('Date', 'Wed, 24 May 2023 21:28:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:09.753 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:09.754 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:09.754 DEBUG:boto:Method: HEAD 2023-05-24T21:28:09.754 DEBUG:boto:Path: /qskrps-49/ 2023-05-24T21:28:09.755 DEBUG:boto:Data: 2023-05-24T21:28:09.755 DEBUG:boto:Headers: {} 2023-05-24T21:28:09.755 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:09.756 DEBUG:boto:Port: 8001 2023-05-24T21:28:09.756 DEBUG:boto:Params: {} 2023-05-24T21:28:09.756 DEBUG:boto:Token: None 2023-05-24T21:28:09.757 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:09 GMT /qskrps-49/ 2023-05-24T21:28:09.757 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:zJL2cF2YNxCrlC72klLLdgQwjPs= 2023-05-24T21:28:09.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:09 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:zJL2cF2YNxCrlC72klLLdgQwjPs=', 'Content-Length': '0'} 2023-05-24T21:28:09.760 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003522cfa4ac522e7f-00646e8169-1625-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:28:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:09.760 DEBUG:boto:path=/qskrps-49/obj7 2023-05-24T21:28:09.761 DEBUG:boto:auth_path=/qskrps-49/obj7 2023-05-24T21:28:09.761 DEBUG:boto:Method: PUT 2023-05-24T21:28:09.762 DEBUG:boto:Path: /qskrps-49/obj7 2023-05-24T21:28:09.762 DEBUG:boto:Data: 2023-05-24T21:28:09.762 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-24T21:28:09.763 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:09.763 DEBUG:boto:Port: 8001 2023-05-24T21:28:09.763 DEBUG:boto:Params: {} 2023-05-24T21:28:09.764 DEBUG:boto:Token: None 2023-05-24T21:28:09.764 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 24 May 2023 21:28:09 GMT /qskrps-49/obj7 2023-05-24T21:28:09.765 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:jANmfEWILpRYVfNCzrKclXQUb9c= 2023-05-24T21:28:09.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:28:09 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:jANmfEWILpRYVfNCzrKclXQUb9c='} 2023-05-24T21:28:09.771 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a26b889d7a00c9d1-00646e8169-1625-a2'), ('Date', 'Wed, 24 May 2023 21:28:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:09.771 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:09.772 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:09.772 DEBUG:boto:Method: HEAD 2023-05-24T21:28:09.772 DEBUG:boto:Path: /qskrps-49/ 2023-05-24T21:28:09.773 DEBUG:boto:Data: 2023-05-24T21:28:09.773 DEBUG:boto:Headers: {} 2023-05-24T21:28:09.773 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:09.774 DEBUG:boto:Port: 8001 2023-05-24T21:28:09.774 DEBUG:boto:Params: {} 2023-05-24T21:28:09.774 DEBUG:boto:Token: None 2023-05-24T21:28:09.775 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:09 GMT /qskrps-49/ 2023-05-24T21:28:09.775 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:zJL2cF2YNxCrlC72klLLdgQwjPs= 2023-05-24T21:28:09.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:09 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:zJL2cF2YNxCrlC72klLLdgQwjPs=', 'Content-Length': '0'} 2023-05-24T21:28:09.778 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fd851ef1e7549e22-00646e8169-1625-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:28:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:09.779 DEBUG:boto:path=/qskrps-49/obj8 2023-05-24T21:28:09.779 DEBUG:boto:auth_path=/qskrps-49/obj8 2023-05-24T21:28:09.780 DEBUG:boto:Method: PUT 2023-05-24T21:28:09.780 DEBUG:boto:Path: /qskrps-49/obj8 2023-05-24T21:28:09.780 DEBUG:boto:Data: 2023-05-24T21:28:09.781 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-24T21:28:09.781 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:09.781 DEBUG:boto:Port: 8001 2023-05-24T21:28:09.782 DEBUG:boto:Params: {} 2023-05-24T21:28:09.782 DEBUG:boto:Token: None 2023-05-24T21:28:09.782 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 24 May 2023 21:28:09 GMT /qskrps-49/obj8 2023-05-24T21:28:09.783 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:mLL+4CA4aLM8mj0oHbXHZGsSbtU= 2023-05-24T21:28:09.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:28:09 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:mLL+4CA4aLM8mj0oHbXHZGsSbtU='} 2023-05-24T21:28:09.788 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000861c65154bf14c2c-00646e8169-1625-a2'), ('Date', 'Wed, 24 May 2023 21:28:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:09.788 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'qskrps-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:28:09.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'qskrps-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:28:09.789 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket qskrps-48 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-24T21:28:09.845 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:28:09.845 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:28:09.967 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'qskrps-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:28:09.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'qskrps-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-24T21:28:09.969 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket qskrps-49 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-24T21:28:10.022 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:28:10.023 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:28:10.108 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=qskrps-48 2023-05-24T21:28:10.108 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:28:10.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:28:10.109 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-48 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:28:10.169 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:28:10.170 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:28:10.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:28:10.224+0000 7f2a572ea780 1 waiting to reach incremental sync.. 2023-05-24T21:28:15.232 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:28:15.226+0000 7f2a572ea780 1 bucket sync caught up with source: 2023-05-24T21:28:15.232 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [00000000004.22.1, 00000000002.21.1, 00000000004.24.1, 00000000002.25.1, 00000000004.34.1, 00000000002.17.1, 00000000002.61.1, 00000000004.24.1, 00000000002.27.1, 00000000002.22.1, 00000000002.26.1] 2023-05-24T21:28:15.232 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [00000000004.22.1, 00000000002.21.1, 00000000004.24.1, 00000000002.25.1, 00000000004.34.1, 00000000002.17.1, 00000000002.61.1, 00000000004.24.1, 00000000002.27.1, 00000000002.22.1, 00000000002.26.1] 2023-05-24T21:28:15.233 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:28:15.226+0000 7f2a572ea780 0 bucket checkpoint complete 2023-05-24T21:28:15.249 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=qskrps-48 2023-05-24T21:28:15.250 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:28:15.250 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:28:15.251 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-48 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:28:15.312 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:28:15.313 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:28:15.350 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:28:15.345+0000 7f5c76503780 1 waiting to reach incremental sync.. 2023-05-24T21:28:20.352 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:28:20.346+0000 7f5c76503780 1 bucket sync caught up with source: 2023-05-24T21:28:20.353 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [00000000004.666.1, 00000000002.1156.1, 00000000004.351.1, 00000000002.371.1, 00000000004.404.1, 00000000002.979.1, 00000000002.567.1, 00000000004.835.1, 00000000002.1306.1, 00000000002.1159.1, 00000000002.551.1] 2023-05-24T21:28:20.353 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [00000000004.666.1, 00000000002.1156.1, 00000000004.351.1, 00000000002.371.1, 00000000004.404.1, 00000000002.979.1, 00000000002.567.1, 00000000004.835.1, 00000000002.1306.1, 00000000002.1159.1, 00000000002.551.1] 2023-05-24T21:28:20.353 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:28:20.346+0000 7f5c76503780 0 bucket checkpoint complete 2023-05-24T21:28:20.370 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qskrps-48 zones={a2, a1} 2023-05-24T21:28:20.371 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:28:20.372 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:28:20.373 DEBUG:boto:Method: HEAD 2023-05-24T21:28:20.374 DEBUG:boto:Path: /qskrps-48/ 2023-05-24T21:28:20.375 DEBUG:boto:Data: 2023-05-24T21:28:20.376 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.377 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:20.378 DEBUG:boto:Port: 8001 2023-05-24T21:28:20.379 DEBUG:boto:Params: {} 2023-05-24T21:28:20.380 DEBUG:boto:Token: None 2023-05-24T21:28:20.381 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:20 GMT /qskrps-48/ 2023-05-24T21:28:20.382 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:kg/JljpTMcJEzccqPstE7BVS7A4= 2023-05-24T21:28:20.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:kg/JljpTMcJEzccqPstE7BVS7A4=', 'Content-Length': '0'} 2023-05-24T21:28:20.387 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007beae9e58799d430-00646e8174-1625-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.388 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:28:20.388 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:28:20.389 DEBUG:boto:Method: HEAD 2023-05-24T21:28:20.390 DEBUG:boto:Path: /qskrps-48/ 2023-05-24T21:28:20.390 DEBUG:boto:Data: 2023-05-24T21:28:20.391 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.392 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:20.393 DEBUG:boto:Port: 8000 2023-05-24T21:28:20.393 DEBUG:boto:Params: {} 2023-05-24T21:28:20.394 DEBUG:boto:Token: None 2023-05-24T21:28:20.395 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:20 GMT /qskrps-48/ 2023-05-24T21:28:20.396 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:kg/JljpTMcJEzccqPstE7BVS7A4= 2023-05-24T21:28:20.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:kg/JljpTMcJEzccqPstE7BVS7A4=', 'Content-Length': '0'} 2023-05-24T21:28:20.401 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000089dcd10d58d0af5b-00646e8174-1627-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.402 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-24T21:28:20.403 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:28:20.403 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:28:20.404 DEBUG:boto:path=/qskrps-48/?versions&max-keys=999 2023-05-24T21:28:20.405 DEBUG:boto:auth_path=/qskrps-48/?versions&max-keys=999 2023-05-24T21:28:20.405 DEBUG:boto:Method: GET 2023-05-24T21:28:20.406 DEBUG:boto:Path: /qskrps-48/?versions&max-keys=999 2023-05-24T21:28:20.407 DEBUG:boto:Data: 2023-05-24T21:28:20.407 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.408 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:20.408 DEBUG:boto:Port: 8001 2023-05-24T21:28:20.409 DEBUG:boto:Params: {} 2023-05-24T21:28:20.410 DEBUG:boto:Token: None 2023-05-24T21:28:20.410 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:20 GMT /qskrps-48/?versions 2023-05-24T21:28:20.411 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:nS25WVqz7dTgZM78ONQzZFpMs2c= 2023-05-24T21:28:20.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:nS25WVqz7dTgZM78ONQzZFpMs2c=', 'Content-Length': '0'} 2023-05-24T21:28:20.415 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001633a55bd88c3904-00646e8174-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.416 DEBUG:boto:b'qskrps-48999falseobj1nulltrue2023-05-24T21:27:06.310Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-24T21:27:06.330Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-24T21:27:06.348Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-24T21:27:06.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-05-24T21:28:09.543Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-05-24T21:28:09.598Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-05-24T21:28:09.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-05-24T21:28:09.670Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:20.417 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-05-24T21:28:20.418 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-05-24T21:28:20.418 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-05-24T21:28:20.419 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-05-24T21:28:20.419 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-05-24T21:28:20.420 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-05-24T21:28:20.420 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-05-24T21:28:20.421 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-05-24T21:28:20.421 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-24T21:28:20.422 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:28:20.423 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:28:20.423 DEBUG:boto:path=/qskrps-48/?versions&max-keys=999 2023-05-24T21:28:20.424 DEBUG:boto:auth_path=/qskrps-48/?versions&max-keys=999 2023-05-24T21:28:20.424 DEBUG:boto:Method: GET 2023-05-24T21:28:20.425 DEBUG:boto:Path: /qskrps-48/?versions&max-keys=999 2023-05-24T21:28:20.425 DEBUG:boto:Data: 2023-05-24T21:28:20.425 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.426 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:20.426 DEBUG:boto:Port: 8000 2023-05-24T21:28:20.426 DEBUG:boto:Params: {} 2023-05-24T21:28:20.427 DEBUG:boto:Token: None 2023-05-24T21:28:20.427 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:20 GMT /qskrps-48/?versions 2023-05-24T21:28:20.428 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:nS25WVqz7dTgZM78ONQzZFpMs2c= 2023-05-24T21:28:20.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:nS25WVqz7dTgZM78ONQzZFpMs2c=', 'Content-Length': '0'} 2023-05-24T21:28:20.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058f72bc632da9cbb-00646e8174-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.431 DEBUG:boto:b'qskrps-48999falseobj1nulltrue2023-05-24T21:27:06.310Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-24T21:27:06.330Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-24T21:27:06.348Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-24T21:27:06.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-05-24T21:28:09.543Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-05-24T21:28:09.598Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-05-24T21:28:09.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-05-24T21:28:09.670Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:20.432 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-05-24T21:28:20.433 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-05-24T21:28:20.433 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-05-24T21:28:20.433 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-05-24T21:28:20.434 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-05-24T21:28:20.434 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-05-24T21:28:20.434 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-05-24T21:28:20.434 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-05-24T21:28:20.435 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:28:20.435 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:28:20.436 DEBUG:boto:path=/qskrps-48/?versions&max-keys=999 2023-05-24T21:28:20.436 DEBUG:boto:auth_path=/qskrps-48/?versions&max-keys=999 2023-05-24T21:28:20.436 DEBUG:boto:Method: GET 2023-05-24T21:28:20.437 DEBUG:boto:Path: /qskrps-48/?versions&max-keys=999 2023-05-24T21:28:20.437 DEBUG:boto:Data: 2023-05-24T21:28:20.437 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.438 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:20.438 DEBUG:boto:Port: 8001 2023-05-24T21:28:20.438 DEBUG:boto:Params: {} 2023-05-24T21:28:20.439 DEBUG:boto:Token: None 2023-05-24T21:28:20.439 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:20 GMT /qskrps-48/?versions 2023-05-24T21:28:20.440 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:nS25WVqz7dTgZM78ONQzZFpMs2c= 2023-05-24T21:28:20.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:nS25WVqz7dTgZM78ONQzZFpMs2c=', 'Content-Length': '0'} 2023-05-24T21:28:20.443 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f734b585ee23da70-00646e8174-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.443 DEBUG:boto:b'qskrps-48999falseobj1nulltrue2023-05-24T21:27:06.310Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-24T21:27:06.330Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-24T21:27:06.348Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-24T21:27:06.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-05-24T21:28:09.543Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-05-24T21:28:09.598Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-05-24T21:28:09.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-05-24T21:28:09.670Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:20.444 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:28:20.445 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:28:20.445 DEBUG:boto:path=/qskrps-48/?versions&max-keys=999 2023-05-24T21:28:20.445 DEBUG:boto:auth_path=/qskrps-48/?versions&max-keys=999 2023-05-24T21:28:20.446 DEBUG:boto:Method: GET 2023-05-24T21:28:20.446 DEBUG:boto:Path: /qskrps-48/?versions&max-keys=999 2023-05-24T21:28:20.446 DEBUG:boto:Data: 2023-05-24T21:28:20.447 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.447 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:20.447 DEBUG:boto:Port: 8000 2023-05-24T21:28:20.448 DEBUG:boto:Params: {} 2023-05-24T21:28:20.448 DEBUG:boto:Token: None 2023-05-24T21:28:20.449 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:20 GMT /qskrps-48/?versions 2023-05-24T21:28:20.449 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:nS25WVqz7dTgZM78ONQzZFpMs2c= 2023-05-24T21:28:20.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:nS25WVqz7dTgZM78ONQzZFpMs2c=', 'Content-Length': '0'} 2023-05-24T21:28:20.452 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000594fff7ec0df39eb-00646e8174-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.452 DEBUG:boto:b'qskrps-48999falseobj1nulltrue2023-05-24T21:27:06.310Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-24T21:27:06.330Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-24T21:27:06.348Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-24T21:27:06.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-05-24T21:28:09.543Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-05-24T21:28:09.598Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-05-24T21:28:09.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-05-24T21:28:09.670Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:20.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-24T21:28:20.454 DEBUG:boto:path=/qskrps-48/obj1 2023-05-24T21:28:20.454 DEBUG:boto:auth_path=/qskrps-48/obj1 2023-05-24T21:28:20.454 DEBUG:boto:path=/qskrps-48/obj1?versionId=null 2023-05-24T21:28:20.455 DEBUG:boto:auth_path=/qskrps-48/obj1?versionId=null 2023-05-24T21:28:20.455 DEBUG:boto:Method: GET 2023-05-24T21:28:20.456 DEBUG:boto:Path: /qskrps-48/obj1?versionId=null 2023-05-24T21:28:20.456 DEBUG:boto:Data: 2023-05-24T21:28:20.456 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.457 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:20.457 DEBUG:boto:Port: 8001 2023-05-24T21:28:20.457 DEBUG:boto:Params: {} 2023-05-24T21:28:20.458 DEBUG:boto:Token: None 2023-05-24T21:28:20.458 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:20 GMT /qskrps-48/obj1?versionId=null 2023-05-24T21:28:20.459 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:h2w5ZXVMO4UaQfgaDfYaCvhbx94= 2023-05-24T21:28:20.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:h2w5ZXVMO4UaQfgaDfYaCvhbx94=', 'Content-Length': '0'} 2023-05-24T21:28:20.461 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000603ef7270aae113d-00646e8174-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.461 DEBUG:boto:path=/qskrps-48/obj1 2023-05-24T21:28:20.462 DEBUG:boto:auth_path=/qskrps-48/obj1 2023-05-24T21:28:20.462 DEBUG:boto:path=/qskrps-48/obj1?versionId=null 2023-05-24T21:28:20.463 DEBUG:boto:auth_path=/qskrps-48/obj1?versionId=null 2023-05-24T21:28:20.463 DEBUG:boto:Method: GET 2023-05-24T21:28:20.463 DEBUG:boto:Path: /qskrps-48/obj1?versionId=null 2023-05-24T21:28:20.464 DEBUG:boto:Data: 2023-05-24T21:28:20.464 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.465 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:20.465 DEBUG:boto:Port: 8000 2023-05-24T21:28:20.465 DEBUG:boto:Params: {} 2023-05-24T21:28:20.466 DEBUG:boto:Token: None 2023-05-24T21:28:20.466 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:20 GMT /qskrps-48/obj1?versionId=null 2023-05-24T21:28:20.467 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:h2w5ZXVMO4UaQfgaDfYaCvhbx94= 2023-05-24T21:28:20.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:h2w5ZXVMO4UaQfgaDfYaCvhbx94=', 'Content-Length': '0'} 2023-05-24T21:28:20.469 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002a27ec0e9d56563a-00646e8174-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.469 DEBUG:boto:path=/qskrps-48/obj1 2023-05-24T21:28:20.470 DEBUG:boto:auth_path=/qskrps-48/obj1 2023-05-24T21:28:20.470 DEBUG:boto:path=/qskrps-48/obj1?versionId=null 2023-05-24T21:28:20.470 DEBUG:boto:auth_path=/qskrps-48/obj1?versionId=null 2023-05-24T21:28:20.471 DEBUG:boto:Method: HEAD 2023-05-24T21:28:20.471 DEBUG:boto:Path: /qskrps-48/obj1?versionId=null 2023-05-24T21:28:20.472 DEBUG:boto:Data: 2023-05-24T21:28:20.472 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.472 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:20.473 DEBUG:boto:Port: 8001 2023-05-24T21:28:20.473 DEBUG:boto:Params: {} 2023-05-24T21:28:20.473 DEBUG:boto:Token: None 2023-05-24T21:28:20.474 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:20 GMT /qskrps-48/obj1?versionId=null 2023-05-24T21:28:20.474 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:24s7+Afmf/21YB2Gp3ZOGKwyD5Q= 2023-05-24T21:28:20.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:24s7+Afmf/21YB2Gp3ZOGKwyD5Q=', 'Content-Length': '0'} 2023-05-24T21:28:20.476 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000950d27d0d131a56-00646e8174-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.477 DEBUG:boto:path=/qskrps-48/obj1 2023-05-24T21:28:20.477 DEBUG:boto:auth_path=/qskrps-48/obj1 2023-05-24T21:28:20.478 DEBUG:boto:path=/qskrps-48/obj1?versionId=null 2023-05-24T21:28:20.478 DEBUG:boto:auth_path=/qskrps-48/obj1?versionId=null 2023-05-24T21:28:20.478 DEBUG:boto:Method: HEAD 2023-05-24T21:28:20.479 DEBUG:boto:Path: /qskrps-48/obj1?versionId=null 2023-05-24T21:28:20.479 DEBUG:boto:Data: 2023-05-24T21:28:20.479 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.480 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:20.480 DEBUG:boto:Port: 8000 2023-05-24T21:28:20.480 DEBUG:boto:Params: {} 2023-05-24T21:28:20.481 DEBUG:boto:Token: None 2023-05-24T21:28:20.481 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:20 GMT /qskrps-48/obj1?versionId=null 2023-05-24T21:28:20.482 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:24s7+Afmf/21YB2Gp3ZOGKwyD5Q= 2023-05-24T21:28:20.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:24s7+Afmf/21YB2Gp3ZOGKwyD5Q=', 'Content-Length': '0'} 2023-05-24T21:28:20.484 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a96381dde23adf1c-00646e8174-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-24T21:28:20.485 DEBUG:boto:path=/qskrps-48/obj1 2023-05-24T21:28:20.485 DEBUG:boto:auth_path=/qskrps-48/obj1 2023-05-24T21:28:20.485 DEBUG:boto:path=/qskrps-48/obj1?versionId=null 2023-05-24T21:28:20.486 DEBUG:boto:auth_path=/qskrps-48/obj1?versionId=null 2023-05-24T21:28:20.486 DEBUG:boto:Method: GET 2023-05-24T21:28:20.486 DEBUG:boto:Path: /qskrps-48/obj1?versionId=null 2023-05-24T21:28:20.487 DEBUG:boto:Data: 2023-05-24T21:28:20.487 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.488 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:20.488 DEBUG:boto:Port: 8001 2023-05-24T21:28:20.488 DEBUG:boto:Params: {} 2023-05-24T21:28:20.489 DEBUG:boto:Token: None 2023-05-24T21:28:20.489 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:20 GMT /qskrps-48/obj1?versionId=null 2023-05-24T21:28:20.490 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:h2w5ZXVMO4UaQfgaDfYaCvhbx94= 2023-05-24T21:28:20.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:h2w5ZXVMO4UaQfgaDfYaCvhbx94=', 'Content-Length': '0'} 2023-05-24T21:28:20.492 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e360dea04c07cee-00646e8174-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.532 DEBUG:boto:path=/qskrps-48/obj1 2023-05-24T21:28:20.533 DEBUG:boto:auth_path=/qskrps-48/obj1 2023-05-24T21:28:20.534 DEBUG:boto:path=/qskrps-48/obj1?versionId=null 2023-05-24T21:28:20.534 DEBUG:boto:auth_path=/qskrps-48/obj1?versionId=null 2023-05-24T21:28:20.535 DEBUG:boto:Method: GET 2023-05-24T21:28:20.535 DEBUG:boto:Path: /qskrps-48/obj1?versionId=null 2023-05-24T21:28:20.535 DEBUG:boto:Data: 2023-05-24T21:28:20.536 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.536 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:20.536 DEBUG:boto:Port: 8000 2023-05-24T21:28:20.537 DEBUG:boto:Params: {} 2023-05-24T21:28:20.537 DEBUG:boto:Token: None 2023-05-24T21:28:20.538 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:20 GMT /qskrps-48/obj1?versionId=null 2023-05-24T21:28:20.538 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:h2w5ZXVMO4UaQfgaDfYaCvhbx94= 2023-05-24T21:28:20.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:h2w5ZXVMO4UaQfgaDfYaCvhbx94=', 'Content-Length': '0'} 2023-05-24T21:28:20.541 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f2d23bd1c5658be9-00646e8174-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.541 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:28:20.542 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:28:20.542 DEBUG:boto:path=/qskrps-48/?prefix=obj1 2023-05-24T21:28:20.543 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj1 2023-05-24T21:28:20.543 DEBUG:boto:Method: GET 2023-05-24T21:28:20.543 DEBUG:boto:Path: /qskrps-48/?prefix=obj1 2023-05-24T21:28:20.544 DEBUG:boto:Data: 2023-05-24T21:28:20.544 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.544 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:20.545 DEBUG:boto:Port: 8001 2023-05-24T21:28:20.545 DEBUG:boto:Params: {} 2023-05-24T21:28:20.546 DEBUG:boto:Token: None 2023-05-24T21:28:20.546 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:20 GMT /qskrps-48/ 2023-05-24T21:28:20.547 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:NwSOcAGxqDVYN1FOu6xegODzDj8= 2023-05-24T21:28:20.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:NwSOcAGxqDVYN1FOu6xegODzDj8=', 'Content-Length': '0'} 2023-05-24T21:28:20.550 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d07158dc413feb89-00646e8174-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.551 DEBUG:boto:b'qskrps-48obj11000falseobj12023-05-24T21:27:06.310Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:20.551 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:28:20.552 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:28:20.552 DEBUG:boto:path=/qskrps-48/?prefix=obj1 2023-05-24T21:28:20.553 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj1 2023-05-24T21:28:20.553 DEBUG:boto:Method: GET 2023-05-24T21:28:20.554 DEBUG:boto:Path: /qskrps-48/?prefix=obj1 2023-05-24T21:28:20.554 DEBUG:boto:Data: 2023-05-24T21:28:20.554 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.555 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:20.555 DEBUG:boto:Port: 8000 2023-05-24T21:28:20.556 DEBUG:boto:Params: {} 2023-05-24T21:28:20.556 DEBUG:boto:Token: None 2023-05-24T21:28:20.557 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:20 GMT /qskrps-48/ 2023-05-24T21:28:20.557 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:NwSOcAGxqDVYN1FOu6xegODzDj8= 2023-05-24T21:28:20.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:NwSOcAGxqDVYN1FOu6xegODzDj8=', 'Content-Length': '0'} 2023-05-24T21:28:20.560 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002921410567e34993-00646e8174-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.560 DEBUG:boto:b'qskrps-48obj11000falseobj12023-05-24T21:27:06.310Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:20.561 DEBUG:boto:path=/qskrps-48/obj1 2023-05-24T21:28:20.561 DEBUG:boto:auth_path=/qskrps-48/obj1 2023-05-24T21:28:20.562 DEBUG:boto:Method: HEAD 2023-05-24T21:28:20.562 DEBUG:boto:Path: /qskrps-48/obj1 2023-05-24T21:28:20.562 DEBUG:boto:Data: 2023-05-24T21:28:20.563 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.563 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:20.563 DEBUG:boto:Port: 8001 2023-05-24T21:28:20.564 DEBUG:boto:Params: {} 2023-05-24T21:28:20.564 DEBUG:boto:Token: None 2023-05-24T21:28:20.565 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:20 GMT /qskrps-48/obj1 2023-05-24T21:28:20.565 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:vZi/zvrZW9GopnvmAGOy9nAn00A= 2023-05-24T21:28:20.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:vZi/zvrZW9GopnvmAGOy9nAn00A=', 'Content-Length': '0'} 2023-05-24T21:28:20.567 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ceb53b4bfdbe6c7c-00646e8174-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.568 DEBUG:boto:path=/qskrps-48/obj1 2023-05-24T21:28:20.568 DEBUG:boto:auth_path=/qskrps-48/obj1 2023-05-24T21:28:20.569 DEBUG:boto:Method: HEAD 2023-05-24T21:28:20.569 DEBUG:boto:Path: /qskrps-48/obj1 2023-05-24T21:28:20.569 DEBUG:boto:Data: 2023-05-24T21:28:20.570 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.570 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:20.570 DEBUG:boto:Port: 8000 2023-05-24T21:28:20.571 DEBUG:boto:Params: {} 2023-05-24T21:28:20.571 DEBUG:boto:Token: None 2023-05-24T21:28:20.572 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:20 GMT /qskrps-48/obj1 2023-05-24T21:28:20.572 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:vZi/zvrZW9GopnvmAGOy9nAn00A= 2023-05-24T21:28:20.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:vZi/zvrZW9GopnvmAGOy9nAn00A=', 'Content-Length': '0'} 2023-05-24T21:28:20.574 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d4f34c860e801ba7-00646e8174-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-24T21:28:20.575 DEBUG:boto:path=/qskrps-48/obj1 2023-05-24T21:28:20.576 DEBUG:boto:auth_path=/qskrps-48/obj1 2023-05-24T21:28:20.576 DEBUG:boto:Method: GET 2023-05-24T21:28:20.577 DEBUG:boto:Path: /qskrps-48/obj1 2023-05-24T21:28:20.577 DEBUG:boto:Data: 2023-05-24T21:28:20.578 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.578 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:20.578 DEBUG:boto:Port: 8001 2023-05-24T21:28:20.579 DEBUG:boto:Params: {} 2023-05-24T21:28:20.579 DEBUG:boto:Token: None 2023-05-24T21:28:20.580 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:20 GMT /qskrps-48/obj1 2023-05-24T21:28:20.580 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:DE+4EpkIDPrBEnWMfdsVMBawZOU= 2023-05-24T21:28:20.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:DE+4EpkIDPrBEnWMfdsVMBawZOU=', 'Content-Length': '0'} 2023-05-24T21:28:20.582 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000041e7446564dabe63-00646e8174-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.583 DEBUG:boto:path=/qskrps-48/obj1 2023-05-24T21:28:20.583 DEBUG:boto:auth_path=/qskrps-48/obj1 2023-05-24T21:28:20.584 DEBUG:boto:Method: GET 2023-05-24T21:28:20.584 DEBUG:boto:Path: /qskrps-48/obj1 2023-05-24T21:28:20.584 DEBUG:boto:Data: 2023-05-24T21:28:20.585 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.585 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:20.585 DEBUG:boto:Port: 8000 2023-05-24T21:28:20.586 DEBUG:boto:Params: {} 2023-05-24T21:28:20.586 DEBUG:boto:Token: None 2023-05-24T21:28:20.587 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:20 GMT /qskrps-48/obj1 2023-05-24T21:28:20.587 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:DE+4EpkIDPrBEnWMfdsVMBawZOU= 2023-05-24T21:28:20.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:DE+4EpkIDPrBEnWMfdsVMBawZOU=', 'Content-Length': '0'} 2023-05-24T21:28:20.589 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000690906f7fb2e9fab-00646e8174-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.590 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:28:20.591 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:28:20.591 DEBUG:boto:path=/qskrps-48/?prefix=obj1 2023-05-24T21:28:20.591 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj1 2023-05-24T21:28:20.592 DEBUG:boto:Method: GET 2023-05-24T21:28:20.592 DEBUG:boto:Path: /qskrps-48/?prefix=obj1 2023-05-24T21:28:20.593 DEBUG:boto:Data: 2023-05-24T21:28:20.593 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.593 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:20.594 DEBUG:boto:Port: 8001 2023-05-24T21:28:20.594 DEBUG:boto:Params: {} 2023-05-24T21:28:20.595 DEBUG:boto:Token: None 2023-05-24T21:28:20.595 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:20 GMT /qskrps-48/ 2023-05-24T21:28:20.596 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:NwSOcAGxqDVYN1FOu6xegODzDj8= 2023-05-24T21:28:20.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:NwSOcAGxqDVYN1FOu6xegODzDj8=', 'Content-Length': '0'} 2023-05-24T21:28:20.599 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000837244db78468c52-00646e8174-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.600 DEBUG:boto:b'qskrps-48obj11000falseobj12023-05-24T21:27:06.310Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:20.600 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:28:20.601 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:28:20.601 DEBUG:boto:path=/qskrps-48/?prefix=obj1 2023-05-24T21:28:20.602 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj1 2023-05-24T21:28:20.602 DEBUG:boto:Method: GET 2023-05-24T21:28:20.602 DEBUG:boto:Path: /qskrps-48/?prefix=obj1 2023-05-24T21:28:20.603 DEBUG:boto:Data: 2023-05-24T21:28:20.603 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.603 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:20.604 DEBUG:boto:Port: 8000 2023-05-24T21:28:20.604 DEBUG:boto:Params: {} 2023-05-24T21:28:20.605 DEBUG:boto:Token: None 2023-05-24T21:28:20.605 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:20 GMT /qskrps-48/ 2023-05-24T21:28:20.605 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:NwSOcAGxqDVYN1FOu6xegODzDj8= 2023-05-24T21:28:20.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:NwSOcAGxqDVYN1FOu6xegODzDj8=', 'Content-Length': '0'} 2023-05-24T21:28:20.608 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004837a9895bd96d45-00646e8174-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.609 DEBUG:boto:b'qskrps-48obj11000falseobj12023-05-24T21:27:06.310Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:20.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-24T21:28:20.610 DEBUG:boto:path=/qskrps-48/obj2 2023-05-24T21:28:20.610 DEBUG:boto:auth_path=/qskrps-48/obj2 2023-05-24T21:28:20.610 DEBUG:boto:path=/qskrps-48/obj2?versionId=null 2023-05-24T21:28:20.611 DEBUG:boto:auth_path=/qskrps-48/obj2?versionId=null 2023-05-24T21:28:20.611 DEBUG:boto:Method: GET 2023-05-24T21:28:20.612 DEBUG:boto:Path: /qskrps-48/obj2?versionId=null 2023-05-24T21:28:20.612 DEBUG:boto:Data: 2023-05-24T21:28:20.612 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.613 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:20.613 DEBUG:boto:Port: 8001 2023-05-24T21:28:20.613 DEBUG:boto:Params: {} 2023-05-24T21:28:20.614 DEBUG:boto:Token: None 2023-05-24T21:28:20.614 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:20 GMT /qskrps-48/obj2?versionId=null 2023-05-24T21:28:20.615 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:pi/yD3TWZWHPMxtVy+d+3ZFVRLM= 2023-05-24T21:28:20.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:pi/yD3TWZWHPMxtVy+d+3ZFVRLM=', 'Content-Length': '0'} 2023-05-24T21:28:20.617 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002ee429924f3aaf3e-00646e8174-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.618 DEBUG:boto:path=/qskrps-48/obj2 2023-05-24T21:28:20.618 DEBUG:boto:auth_path=/qskrps-48/obj2 2023-05-24T21:28:20.619 DEBUG:boto:path=/qskrps-48/obj2?versionId=null 2023-05-24T21:28:20.619 DEBUG:boto:auth_path=/qskrps-48/obj2?versionId=null 2023-05-24T21:28:20.620 DEBUG:boto:Method: GET 2023-05-24T21:28:20.620 DEBUG:boto:Path: /qskrps-48/obj2?versionId=null 2023-05-24T21:28:20.621 DEBUG:boto:Data: 2023-05-24T21:28:20.622 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.622 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:20.622 DEBUG:boto:Port: 8000 2023-05-24T21:28:20.623 DEBUG:boto:Params: {} 2023-05-24T21:28:20.623 DEBUG:boto:Token: None 2023-05-24T21:28:20.624 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:20 GMT /qskrps-48/obj2?versionId=null 2023-05-24T21:28:20.624 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:pi/yD3TWZWHPMxtVy+d+3ZFVRLM= 2023-05-24T21:28:20.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:pi/yD3TWZWHPMxtVy+d+3ZFVRLM=', 'Content-Length': '0'} 2023-05-24T21:28:20.627 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003cc4c9a14410aed0-00646e8174-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.627 DEBUG:boto:path=/qskrps-48/obj2 2023-05-24T21:28:20.628 DEBUG:boto:auth_path=/qskrps-48/obj2 2023-05-24T21:28:20.628 DEBUG:boto:path=/qskrps-48/obj2?versionId=null 2023-05-24T21:28:20.628 DEBUG:boto:auth_path=/qskrps-48/obj2?versionId=null 2023-05-24T21:28:20.629 DEBUG:boto:Method: HEAD 2023-05-24T21:28:20.629 DEBUG:boto:Path: /qskrps-48/obj2?versionId=null 2023-05-24T21:28:20.629 DEBUG:boto:Data: 2023-05-24T21:28:20.630 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.630 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:20.630 DEBUG:boto:Port: 8001 2023-05-24T21:28:20.631 DEBUG:boto:Params: {} 2023-05-24T21:28:20.631 DEBUG:boto:Token: None 2023-05-24T21:28:20.632 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:20 GMT /qskrps-48/obj2?versionId=null 2023-05-24T21:28:20.632 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:oG0Z/Wj61Wu3XhzgmufzOPSy9zM= 2023-05-24T21:28:20.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:oG0Z/Wj61Wu3XhzgmufzOPSy9zM=', 'Content-Length': '0'} 2023-05-24T21:28:20.634 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ab85b8c68237ba01-00646e8174-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.635 DEBUG:boto:path=/qskrps-48/obj2 2023-05-24T21:28:20.635 DEBUG:boto:auth_path=/qskrps-48/obj2 2023-05-24T21:28:20.635 DEBUG:boto:path=/qskrps-48/obj2?versionId=null 2023-05-24T21:28:20.636 DEBUG:boto:auth_path=/qskrps-48/obj2?versionId=null 2023-05-24T21:28:20.636 DEBUG:boto:Method: HEAD 2023-05-24T21:28:20.636 DEBUG:boto:Path: /qskrps-48/obj2?versionId=null 2023-05-24T21:28:20.637 DEBUG:boto:Data: 2023-05-24T21:28:20.637 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.637 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:20.638 DEBUG:boto:Port: 8000 2023-05-24T21:28:20.638 DEBUG:boto:Params: {} 2023-05-24T21:28:20.638 DEBUG:boto:Token: None 2023-05-24T21:28:20.639 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:20 GMT /qskrps-48/obj2?versionId=null 2023-05-24T21:28:20.639 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:oG0Z/Wj61Wu3XhzgmufzOPSy9zM= 2023-05-24T21:28:20.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:oG0Z/Wj61Wu3XhzgmufzOPSy9zM=', 'Content-Length': '0'} 2023-05-24T21:28:20.641 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000be5d2f17b60bb564-00646e8174-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-24T21:28:20.642 DEBUG:boto:path=/qskrps-48/obj2 2023-05-24T21:28:20.643 DEBUG:boto:auth_path=/qskrps-48/obj2 2023-05-24T21:28:20.643 DEBUG:boto:path=/qskrps-48/obj2?versionId=null 2023-05-24T21:28:20.644 DEBUG:boto:auth_path=/qskrps-48/obj2?versionId=null 2023-05-24T21:28:20.644 DEBUG:boto:Method: GET 2023-05-24T21:28:20.644 DEBUG:boto:Path: /qskrps-48/obj2?versionId=null 2023-05-24T21:28:20.645 DEBUG:boto:Data: 2023-05-24T21:28:20.645 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.645 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:20.646 DEBUG:boto:Port: 8001 2023-05-24T21:28:20.646 DEBUG:boto:Params: {} 2023-05-24T21:28:20.647 DEBUG:boto:Token: None 2023-05-24T21:28:20.647 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:20 GMT /qskrps-48/obj2?versionId=null 2023-05-24T21:28:20.647 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:pi/yD3TWZWHPMxtVy+d+3ZFVRLM= 2023-05-24T21:28:20.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:pi/yD3TWZWHPMxtVy+d+3ZFVRLM=', 'Content-Length': '0'} 2023-05-24T21:28:20.650 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009373685aadc4dc72-00646e8174-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.651 DEBUG:boto:path=/qskrps-48/obj2 2023-05-24T21:28:20.651 DEBUG:boto:auth_path=/qskrps-48/obj2 2023-05-24T21:28:20.652 DEBUG:boto:path=/qskrps-48/obj2?versionId=null 2023-05-24T21:28:20.652 DEBUG:boto:auth_path=/qskrps-48/obj2?versionId=null 2023-05-24T21:28:20.652 DEBUG:boto:Method: GET 2023-05-24T21:28:20.653 DEBUG:boto:Path: /qskrps-48/obj2?versionId=null 2023-05-24T21:28:20.653 DEBUG:boto:Data: 2023-05-24T21:28:20.653 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.654 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:20.654 DEBUG:boto:Port: 8000 2023-05-24T21:28:20.654 DEBUG:boto:Params: {} 2023-05-24T21:28:20.655 DEBUG:boto:Token: None 2023-05-24T21:28:20.655 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:20 GMT /qskrps-48/obj2?versionId=null 2023-05-24T21:28:20.656 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:pi/yD3TWZWHPMxtVy+d+3ZFVRLM= 2023-05-24T21:28:20.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:pi/yD3TWZWHPMxtVy+d+3ZFVRLM=', 'Content-Length': '0'} 2023-05-24T21:28:20.658 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002cc00d03362cfd10-00646e8174-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.659 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:28:20.659 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:28:20.660 DEBUG:boto:path=/qskrps-48/?prefix=obj2 2023-05-24T21:28:20.660 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj2 2023-05-24T21:28:20.660 DEBUG:boto:Method: GET 2023-05-24T21:28:20.661 DEBUG:boto:Path: /qskrps-48/?prefix=obj2 2023-05-24T21:28:20.661 DEBUG:boto:Data: 2023-05-24T21:28:20.661 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.662 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:20.662 DEBUG:boto:Port: 8001 2023-05-24T21:28:20.662 DEBUG:boto:Params: {} 2023-05-24T21:28:20.663 DEBUG:boto:Token: None 2023-05-24T21:28:20.663 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:20 GMT /qskrps-48/ 2023-05-24T21:28:20.664 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:NwSOcAGxqDVYN1FOu6xegODzDj8= 2023-05-24T21:28:20.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:NwSOcAGxqDVYN1FOu6xegODzDj8=', 'Content-Length': '0'} 2023-05-24T21:28:20.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a024046038a91533-00646e8174-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.667 DEBUG:boto:b'qskrps-48obj21000falseobj22023-05-24T21:27:06.330Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:20.668 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:28:20.668 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:28:20.669 DEBUG:boto:path=/qskrps-48/?prefix=obj2 2023-05-24T21:28:20.669 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj2 2023-05-24T21:28:20.669 DEBUG:boto:Method: GET 2023-05-24T21:28:20.670 DEBUG:boto:Path: /qskrps-48/?prefix=obj2 2023-05-24T21:28:20.670 DEBUG:boto:Data: 2023-05-24T21:28:20.671 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.671 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:20.672 DEBUG:boto:Port: 8000 2023-05-24T21:28:20.672 DEBUG:boto:Params: {} 2023-05-24T21:28:20.672 DEBUG:boto:Token: None 2023-05-24T21:28:20.673 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:20 GMT /qskrps-48/ 2023-05-24T21:28:20.673 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:NwSOcAGxqDVYN1FOu6xegODzDj8= 2023-05-24T21:28:20.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:NwSOcAGxqDVYN1FOu6xegODzDj8=', 'Content-Length': '0'} 2023-05-24T21:28:20.676 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8832e8056bab1a1-00646e8174-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.677 DEBUG:boto:b'qskrps-48obj21000falseobj22023-05-24T21:27:06.330Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:20.677 DEBUG:boto:path=/qskrps-48/obj2 2023-05-24T21:28:20.678 DEBUG:boto:auth_path=/qskrps-48/obj2 2023-05-24T21:28:20.678 DEBUG:boto:Method: HEAD 2023-05-24T21:28:20.679 DEBUG:boto:Path: /qskrps-48/obj2 2023-05-24T21:28:20.679 DEBUG:boto:Data: 2023-05-24T21:28:20.679 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.680 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:20.680 DEBUG:boto:Port: 8001 2023-05-24T21:28:20.681 DEBUG:boto:Params: {} 2023-05-24T21:28:20.681 DEBUG:boto:Token: None 2023-05-24T21:28:20.681 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:20 GMT /qskrps-48/obj2 2023-05-24T21:28:20.682 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:lNmN0MRHKbQ8zePeTKlIogjZ/58= 2023-05-24T21:28:20.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:lNmN0MRHKbQ8zePeTKlIogjZ/58=', 'Content-Length': '0'} 2023-05-24T21:28:20.684 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a605b6776a089c87-00646e8174-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.685 DEBUG:boto:path=/qskrps-48/obj2 2023-05-24T21:28:20.685 DEBUG:boto:auth_path=/qskrps-48/obj2 2023-05-24T21:28:20.686 DEBUG:boto:Method: HEAD 2023-05-24T21:28:20.686 DEBUG:boto:Path: /qskrps-48/obj2 2023-05-24T21:28:20.686 DEBUG:boto:Data: 2023-05-24T21:28:20.687 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.687 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:20.687 DEBUG:boto:Port: 8000 2023-05-24T21:28:20.688 DEBUG:boto:Params: {} 2023-05-24T21:28:20.688 DEBUG:boto:Token: None 2023-05-24T21:28:20.689 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:20 GMT /qskrps-48/obj2 2023-05-24T21:28:20.689 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:lNmN0MRHKbQ8zePeTKlIogjZ/58= 2023-05-24T21:28:20.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:lNmN0MRHKbQ8zePeTKlIogjZ/58=', 'Content-Length': '0'} 2023-05-24T21:28:20.691 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dcf2fb07b4b51b70-00646e8174-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-24T21:28:20.692 DEBUG:boto:path=/qskrps-48/obj2 2023-05-24T21:28:20.692 DEBUG:boto:auth_path=/qskrps-48/obj2 2023-05-24T21:28:20.693 DEBUG:boto:Method: GET 2023-05-24T21:28:20.693 DEBUG:boto:Path: /qskrps-48/obj2 2023-05-24T21:28:20.693 DEBUG:boto:Data: 2023-05-24T21:28:20.694 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.694 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:20.695 DEBUG:boto:Port: 8001 2023-05-24T21:28:20.695 DEBUG:boto:Params: {} 2023-05-24T21:28:20.695 DEBUG:boto:Token: None 2023-05-24T21:28:20.696 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:20 GMT /qskrps-48/obj2 2023-05-24T21:28:20.696 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Six9yTnP5yPW14/Ivvf+DNMbV54= 2023-05-24T21:28:20.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Six9yTnP5yPW14/Ivvf+DNMbV54=', 'Content-Length': '0'} 2023-05-24T21:28:20.698 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000067a4c03343fb5a4b-00646e8174-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.740 DEBUG:boto:path=/qskrps-48/obj2 2023-05-24T21:28:20.741 DEBUG:boto:auth_path=/qskrps-48/obj2 2023-05-24T21:28:20.742 DEBUG:boto:Method: GET 2023-05-24T21:28:20.742 DEBUG:boto:Path: /qskrps-48/obj2 2023-05-24T21:28:20.742 DEBUG:boto:Data: 2023-05-24T21:28:20.743 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.743 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:20.744 DEBUG:boto:Port: 8000 2023-05-24T21:28:20.744 DEBUG:boto:Params: {} 2023-05-24T21:28:20.744 DEBUG:boto:Token: None 2023-05-24T21:28:20.745 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:20 GMT /qskrps-48/obj2 2023-05-24T21:28:20.745 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Six9yTnP5yPW14/Ivvf+DNMbV54= 2023-05-24T21:28:20.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Six9yTnP5yPW14/Ivvf+DNMbV54=', 'Content-Length': '0'} 2023-05-24T21:28:20.748 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b00ccf6771e2292-00646e8174-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.748 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:28:20.749 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:28:20.749 DEBUG:boto:path=/qskrps-48/?prefix=obj2 2023-05-24T21:28:20.750 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj2 2023-05-24T21:28:20.750 DEBUG:boto:Method: GET 2023-05-24T21:28:20.750 DEBUG:boto:Path: /qskrps-48/?prefix=obj2 2023-05-24T21:28:20.751 DEBUG:boto:Data: 2023-05-24T21:28:20.751 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.751 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:20.752 DEBUG:boto:Port: 8001 2023-05-24T21:28:20.752 DEBUG:boto:Params: {} 2023-05-24T21:28:20.753 DEBUG:boto:Token: None 2023-05-24T21:28:20.753 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:20 GMT /qskrps-48/ 2023-05-24T21:28:20.753 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:NwSOcAGxqDVYN1FOu6xegODzDj8= 2023-05-24T21:28:20.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:NwSOcAGxqDVYN1FOu6xegODzDj8=', 'Content-Length': '0'} 2023-05-24T21:28:20.756 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ae26e7843fd0c90-00646e8174-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.757 DEBUG:boto:b'qskrps-48obj21000falseobj22023-05-24T21:27:06.330Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:20.757 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:28:20.758 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:28:20.758 DEBUG:boto:path=/qskrps-48/?prefix=obj2 2023-05-24T21:28:20.759 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj2 2023-05-24T21:28:20.759 DEBUG:boto:Method: GET 2023-05-24T21:28:20.760 DEBUG:boto:Path: /qskrps-48/?prefix=obj2 2023-05-24T21:28:20.760 DEBUG:boto:Data: 2023-05-24T21:28:20.760 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.761 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:20.761 DEBUG:boto:Port: 8000 2023-05-24T21:28:20.762 DEBUG:boto:Params: {} 2023-05-24T21:28:20.762 DEBUG:boto:Token: None 2023-05-24T21:28:20.763 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:20 GMT /qskrps-48/ 2023-05-24T21:28:20.763 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:NwSOcAGxqDVYN1FOu6xegODzDj8= 2023-05-24T21:28:20.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:NwSOcAGxqDVYN1FOu6xegODzDj8=', 'Content-Length': '0'} 2023-05-24T21:28:20.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005dafe715181a3369-00646e8174-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.767 DEBUG:boto:b'qskrps-48obj21000falseobj22023-05-24T21:27:06.330Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:20.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-24T21:28:20.768 DEBUG:boto:path=/qskrps-48/obj3 2023-05-24T21:28:20.769 DEBUG:boto:auth_path=/qskrps-48/obj3 2023-05-24T21:28:20.769 DEBUG:boto:path=/qskrps-48/obj3?versionId=null 2023-05-24T21:28:20.769 DEBUG:boto:auth_path=/qskrps-48/obj3?versionId=null 2023-05-24T21:28:20.770 DEBUG:boto:Method: GET 2023-05-24T21:28:20.770 DEBUG:boto:Path: /qskrps-48/obj3?versionId=null 2023-05-24T21:28:20.770 DEBUG:boto:Data: 2023-05-24T21:28:20.771 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.771 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:20.772 DEBUG:boto:Port: 8001 2023-05-24T21:28:20.772 DEBUG:boto:Params: {} 2023-05-24T21:28:20.772 DEBUG:boto:Token: None 2023-05-24T21:28:20.773 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:20 GMT /qskrps-48/obj3?versionId=null 2023-05-24T21:28:20.773 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:/yoRwGyvzwExCOvX3Mmt9PQ6zjY= 2023-05-24T21:28:20.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:/yoRwGyvzwExCOvX3Mmt9PQ6zjY=', 'Content-Length': '0'} 2023-05-24T21:28:20.776 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000215710b2b0b6d097-00646e8174-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.776 DEBUG:boto:path=/qskrps-48/obj3 2023-05-24T21:28:20.777 DEBUG:boto:auth_path=/qskrps-48/obj3 2023-05-24T21:28:20.777 DEBUG:boto:path=/qskrps-48/obj3?versionId=null 2023-05-24T21:28:20.777 DEBUG:boto:auth_path=/qskrps-48/obj3?versionId=null 2023-05-24T21:28:20.778 DEBUG:boto:Method: GET 2023-05-24T21:28:20.778 DEBUG:boto:Path: /qskrps-48/obj3?versionId=null 2023-05-24T21:28:20.778 DEBUG:boto:Data: 2023-05-24T21:28:20.779 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.779 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:20.780 DEBUG:boto:Port: 8000 2023-05-24T21:28:20.780 DEBUG:boto:Params: {} 2023-05-24T21:28:20.780 DEBUG:boto:Token: None 2023-05-24T21:28:20.781 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:20 GMT /qskrps-48/obj3?versionId=null 2023-05-24T21:28:20.781 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:/yoRwGyvzwExCOvX3Mmt9PQ6zjY= 2023-05-24T21:28:20.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:/yoRwGyvzwExCOvX3Mmt9PQ6zjY=', 'Content-Length': '0'} 2023-05-24T21:28:20.783 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000054098e562e9a83a7-00646e8174-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.784 DEBUG:boto:path=/qskrps-48/obj3 2023-05-24T21:28:20.784 DEBUG:boto:auth_path=/qskrps-48/obj3 2023-05-24T21:28:20.785 DEBUG:boto:path=/qskrps-48/obj3?versionId=null 2023-05-24T21:28:20.785 DEBUG:boto:auth_path=/qskrps-48/obj3?versionId=null 2023-05-24T21:28:20.785 DEBUG:boto:Method: HEAD 2023-05-24T21:28:20.786 DEBUG:boto:Path: /qskrps-48/obj3?versionId=null 2023-05-24T21:28:20.786 DEBUG:boto:Data: 2023-05-24T21:28:20.786 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.787 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:20.787 DEBUG:boto:Port: 8001 2023-05-24T21:28:20.787 DEBUG:boto:Params: {} 2023-05-24T21:28:20.788 DEBUG:boto:Token: None 2023-05-24T21:28:20.788 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:20 GMT /qskrps-48/obj3?versionId=null 2023-05-24T21:28:20.789 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:MQAwT8PazatmaoWq4Hs2ki0OXec= 2023-05-24T21:28:20.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:MQAwT8PazatmaoWq4Hs2ki0OXec=', 'Content-Length': '0'} 2023-05-24T21:28:20.791 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004e7b726adb3d2b07-00646e8174-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.792 DEBUG:boto:path=/qskrps-48/obj3 2023-05-24T21:28:20.792 DEBUG:boto:auth_path=/qskrps-48/obj3 2023-05-24T21:28:20.792 DEBUG:boto:path=/qskrps-48/obj3?versionId=null 2023-05-24T21:28:20.793 DEBUG:boto:auth_path=/qskrps-48/obj3?versionId=null 2023-05-24T21:28:20.793 DEBUG:boto:Method: HEAD 2023-05-24T21:28:20.793 DEBUG:boto:Path: /qskrps-48/obj3?versionId=null 2023-05-24T21:28:20.794 DEBUG:boto:Data: 2023-05-24T21:28:20.794 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.794 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:20.795 DEBUG:boto:Port: 8000 2023-05-24T21:28:20.795 DEBUG:boto:Params: {} 2023-05-24T21:28:20.795 DEBUG:boto:Token: None 2023-05-24T21:28:20.796 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:20 GMT /qskrps-48/obj3?versionId=null 2023-05-24T21:28:20.796 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:MQAwT8PazatmaoWq4Hs2ki0OXec= 2023-05-24T21:28:20.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:MQAwT8PazatmaoWq4Hs2ki0OXec=', 'Content-Length': '0'} 2023-05-24T21:28:20.798 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bcde249dc4f63e97-00646e8174-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-24T21:28:20.799 DEBUG:boto:path=/qskrps-48/obj3 2023-05-24T21:28:20.800 DEBUG:boto:auth_path=/qskrps-48/obj3 2023-05-24T21:28:20.800 DEBUG:boto:path=/qskrps-48/obj3?versionId=null 2023-05-24T21:28:20.800 DEBUG:boto:auth_path=/qskrps-48/obj3?versionId=null 2023-05-24T21:28:20.801 DEBUG:boto:Method: GET 2023-05-24T21:28:20.801 DEBUG:boto:Path: /qskrps-48/obj3?versionId=null 2023-05-24T21:28:20.802 DEBUG:boto:Data: 2023-05-24T21:28:20.802 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.802 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:20.803 DEBUG:boto:Port: 8001 2023-05-24T21:28:20.803 DEBUG:boto:Params: {} 2023-05-24T21:28:20.803 DEBUG:boto:Token: None 2023-05-24T21:28:20.804 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:20 GMT /qskrps-48/obj3?versionId=null 2023-05-24T21:28:20.804 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:/yoRwGyvzwExCOvX3Mmt9PQ6zjY= 2023-05-24T21:28:20.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:/yoRwGyvzwExCOvX3Mmt9PQ6zjY=', 'Content-Length': '0'} 2023-05-24T21:28:20.807 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e4630c8db411c5cd-00646e8174-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.807 DEBUG:boto:path=/qskrps-48/obj3 2023-05-24T21:28:20.808 DEBUG:boto:auth_path=/qskrps-48/obj3 2023-05-24T21:28:20.808 DEBUG:boto:path=/qskrps-48/obj3?versionId=null 2023-05-24T21:28:20.808 DEBUG:boto:auth_path=/qskrps-48/obj3?versionId=null 2023-05-24T21:28:20.809 DEBUG:boto:Method: GET 2023-05-24T21:28:20.809 DEBUG:boto:Path: /qskrps-48/obj3?versionId=null 2023-05-24T21:28:20.810 DEBUG:boto:Data: 2023-05-24T21:28:20.810 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.810 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:20.811 DEBUG:boto:Port: 8000 2023-05-24T21:28:20.811 DEBUG:boto:Params: {} 2023-05-24T21:28:20.811 DEBUG:boto:Token: None 2023-05-24T21:28:20.812 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:20 GMT /qskrps-48/obj3?versionId=null 2023-05-24T21:28:20.812 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:/yoRwGyvzwExCOvX3Mmt9PQ6zjY= 2023-05-24T21:28:20.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:/yoRwGyvzwExCOvX3Mmt9PQ6zjY=', 'Content-Length': '0'} 2023-05-24T21:28:20.815 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d14b38b07dc41bb0-00646e8174-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.815 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:28:20.816 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:28:20.816 DEBUG:boto:path=/qskrps-48/?prefix=obj3 2023-05-24T21:28:20.816 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj3 2023-05-24T21:28:20.817 DEBUG:boto:Method: GET 2023-05-24T21:28:20.817 DEBUG:boto:Path: /qskrps-48/?prefix=obj3 2023-05-24T21:28:20.818 DEBUG:boto:Data: 2023-05-24T21:28:20.818 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.818 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:20.819 DEBUG:boto:Port: 8001 2023-05-24T21:28:20.819 DEBUG:boto:Params: {} 2023-05-24T21:28:20.819 DEBUG:boto:Token: None 2023-05-24T21:28:20.820 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:20 GMT /qskrps-48/ 2023-05-24T21:28:20.820 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:NwSOcAGxqDVYN1FOu6xegODzDj8= 2023-05-24T21:28:20.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:NwSOcAGxqDVYN1FOu6xegODzDj8=', 'Content-Length': '0'} 2023-05-24T21:28:20.823 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073d12c7e043d1576-00646e8174-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.824 DEBUG:boto:b'qskrps-48obj31000falseobj32023-05-24T21:27:06.348Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:20.824 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:28:20.825 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:28:20.825 DEBUG:boto:path=/qskrps-48/?prefix=obj3 2023-05-24T21:28:20.826 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj3 2023-05-24T21:28:20.826 DEBUG:boto:Method: GET 2023-05-24T21:28:20.826 DEBUG:boto:Path: /qskrps-48/?prefix=obj3 2023-05-24T21:28:20.827 DEBUG:boto:Data: 2023-05-24T21:28:20.827 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.827 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:20.828 DEBUG:boto:Port: 8000 2023-05-24T21:28:20.828 DEBUG:boto:Params: {} 2023-05-24T21:28:20.828 DEBUG:boto:Token: None 2023-05-24T21:28:20.829 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:20 GMT /qskrps-48/ 2023-05-24T21:28:20.829 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:NwSOcAGxqDVYN1FOu6xegODzDj8= 2023-05-24T21:28:20.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:NwSOcAGxqDVYN1FOu6xegODzDj8=', 'Content-Length': '0'} 2023-05-24T21:28:20.832 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1dce15a512ce53b-00646e8174-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.833 DEBUG:boto:b'qskrps-48obj31000falseobj32023-05-24T21:27:06.348Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:20.833 DEBUG:boto:path=/qskrps-48/obj3 2023-05-24T21:28:20.834 DEBUG:boto:auth_path=/qskrps-48/obj3 2023-05-24T21:28:20.834 DEBUG:boto:Method: HEAD 2023-05-24T21:28:20.834 DEBUG:boto:Path: /qskrps-48/obj3 2023-05-24T21:28:20.835 DEBUG:boto:Data: 2023-05-24T21:28:20.835 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.835 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:20.836 DEBUG:boto:Port: 8001 2023-05-24T21:28:20.836 DEBUG:boto:Params: {} 2023-05-24T21:28:20.836 DEBUG:boto:Token: None 2023-05-24T21:28:20.837 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:20 GMT /qskrps-48/obj3 2023-05-24T21:28:20.837 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:CO/LNULnTy36Bywy0n+E7kMRrAg= 2023-05-24T21:28:20.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:CO/LNULnTy36Bywy0n+E7kMRrAg=', 'Content-Length': '0'} 2023-05-24T21:28:20.839 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b0db3575cbbb98b5-00646e8174-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.840 DEBUG:boto:path=/qskrps-48/obj3 2023-05-24T21:28:20.840 DEBUG:boto:auth_path=/qskrps-48/obj3 2023-05-24T21:28:20.841 DEBUG:boto:Method: HEAD 2023-05-24T21:28:20.841 DEBUG:boto:Path: /qskrps-48/obj3 2023-05-24T21:28:20.841 DEBUG:boto:Data: 2023-05-24T21:28:20.842 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.842 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:20.842 DEBUG:boto:Port: 8000 2023-05-24T21:28:20.843 DEBUG:boto:Params: {} 2023-05-24T21:28:20.843 DEBUG:boto:Token: None 2023-05-24T21:28:20.843 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:20 GMT /qskrps-48/obj3 2023-05-24T21:28:20.844 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:CO/LNULnTy36Bywy0n+E7kMRrAg= 2023-05-24T21:28:20.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:CO/LNULnTy36Bywy0n+E7kMRrAg=', 'Content-Length': '0'} 2023-05-24T21:28:20.846 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000603f38b5ac4cefee-00646e8174-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-24T21:28:20.847 DEBUG:boto:path=/qskrps-48/obj3 2023-05-24T21:28:20.847 DEBUG:boto:auth_path=/qskrps-48/obj3 2023-05-24T21:28:20.848 DEBUG:boto:Method: GET 2023-05-24T21:28:20.848 DEBUG:boto:Path: /qskrps-48/obj3 2023-05-24T21:28:20.848 DEBUG:boto:Data: 2023-05-24T21:28:20.849 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.849 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:20.849 DEBUG:boto:Port: 8001 2023-05-24T21:28:20.850 DEBUG:boto:Params: {} 2023-05-24T21:28:20.850 DEBUG:boto:Token: None 2023-05-24T21:28:20.851 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:20 GMT /qskrps-48/obj3 2023-05-24T21:28:20.851 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Sa+QVNaLr9qvsvP3KfUIpMEEC2M= 2023-05-24T21:28:20.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Sa+QVNaLr9qvsvP3KfUIpMEEC2M=', 'Content-Length': '0'} 2023-05-24T21:28:20.853 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004fa3ab8ec61b1c9f-00646e8174-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.854 DEBUG:boto:path=/qskrps-48/obj3 2023-05-24T21:28:20.854 DEBUG:boto:auth_path=/qskrps-48/obj3 2023-05-24T21:28:20.855 DEBUG:boto:Method: GET 2023-05-24T21:28:20.855 DEBUG:boto:Path: /qskrps-48/obj3 2023-05-24T21:28:20.856 DEBUG:boto:Data: 2023-05-24T21:28:20.856 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.856 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:20.857 DEBUG:boto:Port: 8000 2023-05-24T21:28:20.857 DEBUG:boto:Params: {} 2023-05-24T21:28:20.858 DEBUG:boto:Token: None 2023-05-24T21:28:20.858 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:20 GMT /qskrps-48/obj3 2023-05-24T21:28:20.858 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Sa+QVNaLr9qvsvP3KfUIpMEEC2M= 2023-05-24T21:28:20.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Sa+QVNaLr9qvsvP3KfUIpMEEC2M=', 'Content-Length': '0'} 2023-05-24T21:28:20.860 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000beff1a1bc01cb13-00646e8174-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.861 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:28:20.862 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:28:20.862 DEBUG:boto:path=/qskrps-48/?prefix=obj3 2023-05-24T21:28:20.862 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj3 2023-05-24T21:28:20.863 DEBUG:boto:Method: GET 2023-05-24T21:28:20.863 DEBUG:boto:Path: /qskrps-48/?prefix=obj3 2023-05-24T21:28:20.863 DEBUG:boto:Data: 2023-05-24T21:28:20.864 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.864 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:20.864 DEBUG:boto:Port: 8001 2023-05-24T21:28:20.865 DEBUG:boto:Params: {} 2023-05-24T21:28:20.865 DEBUG:boto:Token: None 2023-05-24T21:28:20.866 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:20 GMT /qskrps-48/ 2023-05-24T21:28:20.866 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:NwSOcAGxqDVYN1FOu6xegODzDj8= 2023-05-24T21:28:20.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:NwSOcAGxqDVYN1FOu6xegODzDj8=', 'Content-Length': '0'} 2023-05-24T21:28:20.869 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007204166a0ca875d5-00646e8174-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.869 DEBUG:boto:b'qskrps-48obj31000falseobj32023-05-24T21:27:06.348Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:20.870 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:28:20.870 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:28:20.871 DEBUG:boto:path=/qskrps-48/?prefix=obj3 2023-05-24T21:28:20.871 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj3 2023-05-24T21:28:20.871 DEBUG:boto:Method: GET 2023-05-24T21:28:20.872 DEBUG:boto:Path: /qskrps-48/?prefix=obj3 2023-05-24T21:28:20.872 DEBUG:boto:Data: 2023-05-24T21:28:20.872 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.873 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:20.873 DEBUG:boto:Port: 8000 2023-05-24T21:28:20.874 DEBUG:boto:Params: {} 2023-05-24T21:28:20.874 DEBUG:boto:Token: None 2023-05-24T21:28:20.874 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:20 GMT /qskrps-48/ 2023-05-24T21:28:20.875 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:NwSOcAGxqDVYN1FOu6xegODzDj8= 2023-05-24T21:28:20.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:NwSOcAGxqDVYN1FOu6xegODzDj8=', 'Content-Length': '0'} 2023-05-24T21:28:20.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1890fcf74b98a5e-00646e8174-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.920 DEBUG:boto:b'qskrps-48obj31000falseobj32023-05-24T21:27:06.348Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:20.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-24T21:28:20.922 DEBUG:boto:path=/qskrps-48/obj4 2023-05-24T21:28:20.922 DEBUG:boto:auth_path=/qskrps-48/obj4 2023-05-24T21:28:20.923 DEBUG:boto:path=/qskrps-48/obj4?versionId=null 2023-05-24T21:28:20.923 DEBUG:boto:auth_path=/qskrps-48/obj4?versionId=null 2023-05-24T21:28:20.923 DEBUG:boto:Method: GET 2023-05-24T21:28:20.924 DEBUG:boto:Path: /qskrps-48/obj4?versionId=null 2023-05-24T21:28:20.924 DEBUG:boto:Data: 2023-05-24T21:28:20.924 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.925 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:20.925 DEBUG:boto:Port: 8001 2023-05-24T21:28:20.926 DEBUG:boto:Params: {} 2023-05-24T21:28:20.926 DEBUG:boto:Token: None 2023-05-24T21:28:20.926 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:20 GMT /qskrps-48/obj4?versionId=null 2023-05-24T21:28:20.927 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:IszhZ4DmGPGlbIw74UWOJEueSzQ= 2023-05-24T21:28:20.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:IszhZ4DmGPGlbIw74UWOJEueSzQ=', 'Content-Length': '0'} 2023-05-24T21:28:20.929 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1f8eec35426d959-00646e8174-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.930 DEBUG:boto:path=/qskrps-48/obj4 2023-05-24T21:28:20.930 DEBUG:boto:auth_path=/qskrps-48/obj4 2023-05-24T21:28:20.931 DEBUG:boto:path=/qskrps-48/obj4?versionId=null 2023-05-24T21:28:20.931 DEBUG:boto:auth_path=/qskrps-48/obj4?versionId=null 2023-05-24T21:28:20.932 DEBUG:boto:Method: GET 2023-05-24T21:28:20.932 DEBUG:boto:Path: /qskrps-48/obj4?versionId=null 2023-05-24T21:28:20.932 DEBUG:boto:Data: 2023-05-24T21:28:20.933 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.933 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:20.933 DEBUG:boto:Port: 8000 2023-05-24T21:28:20.934 DEBUG:boto:Params: {} 2023-05-24T21:28:20.934 DEBUG:boto:Token: None 2023-05-24T21:28:20.935 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:20 GMT /qskrps-48/obj4?versionId=null 2023-05-24T21:28:20.935 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:IszhZ4DmGPGlbIw74UWOJEueSzQ= 2023-05-24T21:28:20.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:IszhZ4DmGPGlbIw74UWOJEueSzQ=', 'Content-Length': '0'} 2023-05-24T21:28:20.937 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cb46b93d733f6546-00646e8174-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.938 DEBUG:boto:path=/qskrps-48/obj4 2023-05-24T21:28:20.938 DEBUG:boto:auth_path=/qskrps-48/obj4 2023-05-24T21:28:20.939 DEBUG:boto:path=/qskrps-48/obj4?versionId=null 2023-05-24T21:28:20.939 DEBUG:boto:auth_path=/qskrps-48/obj4?versionId=null 2023-05-24T21:28:20.940 DEBUG:boto:Method: HEAD 2023-05-24T21:28:20.940 DEBUG:boto:Path: /qskrps-48/obj4?versionId=null 2023-05-24T21:28:20.940 DEBUG:boto:Data: 2023-05-24T21:28:20.941 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.941 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:20.941 DEBUG:boto:Port: 8001 2023-05-24T21:28:20.942 DEBUG:boto:Params: {} 2023-05-24T21:28:20.942 DEBUG:boto:Token: None 2023-05-24T21:28:20.942 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:20 GMT /qskrps-48/obj4?versionId=null 2023-05-24T21:28:20.943 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:hk6bYmYPa6TrkCQo4okU3CstGbM= 2023-05-24T21:28:20.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:hk6bYmYPa6TrkCQo4okU3CstGbM=', 'Content-Length': '0'} 2023-05-24T21:28:20.945 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c5685cc977a949b9-00646e8174-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.946 DEBUG:boto:path=/qskrps-48/obj4 2023-05-24T21:28:20.946 DEBUG:boto:auth_path=/qskrps-48/obj4 2023-05-24T21:28:20.947 DEBUG:boto:path=/qskrps-48/obj4?versionId=null 2023-05-24T21:28:20.947 DEBUG:boto:auth_path=/qskrps-48/obj4?versionId=null 2023-05-24T21:28:20.948 DEBUG:boto:Method: HEAD 2023-05-24T21:28:20.948 DEBUG:boto:Path: /qskrps-48/obj4?versionId=null 2023-05-24T21:28:20.948 DEBUG:boto:Data: 2023-05-24T21:28:20.949 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.949 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:20.950 DEBUG:boto:Port: 8000 2023-05-24T21:28:20.950 DEBUG:boto:Params: {} 2023-05-24T21:28:20.951 DEBUG:boto:Token: None 2023-05-24T21:28:20.951 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:20 GMT /qskrps-48/obj4?versionId=null 2023-05-24T21:28:20.952 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:hk6bYmYPa6TrkCQo4okU3CstGbM= 2023-05-24T21:28:20.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:hk6bYmYPa6TrkCQo4okU3CstGbM=', 'Content-Length': '0'} 2023-05-24T21:28:20.954 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000461faf2b7bcd3399-00646e8174-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-24T21:28:20.955 DEBUG:boto:path=/qskrps-48/obj4 2023-05-24T21:28:20.956 DEBUG:boto:auth_path=/qskrps-48/obj4 2023-05-24T21:28:20.956 DEBUG:boto:path=/qskrps-48/obj4?versionId=null 2023-05-24T21:28:20.956 DEBUG:boto:auth_path=/qskrps-48/obj4?versionId=null 2023-05-24T21:28:20.957 DEBUG:boto:Method: GET 2023-05-24T21:28:20.957 DEBUG:boto:Path: /qskrps-48/obj4?versionId=null 2023-05-24T21:28:20.957 DEBUG:boto:Data: 2023-05-24T21:28:20.958 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.958 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:20.959 DEBUG:boto:Port: 8001 2023-05-24T21:28:20.959 DEBUG:boto:Params: {} 2023-05-24T21:28:20.959 DEBUG:boto:Token: None 2023-05-24T21:28:20.960 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:20 GMT /qskrps-48/obj4?versionId=null 2023-05-24T21:28:20.960 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:IszhZ4DmGPGlbIw74UWOJEueSzQ= 2023-05-24T21:28:20.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:IszhZ4DmGPGlbIw74UWOJEueSzQ=', 'Content-Length': '0'} 2023-05-24T21:28:20.962 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000057b69c82cce1a76f-00646e8174-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.963 DEBUG:boto:path=/qskrps-48/obj4 2023-05-24T21:28:20.963 DEBUG:boto:auth_path=/qskrps-48/obj4 2023-05-24T21:28:20.964 DEBUG:boto:path=/qskrps-48/obj4?versionId=null 2023-05-24T21:28:20.964 DEBUG:boto:auth_path=/qskrps-48/obj4?versionId=null 2023-05-24T21:28:20.965 DEBUG:boto:Method: GET 2023-05-24T21:28:20.965 DEBUG:boto:Path: /qskrps-48/obj4?versionId=null 2023-05-24T21:28:20.966 DEBUG:boto:Data: 2023-05-24T21:28:20.966 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.966 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:20.967 DEBUG:boto:Port: 8000 2023-05-24T21:28:20.967 DEBUG:boto:Params: {} 2023-05-24T21:28:20.968 DEBUG:boto:Token: None 2023-05-24T21:28:20.968 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:20 GMT /qskrps-48/obj4?versionId=null 2023-05-24T21:28:20.969 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:IszhZ4DmGPGlbIw74UWOJEueSzQ= 2023-05-24T21:28:20.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:IszhZ4DmGPGlbIw74UWOJEueSzQ=', 'Content-Length': '0'} 2023-05-24T21:28:20.971 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000076d60d49e2ae7787-00646e8174-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.972 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:28:20.973 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:28:20.973 DEBUG:boto:path=/qskrps-48/?prefix=obj4 2023-05-24T21:28:20.974 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj4 2023-05-24T21:28:20.974 DEBUG:boto:Method: GET 2023-05-24T21:28:20.974 DEBUG:boto:Path: /qskrps-48/?prefix=obj4 2023-05-24T21:28:20.975 DEBUG:boto:Data: 2023-05-24T21:28:20.975 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.976 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:20.976 DEBUG:boto:Port: 8001 2023-05-24T21:28:20.976 DEBUG:boto:Params: {} 2023-05-24T21:28:20.977 DEBUG:boto:Token: None 2023-05-24T21:28:20.977 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:20 GMT /qskrps-48/ 2023-05-24T21:28:20.978 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:NwSOcAGxqDVYN1FOu6xegODzDj8= 2023-05-24T21:28:20.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:NwSOcAGxqDVYN1FOu6xegODzDj8=', 'Content-Length': '0'} 2023-05-24T21:28:20.980 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005aba6baf9f02a85-00646e8174-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.981 DEBUG:boto:b'qskrps-48obj41000falseobj42023-05-24T21:27:06.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:20.981 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:28:20.982 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:28:20.982 DEBUG:boto:path=/qskrps-48/?prefix=obj4 2023-05-24T21:28:20.983 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj4 2023-05-24T21:28:20.983 DEBUG:boto:Method: GET 2023-05-24T21:28:20.983 DEBUG:boto:Path: /qskrps-48/?prefix=obj4 2023-05-24T21:28:20.984 DEBUG:boto:Data: 2023-05-24T21:28:20.984 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.985 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:20.985 DEBUG:boto:Port: 8000 2023-05-24T21:28:20.985 DEBUG:boto:Params: {} 2023-05-24T21:28:20.986 DEBUG:boto:Token: None 2023-05-24T21:28:20.986 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:20 GMT /qskrps-48/ 2023-05-24T21:28:20.986 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:NwSOcAGxqDVYN1FOu6xegODzDj8= 2023-05-24T21:28:20.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:NwSOcAGxqDVYN1FOu6xegODzDj8=', 'Content-Length': '0'} 2023-05-24T21:28:20.989 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001739bfc6c3d49dff-00646e8174-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.989 DEBUG:boto:b'qskrps-48obj41000falseobj42023-05-24T21:27:06.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:20.990 DEBUG:boto:path=/qskrps-48/obj4 2023-05-24T21:28:20.990 DEBUG:boto:auth_path=/qskrps-48/obj4 2023-05-24T21:28:20.991 DEBUG:boto:Method: HEAD 2023-05-24T21:28:20.991 DEBUG:boto:Path: /qskrps-48/obj4 2023-05-24T21:28:20.991 DEBUG:boto:Data: 2023-05-24T21:28:20.992 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.992 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:20.992 DEBUG:boto:Port: 8001 2023-05-24T21:28:20.993 DEBUG:boto:Params: {} 2023-05-24T21:28:20.993 DEBUG:boto:Token: None 2023-05-24T21:28:20.994 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:20 GMT /qskrps-48/obj4 2023-05-24T21:28:20.994 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:XU2aFuDk15XqcS5YkU5gm9g7yGo= 2023-05-24T21:28:20.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:XU2aFuDk15XqcS5YkU5gm9g7yGo=', 'Content-Length': '0'} 2023-05-24T21:28:20.996 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e56e0ff723ffbfde-00646e8174-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:20.997 DEBUG:boto:path=/qskrps-48/obj4 2023-05-24T21:28:20.997 DEBUG:boto:auth_path=/qskrps-48/obj4 2023-05-24T21:28:20.997 DEBUG:boto:Method: HEAD 2023-05-24T21:28:20.998 DEBUG:boto:Path: /qskrps-48/obj4 2023-05-24T21:28:20.998 DEBUG:boto:Data: 2023-05-24T21:28:20.998 DEBUG:boto:Headers: {} 2023-05-24T21:28:20.999 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:20.999 DEBUG:boto:Port: 8000 2023-05-24T21:28:20.999 DEBUG:boto:Params: {} 2023-05-24T21:28:21.000 DEBUG:boto:Token: None 2023-05-24T21:28:21.000 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:21 GMT /qskrps-48/obj4 2023-05-24T21:28:21.001 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:GlsUfiJTiZITKTdrSWcm2eAJ6Jo= 2023-05-24T21:28:21.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:GlsUfiJTiZITKTdrSWcm2eAJ6Jo=', 'Content-Length': '0'} 2023-05-24T21:28:21.002 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008238c60fb1c58018-00646e8174-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-24T21:28:21.003 DEBUG:boto:path=/qskrps-48/obj4 2023-05-24T21:28:21.004 DEBUG:boto:auth_path=/qskrps-48/obj4 2023-05-24T21:28:21.004 DEBUG:boto:Method: GET 2023-05-24T21:28:21.005 DEBUG:boto:Path: /qskrps-48/obj4 2023-05-24T21:28:21.005 DEBUG:boto:Data: 2023-05-24T21:28:21.006 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.006 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:21.007 DEBUG:boto:Port: 8001 2023-05-24T21:28:21.007 DEBUG:boto:Params: {} 2023-05-24T21:28:21.007 DEBUG:boto:Token: None 2023-05-24T21:28:21.008 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:21 GMT /qskrps-48/obj4 2023-05-24T21:28:21.008 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:QwiLOQrtr/2fgi2sWcAnQc2vpks= 2023-05-24T21:28:21.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:QwiLOQrtr/2fgi2sWcAnQc2vpks=', 'Content-Length': '0'} 2023-05-24T21:28:21.010 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f9e5e052de412375-00646e8175-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.052 DEBUG:boto:path=/qskrps-48/obj4 2023-05-24T21:28:21.053 DEBUG:boto:auth_path=/qskrps-48/obj4 2023-05-24T21:28:21.054 DEBUG:boto:Method: GET 2023-05-24T21:28:21.054 DEBUG:boto:Path: /qskrps-48/obj4 2023-05-24T21:28:21.054 DEBUG:boto:Data: 2023-05-24T21:28:21.055 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.055 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:21.055 DEBUG:boto:Port: 8000 2023-05-24T21:28:21.056 DEBUG:boto:Params: {} 2023-05-24T21:28:21.056 DEBUG:boto:Token: None 2023-05-24T21:28:21.057 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:21 GMT /qskrps-48/obj4 2023-05-24T21:28:21.057 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:QwiLOQrtr/2fgi2sWcAnQc2vpks= 2023-05-24T21:28:21.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:QwiLOQrtr/2fgi2sWcAnQc2vpks=', 'Content-Length': '0'} 2023-05-24T21:28:21.060 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000abaf5b01dc4511df-00646e8175-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.060 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:28:21.061 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:28:21.061 DEBUG:boto:path=/qskrps-48/?prefix=obj4 2023-05-24T21:28:21.062 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj4 2023-05-24T21:28:21.062 DEBUG:boto:Method: GET 2023-05-24T21:28:21.062 DEBUG:boto:Path: /qskrps-48/?prefix=obj4 2023-05-24T21:28:21.063 DEBUG:boto:Data: 2023-05-24T21:28:21.063 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.063 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:21.064 DEBUG:boto:Port: 8001 2023-05-24T21:28:21.064 DEBUG:boto:Params: {} 2023-05-24T21:28:21.065 DEBUG:boto:Token: None 2023-05-24T21:28:21.065 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:21 GMT /qskrps-48/ 2023-05-24T21:28:21.066 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:6IoPBa2Mi8L1eD/Ct15giFLjQ8o= 2023-05-24T21:28:21.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:6IoPBa2Mi8L1eD/Ct15giFLjQ8o=', 'Content-Length': '0'} 2023-05-24T21:28:21.068 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eaf097e9874c291e-00646e8175-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.069 DEBUG:boto:b'qskrps-48obj41000falseobj42023-05-24T21:27:06.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:21.070 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:28:21.070 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:28:21.070 DEBUG:boto:path=/qskrps-48/?prefix=obj4 2023-05-24T21:28:21.071 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj4 2023-05-24T21:28:21.071 DEBUG:boto:Method: GET 2023-05-24T21:28:21.071 DEBUG:boto:Path: /qskrps-48/?prefix=obj4 2023-05-24T21:28:21.072 DEBUG:boto:Data: 2023-05-24T21:28:21.072 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.072 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:21.073 DEBUG:boto:Port: 8000 2023-05-24T21:28:21.073 DEBUG:boto:Params: {} 2023-05-24T21:28:21.074 DEBUG:boto:Token: None 2023-05-24T21:28:21.074 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:21 GMT /qskrps-48/ 2023-05-24T21:28:21.074 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:6IoPBa2Mi8L1eD/Ct15giFLjQ8o= 2023-05-24T21:28:21.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:6IoPBa2Mi8L1eD/Ct15giFLjQ8o=', 'Content-Length': '0'} 2023-05-24T21:28:21.077 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056ebba9b40a2e334-00646e8175-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.078 DEBUG:boto:b'qskrps-48obj41000falseobj42023-05-24T21:27:06.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:21.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-05-24T21:28:21.079 DEBUG:boto:path=/qskrps-48/obj5 2023-05-24T21:28:21.079 DEBUG:boto:auth_path=/qskrps-48/obj5 2023-05-24T21:28:21.079 DEBUG:boto:path=/qskrps-48/obj5?versionId=null 2023-05-24T21:28:21.080 DEBUG:boto:auth_path=/qskrps-48/obj5?versionId=null 2023-05-24T21:28:21.080 DEBUG:boto:Method: GET 2023-05-24T21:28:21.081 DEBUG:boto:Path: /qskrps-48/obj5?versionId=null 2023-05-24T21:28:21.081 DEBUG:boto:Data: 2023-05-24T21:28:21.081 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.082 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:21.082 DEBUG:boto:Port: 8001 2023-05-24T21:28:21.082 DEBUG:boto:Params: {} 2023-05-24T21:28:21.083 DEBUG:boto:Token: None 2023-05-24T21:28:21.083 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:21 GMT /qskrps-48/obj5?versionId=null 2023-05-24T21:28:21.084 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:uB3AY7uDARRuymxbVn97hXLbYyY= 2023-05-24T21:28:21.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:uB3AY7uDARRuymxbVn97hXLbYyY=', 'Content-Length': '0'} 2023-05-24T21:28:21.086 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a8bc989b489e2812-00646e8175-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.086 DEBUG:boto:path=/qskrps-48/obj5 2023-05-24T21:28:21.087 DEBUG:boto:auth_path=/qskrps-48/obj5 2023-05-24T21:28:21.088 DEBUG:boto:path=/qskrps-48/obj5?versionId=null 2023-05-24T21:28:21.088 DEBUG:boto:auth_path=/qskrps-48/obj5?versionId=null 2023-05-24T21:28:21.088 DEBUG:boto:Method: GET 2023-05-24T21:28:21.089 DEBUG:boto:Path: /qskrps-48/obj5?versionId=null 2023-05-24T21:28:21.089 DEBUG:boto:Data: 2023-05-24T21:28:21.089 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.090 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:21.090 DEBUG:boto:Port: 8000 2023-05-24T21:28:21.091 DEBUG:boto:Params: {} 2023-05-24T21:28:21.091 DEBUG:boto:Token: None 2023-05-24T21:28:21.091 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:21 GMT /qskrps-48/obj5?versionId=null 2023-05-24T21:28:21.092 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:uB3AY7uDARRuymxbVn97hXLbYyY= 2023-05-24T21:28:21.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:uB3AY7uDARRuymxbVn97hXLbYyY=', 'Content-Length': '0'} 2023-05-24T21:28:21.094 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000007be1ddd863d441e-00646e8175-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.136 DEBUG:boto:path=/qskrps-48/obj5 2023-05-24T21:28:21.137 DEBUG:boto:auth_path=/qskrps-48/obj5 2023-05-24T21:28:21.137 DEBUG:boto:path=/qskrps-48/obj5?versionId=null 2023-05-24T21:28:21.138 DEBUG:boto:auth_path=/qskrps-48/obj5?versionId=null 2023-05-24T21:28:21.138 DEBUG:boto:Method: HEAD 2023-05-24T21:28:21.138 DEBUG:boto:Path: /qskrps-48/obj5?versionId=null 2023-05-24T21:28:21.139 DEBUG:boto:Data: 2023-05-24T21:28:21.139 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.140 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:21.140 DEBUG:boto:Port: 8001 2023-05-24T21:28:21.140 DEBUG:boto:Params: {} 2023-05-24T21:28:21.141 DEBUG:boto:Token: None 2023-05-24T21:28:21.141 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:21 GMT /qskrps-48/obj5?versionId=null 2023-05-24T21:28:21.142 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:qdRjrFYacMtxRthnKxwyXsOz+Ug= 2023-05-24T21:28:21.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qdRjrFYacMtxRthnKxwyXsOz+Ug=', 'Content-Length': '0'} 2023-05-24T21:28:21.144 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009bc3fa2ecfaec9e7-00646e8175-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.145 DEBUG:boto:path=/qskrps-48/obj5 2023-05-24T21:28:21.145 DEBUG:boto:auth_path=/qskrps-48/obj5 2023-05-24T21:28:21.145 DEBUG:boto:path=/qskrps-48/obj5?versionId=null 2023-05-24T21:28:21.146 DEBUG:boto:auth_path=/qskrps-48/obj5?versionId=null 2023-05-24T21:28:21.146 DEBUG:boto:Method: HEAD 2023-05-24T21:28:21.147 DEBUG:boto:Path: /qskrps-48/obj5?versionId=null 2023-05-24T21:28:21.147 DEBUG:boto:Data: 2023-05-24T21:28:21.148 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.148 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:21.149 DEBUG:boto:Port: 8000 2023-05-24T21:28:21.149 DEBUG:boto:Params: {} 2023-05-24T21:28:21.149 DEBUG:boto:Token: None 2023-05-24T21:28:21.150 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:21 GMT /qskrps-48/obj5?versionId=null 2023-05-24T21:28:21.150 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:qdRjrFYacMtxRthnKxwyXsOz+Ug= 2023-05-24T21:28:21.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qdRjrFYacMtxRthnKxwyXsOz+Ug=', 'Content-Length': '0'} 2023-05-24T21:28:21.153 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000024f440a32baec151-00646e8175-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-05-24T21:28:21.154 DEBUG:boto:path=/qskrps-48/obj5 2023-05-24T21:28:21.154 DEBUG:boto:auth_path=/qskrps-48/obj5 2023-05-24T21:28:21.155 DEBUG:boto:path=/qskrps-48/obj5?versionId=null 2023-05-24T21:28:21.155 DEBUG:boto:auth_path=/qskrps-48/obj5?versionId=null 2023-05-24T21:28:21.155 DEBUG:boto:Method: GET 2023-05-24T21:28:21.156 DEBUG:boto:Path: /qskrps-48/obj5?versionId=null 2023-05-24T21:28:21.156 DEBUG:boto:Data: 2023-05-24T21:28:21.157 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.157 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:21.157 DEBUG:boto:Port: 8001 2023-05-24T21:28:21.158 DEBUG:boto:Params: {} 2023-05-24T21:28:21.158 DEBUG:boto:Token: None 2023-05-24T21:28:21.159 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:21 GMT /qskrps-48/obj5?versionId=null 2023-05-24T21:28:21.159 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:uB3AY7uDARRuymxbVn97hXLbYyY= 2023-05-24T21:28:21.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:uB3AY7uDARRuymxbVn97hXLbYyY=', 'Content-Length': '0'} 2023-05-24T21:28:21.161 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cb58b4b1ac1e70e0-00646e8175-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.162 DEBUG:boto:path=/qskrps-48/obj5 2023-05-24T21:28:21.162 DEBUG:boto:auth_path=/qskrps-48/obj5 2023-05-24T21:28:21.162 DEBUG:boto:path=/qskrps-48/obj5?versionId=null 2023-05-24T21:28:21.163 DEBUG:boto:auth_path=/qskrps-48/obj5?versionId=null 2023-05-24T21:28:21.163 DEBUG:boto:Method: GET 2023-05-24T21:28:21.164 DEBUG:boto:Path: /qskrps-48/obj5?versionId=null 2023-05-24T21:28:21.164 DEBUG:boto:Data: 2023-05-24T21:28:21.164 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.165 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:21.165 DEBUG:boto:Port: 8000 2023-05-24T21:28:21.166 DEBUG:boto:Params: {} 2023-05-24T21:28:21.166 DEBUG:boto:Token: None 2023-05-24T21:28:21.166 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:21 GMT /qskrps-48/obj5?versionId=null 2023-05-24T21:28:21.167 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:uB3AY7uDARRuymxbVn97hXLbYyY= 2023-05-24T21:28:21.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:uB3AY7uDARRuymxbVn97hXLbYyY=', 'Content-Length': '0'} 2023-05-24T21:28:21.169 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004b9fd511263c9e14-00646e8175-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.170 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:28:21.170 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:28:21.171 DEBUG:boto:path=/qskrps-48/?prefix=obj5 2023-05-24T21:28:21.171 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj5 2023-05-24T21:28:21.171 DEBUG:boto:Method: GET 2023-05-24T21:28:21.172 DEBUG:boto:Path: /qskrps-48/?prefix=obj5 2023-05-24T21:28:21.172 DEBUG:boto:Data: 2023-05-24T21:28:21.173 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.173 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:21.173 DEBUG:boto:Port: 8001 2023-05-24T21:28:21.174 DEBUG:boto:Params: {} 2023-05-24T21:28:21.174 DEBUG:boto:Token: None 2023-05-24T21:28:21.175 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:21 GMT /qskrps-48/ 2023-05-24T21:28:21.175 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:6IoPBa2Mi8L1eD/Ct15giFLjQ8o= 2023-05-24T21:28:21.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:6IoPBa2Mi8L1eD/Ct15giFLjQ8o=', 'Content-Length': '0'} 2023-05-24T21:28:21.178 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057519e418194a398-00646e8175-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.220 DEBUG:boto:b'qskrps-48obj51000falseobj52023-05-24T21:28:09.543Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:21.221 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:28:21.222 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:28:21.222 DEBUG:boto:path=/qskrps-48/?prefix=obj5 2023-05-24T21:28:21.222 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj5 2023-05-24T21:28:21.223 DEBUG:boto:Method: GET 2023-05-24T21:28:21.223 DEBUG:boto:Path: /qskrps-48/?prefix=obj5 2023-05-24T21:28:21.223 DEBUG:boto:Data: 2023-05-24T21:28:21.224 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.224 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:21.224 DEBUG:boto:Port: 8000 2023-05-24T21:28:21.225 DEBUG:boto:Params: {} 2023-05-24T21:28:21.225 DEBUG:boto:Token: None 2023-05-24T21:28:21.226 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:21 GMT /qskrps-48/ 2023-05-24T21:28:21.226 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:6IoPBa2Mi8L1eD/Ct15giFLjQ8o= 2023-05-24T21:28:21.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:6IoPBa2Mi8L1eD/Ct15giFLjQ8o=', 'Content-Length': '0'} 2023-05-24T21:28:21.229 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051448e5312a6f6f2-00646e8175-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.230 DEBUG:boto:b'qskrps-48obj51000falseobj52023-05-24T21:28:09.543Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:21.230 DEBUG:boto:path=/qskrps-48/obj5 2023-05-24T21:28:21.231 DEBUG:boto:auth_path=/qskrps-48/obj5 2023-05-24T21:28:21.231 DEBUG:boto:Method: HEAD 2023-05-24T21:28:21.232 DEBUG:boto:Path: /qskrps-48/obj5 2023-05-24T21:28:21.232 DEBUG:boto:Data: 2023-05-24T21:28:21.232 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.233 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:21.233 DEBUG:boto:Port: 8001 2023-05-24T21:28:21.234 DEBUG:boto:Params: {} 2023-05-24T21:28:21.234 DEBUG:boto:Token: None 2023-05-24T21:28:21.235 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:21 GMT /qskrps-48/obj5 2023-05-24T21:28:21.235 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:LoYnfseaihZcmEqylEaArPHa/88= 2023-05-24T21:28:21.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:LoYnfseaihZcmEqylEaArPHa/88=', 'Content-Length': '0'} 2023-05-24T21:28:21.238 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000598a8b2ec746ab07-00646e8175-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.238 DEBUG:boto:path=/qskrps-48/obj5 2023-05-24T21:28:21.239 DEBUG:boto:auth_path=/qskrps-48/obj5 2023-05-24T21:28:21.239 DEBUG:boto:Method: HEAD 2023-05-24T21:28:21.239 DEBUG:boto:Path: /qskrps-48/obj5 2023-05-24T21:28:21.240 DEBUG:boto:Data: 2023-05-24T21:28:21.240 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.240 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:21.241 DEBUG:boto:Port: 8000 2023-05-24T21:28:21.241 DEBUG:boto:Params: {} 2023-05-24T21:28:21.241 DEBUG:boto:Token: None 2023-05-24T21:28:21.242 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:21 GMT /qskrps-48/obj5 2023-05-24T21:28:21.242 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:LoYnfseaihZcmEqylEaArPHa/88= 2023-05-24T21:28:21.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:LoYnfseaihZcmEqylEaArPHa/88=', 'Content-Length': '0'} 2023-05-24T21:28:21.244 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000006db1eec1c2bc87-00646e8175-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-05-24T21:28:21.245 DEBUG:boto:path=/qskrps-48/obj5 2023-05-24T21:28:21.245 DEBUG:boto:auth_path=/qskrps-48/obj5 2023-05-24T21:28:21.246 DEBUG:boto:Method: GET 2023-05-24T21:28:21.246 DEBUG:boto:Path: /qskrps-48/obj5 2023-05-24T21:28:21.247 DEBUG:boto:Data: 2023-05-24T21:28:21.247 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.247 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:21.248 DEBUG:boto:Port: 8001 2023-05-24T21:28:21.248 DEBUG:boto:Params: {} 2023-05-24T21:28:21.248 DEBUG:boto:Token: None 2023-05-24T21:28:21.249 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:21 GMT /qskrps-48/obj5 2023-05-24T21:28:21.249 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:gc8Ia41iI8Dt1Ij40xJvw4awZAI= 2023-05-24T21:28:21.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:gc8Ia41iI8Dt1Ij40xJvw4awZAI=', 'Content-Length': '0'} 2023-05-24T21:28:21.251 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000070f8a4f5644b0b96-00646e8175-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.252 DEBUG:boto:path=/qskrps-48/obj5 2023-05-24T21:28:21.252 DEBUG:boto:auth_path=/qskrps-48/obj5 2023-05-24T21:28:21.253 DEBUG:boto:Method: GET 2023-05-24T21:28:21.253 DEBUG:boto:Path: /qskrps-48/obj5 2023-05-24T21:28:21.253 DEBUG:boto:Data: 2023-05-24T21:28:21.254 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.254 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:21.255 DEBUG:boto:Port: 8000 2023-05-24T21:28:21.255 DEBUG:boto:Params: {} 2023-05-24T21:28:21.255 DEBUG:boto:Token: None 2023-05-24T21:28:21.256 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:21 GMT /qskrps-48/obj5 2023-05-24T21:28:21.257 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:gc8Ia41iI8Dt1Ij40xJvw4awZAI= 2023-05-24T21:28:21.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:gc8Ia41iI8Dt1Ij40xJvw4awZAI=', 'Content-Length': '0'} 2023-05-24T21:28:21.259 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bd2d68eeebd57152-00646e8175-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.300 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:28:21.301 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:28:21.301 DEBUG:boto:path=/qskrps-48/?prefix=obj5 2023-05-24T21:28:21.302 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj5 2023-05-24T21:28:21.302 DEBUG:boto:Method: GET 2023-05-24T21:28:21.303 DEBUG:boto:Path: /qskrps-48/?prefix=obj5 2023-05-24T21:28:21.303 DEBUG:boto:Data: 2023-05-24T21:28:21.304 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.304 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:21.304 DEBUG:boto:Port: 8001 2023-05-24T21:28:21.305 DEBUG:boto:Params: {} 2023-05-24T21:28:21.305 DEBUG:boto:Token: None 2023-05-24T21:28:21.306 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:21 GMT /qskrps-48/ 2023-05-24T21:28:21.306 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:6IoPBa2Mi8L1eD/Ct15giFLjQ8o= 2023-05-24T21:28:21.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:6IoPBa2Mi8L1eD/Ct15giFLjQ8o=', 'Content-Length': '0'} 2023-05-24T21:28:21.309 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a204ab1ab8eab889-00646e8175-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.309 DEBUG:boto:b'qskrps-48obj51000falseobj52023-05-24T21:28:09.543Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:21.310 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:28:21.310 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:28:21.311 DEBUG:boto:path=/qskrps-48/?prefix=obj5 2023-05-24T21:28:21.311 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj5 2023-05-24T21:28:21.312 DEBUG:boto:Method: GET 2023-05-24T21:28:21.312 DEBUG:boto:Path: /qskrps-48/?prefix=obj5 2023-05-24T21:28:21.312 DEBUG:boto:Data: 2023-05-24T21:28:21.313 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.313 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:21.314 DEBUG:boto:Port: 8000 2023-05-24T21:28:21.314 DEBUG:boto:Params: {} 2023-05-24T21:28:21.314 DEBUG:boto:Token: None 2023-05-24T21:28:21.315 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:21 GMT /qskrps-48/ 2023-05-24T21:28:21.315 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:6IoPBa2Mi8L1eD/Ct15giFLjQ8o= 2023-05-24T21:28:21.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:6IoPBa2Mi8L1eD/Ct15giFLjQ8o=', 'Content-Length': '0'} 2023-05-24T21:28:21.318 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003dfe0ec36cff11e5-00646e8175-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.319 DEBUG:boto:b'qskrps-48obj51000falseobj52023-05-24T21:28:09.543Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:21.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-05-24T21:28:21.320 DEBUG:boto:path=/qskrps-48/obj6 2023-05-24T21:28:21.321 DEBUG:boto:auth_path=/qskrps-48/obj6 2023-05-24T21:28:21.321 DEBUG:boto:path=/qskrps-48/obj6?versionId=null 2023-05-24T21:28:21.321 DEBUG:boto:auth_path=/qskrps-48/obj6?versionId=null 2023-05-24T21:28:21.322 DEBUG:boto:Method: GET 2023-05-24T21:28:21.322 DEBUG:boto:Path: /qskrps-48/obj6?versionId=null 2023-05-24T21:28:21.322 DEBUG:boto:Data: 2023-05-24T21:28:21.323 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.323 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:21.324 DEBUG:boto:Port: 8001 2023-05-24T21:28:21.324 DEBUG:boto:Params: {} 2023-05-24T21:28:21.324 DEBUG:boto:Token: None 2023-05-24T21:28:21.325 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:21 GMT /qskrps-48/obj6?versionId=null 2023-05-24T21:28:21.325 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ZNiCzzqokHL90P59r95GMXuaBeo= 2023-05-24T21:28:21.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ZNiCzzqokHL90P59r95GMXuaBeo=', 'Content-Length': '0'} 2023-05-24T21:28:21.328 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000088a1158f6c9b01ab-00646e8175-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.328 DEBUG:boto:path=/qskrps-48/obj6 2023-05-24T21:28:21.329 DEBUG:boto:auth_path=/qskrps-48/obj6 2023-05-24T21:28:21.329 DEBUG:boto:path=/qskrps-48/obj6?versionId=null 2023-05-24T21:28:21.330 DEBUG:boto:auth_path=/qskrps-48/obj6?versionId=null 2023-05-24T21:28:21.330 DEBUG:boto:Method: GET 2023-05-24T21:28:21.330 DEBUG:boto:Path: /qskrps-48/obj6?versionId=null 2023-05-24T21:28:21.331 DEBUG:boto:Data: 2023-05-24T21:28:21.331 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.331 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:21.332 DEBUG:boto:Port: 8000 2023-05-24T21:28:21.332 DEBUG:boto:Params: {} 2023-05-24T21:28:21.332 DEBUG:boto:Token: None 2023-05-24T21:28:21.333 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:21 GMT /qskrps-48/obj6?versionId=null 2023-05-24T21:28:21.333 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ZNiCzzqokHL90P59r95GMXuaBeo= 2023-05-24T21:28:21.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ZNiCzzqokHL90P59r95GMXuaBeo=', 'Content-Length': '0'} 2023-05-24T21:28:21.336 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d232158c63a4964-00646e8175-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.336 DEBUG:boto:path=/qskrps-48/obj6 2023-05-24T21:28:21.337 DEBUG:boto:auth_path=/qskrps-48/obj6 2023-05-24T21:28:21.337 DEBUG:boto:path=/qskrps-48/obj6?versionId=null 2023-05-24T21:28:21.337 DEBUG:boto:auth_path=/qskrps-48/obj6?versionId=null 2023-05-24T21:28:21.338 DEBUG:boto:Method: HEAD 2023-05-24T21:28:21.338 DEBUG:boto:Path: /qskrps-48/obj6?versionId=null 2023-05-24T21:28:21.338 DEBUG:boto:Data: 2023-05-24T21:28:21.339 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.339 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:21.339 DEBUG:boto:Port: 8001 2023-05-24T21:28:21.340 DEBUG:boto:Params: {} 2023-05-24T21:28:21.340 DEBUG:boto:Token: None 2023-05-24T21:28:21.341 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:21 GMT /qskrps-48/obj6?versionId=null 2023-05-24T21:28:21.341 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:9NML3nAVT90enwck/NvNC5uhTXQ= 2023-05-24T21:28:21.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:9NML3nAVT90enwck/NvNC5uhTXQ=', 'Content-Length': '0'} 2023-05-24T21:28:21.343 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eb291274fe54f82b-00646e8175-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.344 DEBUG:boto:path=/qskrps-48/obj6 2023-05-24T21:28:21.344 DEBUG:boto:auth_path=/qskrps-48/obj6 2023-05-24T21:28:21.344 DEBUG:boto:path=/qskrps-48/obj6?versionId=null 2023-05-24T21:28:21.345 DEBUG:boto:auth_path=/qskrps-48/obj6?versionId=null 2023-05-24T21:28:21.345 DEBUG:boto:Method: HEAD 2023-05-24T21:28:21.346 DEBUG:boto:Path: /qskrps-48/obj6?versionId=null 2023-05-24T21:28:21.346 DEBUG:boto:Data: 2023-05-24T21:28:21.346 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.347 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:21.347 DEBUG:boto:Port: 8000 2023-05-24T21:28:21.348 DEBUG:boto:Params: {} 2023-05-24T21:28:21.348 DEBUG:boto:Token: None 2023-05-24T21:28:21.349 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:21 GMT /qskrps-48/obj6?versionId=null 2023-05-24T21:28:21.349 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:9NML3nAVT90enwck/NvNC5uhTXQ= 2023-05-24T21:28:21.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:9NML3nAVT90enwck/NvNC5uhTXQ=', 'Content-Length': '0'} 2023-05-24T21:28:21.351 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bf5ab6f43bdad1a5-00646e8175-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-05-24T21:28:21.353 DEBUG:boto:path=/qskrps-48/obj6 2023-05-24T21:28:21.353 DEBUG:boto:auth_path=/qskrps-48/obj6 2023-05-24T21:28:21.354 DEBUG:boto:path=/qskrps-48/obj6?versionId=null 2023-05-24T21:28:21.354 DEBUG:boto:auth_path=/qskrps-48/obj6?versionId=null 2023-05-24T21:28:21.354 DEBUG:boto:Method: GET 2023-05-24T21:28:21.355 DEBUG:boto:Path: /qskrps-48/obj6?versionId=null 2023-05-24T21:28:21.355 DEBUG:boto:Data: 2023-05-24T21:28:21.356 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.356 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:21.356 DEBUG:boto:Port: 8001 2023-05-24T21:28:21.357 DEBUG:boto:Params: {} 2023-05-24T21:28:21.357 DEBUG:boto:Token: None 2023-05-24T21:28:21.358 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:21 GMT /qskrps-48/obj6?versionId=null 2023-05-24T21:28:21.358 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ZNiCzzqokHL90P59r95GMXuaBeo= 2023-05-24T21:28:21.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ZNiCzzqokHL90P59r95GMXuaBeo=', 'Content-Length': '0'} 2023-05-24T21:28:21.360 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000029789cfbb532c108-00646e8175-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.361 DEBUG:boto:path=/qskrps-48/obj6 2023-05-24T21:28:21.361 DEBUG:boto:auth_path=/qskrps-48/obj6 2023-05-24T21:28:21.361 DEBUG:boto:path=/qskrps-48/obj6?versionId=null 2023-05-24T21:28:21.362 DEBUG:boto:auth_path=/qskrps-48/obj6?versionId=null 2023-05-24T21:28:21.362 DEBUG:boto:Method: GET 2023-05-24T21:28:21.362 DEBUG:boto:Path: /qskrps-48/obj6?versionId=null 2023-05-24T21:28:21.363 DEBUG:boto:Data: 2023-05-24T21:28:21.363 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.363 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:21.364 DEBUG:boto:Port: 8000 2023-05-24T21:28:21.364 DEBUG:boto:Params: {} 2023-05-24T21:28:21.365 DEBUG:boto:Token: None 2023-05-24T21:28:21.365 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:21 GMT /qskrps-48/obj6?versionId=null 2023-05-24T21:28:21.365 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ZNiCzzqokHL90P59r95GMXuaBeo= 2023-05-24T21:28:21.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ZNiCzzqokHL90P59r95GMXuaBeo=', 'Content-Length': '0'} 2023-05-24T21:28:21.368 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c78940aae9c78f97-00646e8175-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.368 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:28:21.369 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:28:21.369 DEBUG:boto:path=/qskrps-48/?prefix=obj6 2023-05-24T21:28:21.369 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj6 2023-05-24T21:28:21.370 DEBUG:boto:Method: GET 2023-05-24T21:28:21.370 DEBUG:boto:Path: /qskrps-48/?prefix=obj6 2023-05-24T21:28:21.371 DEBUG:boto:Data: 2023-05-24T21:28:21.371 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.371 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:21.372 DEBUG:boto:Port: 8001 2023-05-24T21:28:21.372 DEBUG:boto:Params: {} 2023-05-24T21:28:21.373 DEBUG:boto:Token: None 2023-05-24T21:28:21.373 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:21 GMT /qskrps-48/ 2023-05-24T21:28:21.374 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:6IoPBa2Mi8L1eD/Ct15giFLjQ8o= 2023-05-24T21:28:21.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:6IoPBa2Mi8L1eD/Ct15giFLjQ8o=', 'Content-Length': '0'} 2023-05-24T21:28:21.376 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eaf925866a3c6f89-00646e8175-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.377 DEBUG:boto:b'qskrps-48obj61000falseobj62023-05-24T21:28:09.598Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:21.378 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:28:21.378 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:28:21.379 DEBUG:boto:path=/qskrps-48/?prefix=obj6 2023-05-24T21:28:21.379 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj6 2023-05-24T21:28:21.379 DEBUG:boto:Method: GET 2023-05-24T21:28:21.380 DEBUG:boto:Path: /qskrps-48/?prefix=obj6 2023-05-24T21:28:21.380 DEBUG:boto:Data: 2023-05-24T21:28:21.381 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.381 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:21.381 DEBUG:boto:Port: 8000 2023-05-24T21:28:21.382 DEBUG:boto:Params: {} 2023-05-24T21:28:21.382 DEBUG:boto:Token: None 2023-05-24T21:28:21.382 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:21 GMT /qskrps-48/ 2023-05-24T21:28:21.383 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:6IoPBa2Mi8L1eD/Ct15giFLjQ8o= 2023-05-24T21:28:21.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:6IoPBa2Mi8L1eD/Ct15giFLjQ8o=', 'Content-Length': '0'} 2023-05-24T21:28:21.385 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d60722c184f8f36-00646e8175-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.386 DEBUG:boto:b'qskrps-48obj61000falseobj62023-05-24T21:28:09.598Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:21.387 DEBUG:boto:path=/qskrps-48/obj6 2023-05-24T21:28:21.387 DEBUG:boto:auth_path=/qskrps-48/obj6 2023-05-24T21:28:21.388 DEBUG:boto:Method: HEAD 2023-05-24T21:28:21.388 DEBUG:boto:Path: /qskrps-48/obj6 2023-05-24T21:28:21.388 DEBUG:boto:Data: 2023-05-24T21:28:21.389 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.389 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:21.389 DEBUG:boto:Port: 8001 2023-05-24T21:28:21.390 DEBUG:boto:Params: {} 2023-05-24T21:28:21.390 DEBUG:boto:Token: None 2023-05-24T21:28:21.390 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:21 GMT /qskrps-48/obj6 2023-05-24T21:28:21.391 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:C+fya4TDMNuCUgC+KRCr6dR5XbU= 2023-05-24T21:28:21.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:C+fya4TDMNuCUgC+KRCr6dR5XbU=', 'Content-Length': '0'} 2023-05-24T21:28:21.393 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b60976acf6799ceb-00646e8175-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.393 DEBUG:boto:path=/qskrps-48/obj6 2023-05-24T21:28:21.394 DEBUG:boto:auth_path=/qskrps-48/obj6 2023-05-24T21:28:21.394 DEBUG:boto:Method: HEAD 2023-05-24T21:28:21.394 DEBUG:boto:Path: /qskrps-48/obj6 2023-05-24T21:28:21.395 DEBUG:boto:Data: 2023-05-24T21:28:21.395 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.395 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:21.396 DEBUG:boto:Port: 8000 2023-05-24T21:28:21.396 DEBUG:boto:Params: {} 2023-05-24T21:28:21.396 DEBUG:boto:Token: None 2023-05-24T21:28:21.397 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:21 GMT /qskrps-48/obj6 2023-05-24T21:28:21.397 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:C+fya4TDMNuCUgC+KRCr6dR5XbU= 2023-05-24T21:28:21.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:C+fya4TDMNuCUgC+KRCr6dR5XbU=', 'Content-Length': '0'} 2023-05-24T21:28:21.400 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001ec35b9c5c59f1f1-00646e8175-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-05-24T21:28:21.401 DEBUG:boto:path=/qskrps-48/obj6 2023-05-24T21:28:21.401 DEBUG:boto:auth_path=/qskrps-48/obj6 2023-05-24T21:28:21.401 DEBUG:boto:Method: GET 2023-05-24T21:28:21.402 DEBUG:boto:Path: /qskrps-48/obj6 2023-05-24T21:28:21.402 DEBUG:boto:Data: 2023-05-24T21:28:21.403 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.403 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:21.404 DEBUG:boto:Port: 8001 2023-05-24T21:28:21.404 DEBUG:boto:Params: {} 2023-05-24T21:28:21.405 DEBUG:boto:Token: None 2023-05-24T21:28:21.405 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:21 GMT /qskrps-48/obj6 2023-05-24T21:28:21.405 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:FTbadZJvgMa2R5DsLENC0YCexqk= 2023-05-24T21:28:21.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:FTbadZJvgMa2R5DsLENC0YCexqk=', 'Content-Length': '0'} 2023-05-24T21:28:21.408 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dab875029d35ccb5-00646e8175-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.448 DEBUG:boto:path=/qskrps-48/obj6 2023-05-24T21:28:21.449 DEBUG:boto:auth_path=/qskrps-48/obj6 2023-05-24T21:28:21.449 DEBUG:boto:Method: GET 2023-05-24T21:28:21.450 DEBUG:boto:Path: /qskrps-48/obj6 2023-05-24T21:28:21.450 DEBUG:boto:Data: 2023-05-24T21:28:21.450 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.451 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:21.451 DEBUG:boto:Port: 8000 2023-05-24T21:28:21.451 DEBUG:boto:Params: {} 2023-05-24T21:28:21.452 DEBUG:boto:Token: None 2023-05-24T21:28:21.452 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:21 GMT /qskrps-48/obj6 2023-05-24T21:28:21.453 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:FTbadZJvgMa2R5DsLENC0YCexqk= 2023-05-24T21:28:21.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:FTbadZJvgMa2R5DsLENC0YCexqk=', 'Content-Length': '0'} 2023-05-24T21:28:21.455 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000085a594445cbdfc48-00646e8175-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.456 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:28:21.456 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:28:21.456 DEBUG:boto:path=/qskrps-48/?prefix=obj6 2023-05-24T21:28:21.457 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj6 2023-05-24T21:28:21.457 DEBUG:boto:Method: GET 2023-05-24T21:28:21.458 DEBUG:boto:Path: /qskrps-48/?prefix=obj6 2023-05-24T21:28:21.458 DEBUG:boto:Data: 2023-05-24T21:28:21.458 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.459 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:21.459 DEBUG:boto:Port: 8001 2023-05-24T21:28:21.459 DEBUG:boto:Params: {} 2023-05-24T21:28:21.460 DEBUG:boto:Token: None 2023-05-24T21:28:21.461 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:21 GMT /qskrps-48/ 2023-05-24T21:28:21.461 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:6IoPBa2Mi8L1eD/Ct15giFLjQ8o= 2023-05-24T21:28:21.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:6IoPBa2Mi8L1eD/Ct15giFLjQ8o=', 'Content-Length': '0'} 2023-05-24T21:28:21.464 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d59c93aedeec267-00646e8175-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.465 DEBUG:boto:b'qskrps-48obj61000falseobj62023-05-24T21:28:09.598Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:21.465 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:28:21.466 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:28:21.466 DEBUG:boto:path=/qskrps-48/?prefix=obj6 2023-05-24T21:28:21.467 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj6 2023-05-24T21:28:21.467 DEBUG:boto:Method: GET 2023-05-24T21:28:21.467 DEBUG:boto:Path: /qskrps-48/?prefix=obj6 2023-05-24T21:28:21.468 DEBUG:boto:Data: 2023-05-24T21:28:21.468 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.469 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:21.469 DEBUG:boto:Port: 8000 2023-05-24T21:28:21.470 DEBUG:boto:Params: {} 2023-05-24T21:28:21.470 DEBUG:boto:Token: None 2023-05-24T21:28:21.470 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:21 GMT /qskrps-48/ 2023-05-24T21:28:21.471 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:6IoPBa2Mi8L1eD/Ct15giFLjQ8o= 2023-05-24T21:28:21.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:6IoPBa2Mi8L1eD/Ct15giFLjQ8o=', 'Content-Length': '0'} 2023-05-24T21:28:21.474 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a41bc1b72429596b-00646e8175-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.474 DEBUG:boto:b'qskrps-48obj61000falseobj62023-05-24T21:28:09.598Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:21.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-05-24T21:28:21.476 DEBUG:boto:path=/qskrps-48/obj7 2023-05-24T21:28:21.476 DEBUG:boto:auth_path=/qskrps-48/obj7 2023-05-24T21:28:21.476 DEBUG:boto:path=/qskrps-48/obj7?versionId=null 2023-05-24T21:28:21.477 DEBUG:boto:auth_path=/qskrps-48/obj7?versionId=null 2023-05-24T21:28:21.477 DEBUG:boto:Method: GET 2023-05-24T21:28:21.477 DEBUG:boto:Path: /qskrps-48/obj7?versionId=null 2023-05-24T21:28:21.478 DEBUG:boto:Data: 2023-05-24T21:28:21.478 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.478 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:21.479 DEBUG:boto:Port: 8001 2023-05-24T21:28:21.479 DEBUG:boto:Params: {} 2023-05-24T21:28:21.480 DEBUG:boto:Token: None 2023-05-24T21:28:21.480 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:21 GMT /qskrps-48/obj7?versionId=null 2023-05-24T21:28:21.480 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:a4Vq/rud/EwRzcQYgWTNkZzJcpY= 2023-05-24T21:28:21.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:a4Vq/rud/EwRzcQYgWTNkZzJcpY=', 'Content-Length': '0'} 2023-05-24T21:28:21.483 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bf516d68f8892cc8-00646e8175-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.483 DEBUG:boto:path=/qskrps-48/obj7 2023-05-24T21:28:21.484 DEBUG:boto:auth_path=/qskrps-48/obj7 2023-05-24T21:28:21.484 DEBUG:boto:path=/qskrps-48/obj7?versionId=null 2023-05-24T21:28:21.485 DEBUG:boto:auth_path=/qskrps-48/obj7?versionId=null 2023-05-24T21:28:21.485 DEBUG:boto:Method: GET 2023-05-24T21:28:21.485 DEBUG:boto:Path: /qskrps-48/obj7?versionId=null 2023-05-24T21:28:21.486 DEBUG:boto:Data: 2023-05-24T21:28:21.486 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.486 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:21.487 DEBUG:boto:Port: 8000 2023-05-24T21:28:21.487 DEBUG:boto:Params: {} 2023-05-24T21:28:21.488 DEBUG:boto:Token: None 2023-05-24T21:28:21.488 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:21 GMT /qskrps-48/obj7?versionId=null 2023-05-24T21:28:21.489 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:a4Vq/rud/EwRzcQYgWTNkZzJcpY= 2023-05-24T21:28:21.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:a4Vq/rud/EwRzcQYgWTNkZzJcpY=', 'Content-Length': '0'} 2023-05-24T21:28:21.491 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000000b8d07ac81b354c-00646e8175-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.492 DEBUG:boto:path=/qskrps-48/obj7 2023-05-24T21:28:21.492 DEBUG:boto:auth_path=/qskrps-48/obj7 2023-05-24T21:28:21.493 DEBUG:boto:path=/qskrps-48/obj7?versionId=null 2023-05-24T21:28:21.493 DEBUG:boto:auth_path=/qskrps-48/obj7?versionId=null 2023-05-24T21:28:21.494 DEBUG:boto:Method: HEAD 2023-05-24T21:28:21.494 DEBUG:boto:Path: /qskrps-48/obj7?versionId=null 2023-05-24T21:28:21.494 DEBUG:boto:Data: 2023-05-24T21:28:21.495 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.495 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:21.496 DEBUG:boto:Port: 8001 2023-05-24T21:28:21.496 DEBUG:boto:Params: {} 2023-05-24T21:28:21.496 DEBUG:boto:Token: None 2023-05-24T21:28:21.497 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:21 GMT /qskrps-48/obj7?versionId=null 2023-05-24T21:28:21.497 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Q4eMdZsiPJCbT7hDIMI1vvdUIMk= 2023-05-24T21:28:21.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Q4eMdZsiPJCbT7hDIMI1vvdUIMk=', 'Content-Length': '0'} 2023-05-24T21:28:21.499 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b4d1ab140f297671-00646e8175-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.500 DEBUG:boto:path=/qskrps-48/obj7 2023-05-24T21:28:21.500 DEBUG:boto:auth_path=/qskrps-48/obj7 2023-05-24T21:28:21.501 DEBUG:boto:path=/qskrps-48/obj7?versionId=null 2023-05-24T21:28:21.501 DEBUG:boto:auth_path=/qskrps-48/obj7?versionId=null 2023-05-24T21:28:21.501 DEBUG:boto:Method: HEAD 2023-05-24T21:28:21.502 DEBUG:boto:Path: /qskrps-48/obj7?versionId=null 2023-05-24T21:28:21.502 DEBUG:boto:Data: 2023-05-24T21:28:21.502 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.503 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:21.503 DEBUG:boto:Port: 8000 2023-05-24T21:28:21.504 DEBUG:boto:Params: {} 2023-05-24T21:28:21.504 DEBUG:boto:Token: None 2023-05-24T21:28:21.504 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:21 GMT /qskrps-48/obj7?versionId=null 2023-05-24T21:28:21.505 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Q4eMdZsiPJCbT7hDIMI1vvdUIMk= 2023-05-24T21:28:21.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Q4eMdZsiPJCbT7hDIMI1vvdUIMk=', 'Content-Length': '0'} 2023-05-24T21:28:21.507 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009bf6eb45efc73fc1-00646e8175-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-05-24T21:28:21.508 DEBUG:boto:path=/qskrps-48/obj7 2023-05-24T21:28:21.508 DEBUG:boto:auth_path=/qskrps-48/obj7 2023-05-24T21:28:21.509 DEBUG:boto:path=/qskrps-48/obj7?versionId=null 2023-05-24T21:28:21.509 DEBUG:boto:auth_path=/qskrps-48/obj7?versionId=null 2023-05-24T21:28:21.509 DEBUG:boto:Method: GET 2023-05-24T21:28:21.510 DEBUG:boto:Path: /qskrps-48/obj7?versionId=null 2023-05-24T21:28:21.510 DEBUG:boto:Data: 2023-05-24T21:28:21.510 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.511 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:21.511 DEBUG:boto:Port: 8001 2023-05-24T21:28:21.512 DEBUG:boto:Params: {} 2023-05-24T21:28:21.512 DEBUG:boto:Token: None 2023-05-24T21:28:21.512 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:21 GMT /qskrps-48/obj7?versionId=null 2023-05-24T21:28:21.513 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:a4Vq/rud/EwRzcQYgWTNkZzJcpY= 2023-05-24T21:28:21.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:a4Vq/rud/EwRzcQYgWTNkZzJcpY=', 'Content-Length': '0'} 2023-05-24T21:28:21.515 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000540aa87e3f4eccde-00646e8175-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.516 DEBUG:boto:path=/qskrps-48/obj7 2023-05-24T21:28:21.516 DEBUG:boto:auth_path=/qskrps-48/obj7 2023-05-24T21:28:21.516 DEBUG:boto:path=/qskrps-48/obj7?versionId=null 2023-05-24T21:28:21.517 DEBUG:boto:auth_path=/qskrps-48/obj7?versionId=null 2023-05-24T21:28:21.517 DEBUG:boto:Method: GET 2023-05-24T21:28:21.518 DEBUG:boto:Path: /qskrps-48/obj7?versionId=null 2023-05-24T21:28:21.518 DEBUG:boto:Data: 2023-05-24T21:28:21.519 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.519 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:21.519 DEBUG:boto:Port: 8000 2023-05-24T21:28:21.520 DEBUG:boto:Params: {} 2023-05-24T21:28:21.520 DEBUG:boto:Token: None 2023-05-24T21:28:21.521 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:21 GMT /qskrps-48/obj7?versionId=null 2023-05-24T21:28:21.521 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:a4Vq/rud/EwRzcQYgWTNkZzJcpY= 2023-05-24T21:28:21.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:a4Vq/rud/EwRzcQYgWTNkZzJcpY=', 'Content-Length': '0'} 2023-05-24T21:28:21.524 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cef63c79d4234e0b-00646e8175-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.524 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:28:21.525 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:28:21.525 DEBUG:boto:path=/qskrps-48/?prefix=obj7 2023-05-24T21:28:21.526 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj7 2023-05-24T21:28:21.526 DEBUG:boto:Method: GET 2023-05-24T21:28:21.526 DEBUG:boto:Path: /qskrps-48/?prefix=obj7 2023-05-24T21:28:21.527 DEBUG:boto:Data: 2023-05-24T21:28:21.527 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.528 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:21.528 DEBUG:boto:Port: 8001 2023-05-24T21:28:21.529 DEBUG:boto:Params: {} 2023-05-24T21:28:21.529 DEBUG:boto:Token: None 2023-05-24T21:28:21.530 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:21 GMT /qskrps-48/ 2023-05-24T21:28:21.530 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:6IoPBa2Mi8L1eD/Ct15giFLjQ8o= 2023-05-24T21:28:21.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:6IoPBa2Mi8L1eD/Ct15giFLjQ8o=', 'Content-Length': '0'} 2023-05-24T21:28:21.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fe8a7cce720234a-00646e8175-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.533 DEBUG:boto:b'qskrps-48obj71000falseobj72023-05-24T21:28:09.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:21.534 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:28:21.534 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:28:21.535 DEBUG:boto:path=/qskrps-48/?prefix=obj7 2023-05-24T21:28:21.535 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj7 2023-05-24T21:28:21.535 DEBUG:boto:Method: GET 2023-05-24T21:28:21.536 DEBUG:boto:Path: /qskrps-48/?prefix=obj7 2023-05-24T21:28:21.536 DEBUG:boto:Data: 2023-05-24T21:28:21.537 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.537 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:21.537 DEBUG:boto:Port: 8000 2023-05-24T21:28:21.538 DEBUG:boto:Params: {} 2023-05-24T21:28:21.538 DEBUG:boto:Token: None 2023-05-24T21:28:21.539 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:21 GMT /qskrps-48/ 2023-05-24T21:28:21.539 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:6IoPBa2Mi8L1eD/Ct15giFLjQ8o= 2023-05-24T21:28:21.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:6IoPBa2Mi8L1eD/Ct15giFLjQ8o=', 'Content-Length': '0'} 2023-05-24T21:28:21.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062f02102fa4fc198-00646e8175-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.542 DEBUG:boto:b'qskrps-48obj71000falseobj72023-05-24T21:28:09.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:21.543 DEBUG:boto:path=/qskrps-48/obj7 2023-05-24T21:28:21.543 DEBUG:boto:auth_path=/qskrps-48/obj7 2023-05-24T21:28:21.544 DEBUG:boto:Method: HEAD 2023-05-24T21:28:21.544 DEBUG:boto:Path: /qskrps-48/obj7 2023-05-24T21:28:21.545 DEBUG:boto:Data: 2023-05-24T21:28:21.545 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.545 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:21.546 DEBUG:boto:Port: 8001 2023-05-24T21:28:21.546 DEBUG:boto:Params: {} 2023-05-24T21:28:21.547 DEBUG:boto:Token: None 2023-05-24T21:28:21.547 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:21 GMT /qskrps-48/obj7 2023-05-24T21:28:21.547 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:01T/BFqtY2XOPCknkFkILi9QwTA= 2023-05-24T21:28:21.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:01T/BFqtY2XOPCknkFkILi9QwTA=', 'Content-Length': '0'} 2023-05-24T21:28:21.550 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c3f9f35c8dcc07bf-00646e8175-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.551 DEBUG:boto:path=/qskrps-48/obj7 2023-05-24T21:28:21.551 DEBUG:boto:auth_path=/qskrps-48/obj7 2023-05-24T21:28:21.552 DEBUG:boto:Method: HEAD 2023-05-24T21:28:21.552 DEBUG:boto:Path: /qskrps-48/obj7 2023-05-24T21:28:21.552 DEBUG:boto:Data: 2023-05-24T21:28:21.553 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.553 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:21.554 DEBUG:boto:Port: 8000 2023-05-24T21:28:21.554 DEBUG:boto:Params: {} 2023-05-24T21:28:21.555 DEBUG:boto:Token: None 2023-05-24T21:28:21.555 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:21 GMT /qskrps-48/obj7 2023-05-24T21:28:21.556 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:01T/BFqtY2XOPCknkFkILi9QwTA= 2023-05-24T21:28:21.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:01T/BFqtY2XOPCknkFkILi9QwTA=', 'Content-Length': '0'} 2023-05-24T21:28:21.558 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002dd34e7fe040e34d-00646e8175-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-05-24T21:28:21.559 DEBUG:boto:path=/qskrps-48/obj7 2023-05-24T21:28:21.559 DEBUG:boto:auth_path=/qskrps-48/obj7 2023-05-24T21:28:21.560 DEBUG:boto:Method: GET 2023-05-24T21:28:21.560 DEBUG:boto:Path: /qskrps-48/obj7 2023-05-24T21:28:21.560 DEBUG:boto:Data: 2023-05-24T21:28:21.561 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.561 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:21.561 DEBUG:boto:Port: 8001 2023-05-24T21:28:21.562 DEBUG:boto:Params: {} 2023-05-24T21:28:21.562 DEBUG:boto:Token: None 2023-05-24T21:28:21.563 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:21 GMT /qskrps-48/obj7 2023-05-24T21:28:21.563 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:dlVr8PulMTwC7FQ3dg8Z/byIwmg= 2023-05-24T21:28:21.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:dlVr8PulMTwC7FQ3dg8Z/byIwmg=', 'Content-Length': '0'} 2023-05-24T21:28:21.565 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000094eb959519a6e05d-00646e8175-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.566 DEBUG:boto:path=/qskrps-48/obj7 2023-05-24T21:28:21.566 DEBUG:boto:auth_path=/qskrps-48/obj7 2023-05-24T21:28:21.567 DEBUG:boto:Method: GET 2023-05-24T21:28:21.567 DEBUG:boto:Path: /qskrps-48/obj7 2023-05-24T21:28:21.567 DEBUG:boto:Data: 2023-05-24T21:28:21.568 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.568 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:21.568 DEBUG:boto:Port: 8000 2023-05-24T21:28:21.569 DEBUG:boto:Params: {} 2023-05-24T21:28:21.569 DEBUG:boto:Token: None 2023-05-24T21:28:21.570 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:21 GMT /qskrps-48/obj7 2023-05-24T21:28:21.570 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:dlVr8PulMTwC7FQ3dg8Z/byIwmg= 2023-05-24T21:28:21.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:dlVr8PulMTwC7FQ3dg8Z/byIwmg=', 'Content-Length': '0'} 2023-05-24T21:28:21.572 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a4b70491c4407cb3-00646e8175-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.573 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:28:21.574 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:28:21.574 DEBUG:boto:path=/qskrps-48/?prefix=obj7 2023-05-24T21:28:21.575 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj7 2023-05-24T21:28:21.575 DEBUG:boto:Method: GET 2023-05-24T21:28:21.576 DEBUG:boto:Path: /qskrps-48/?prefix=obj7 2023-05-24T21:28:21.576 DEBUG:boto:Data: 2023-05-24T21:28:21.576 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.577 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:21.577 DEBUG:boto:Port: 8001 2023-05-24T21:28:21.577 DEBUG:boto:Params: {} 2023-05-24T21:28:21.578 DEBUG:boto:Token: None 2023-05-24T21:28:21.578 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:21 GMT /qskrps-48/ 2023-05-24T21:28:21.579 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:6IoPBa2Mi8L1eD/Ct15giFLjQ8o= 2023-05-24T21:28:21.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:6IoPBa2Mi8L1eD/Ct15giFLjQ8o=', 'Content-Length': '0'} 2023-05-24T21:28:21.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5e9fc3b51b92a14-00646e8175-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.582 DEBUG:boto:b'qskrps-48obj71000falseobj72023-05-24T21:28:09.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:21.583 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:28:21.583 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:28:21.583 DEBUG:boto:path=/qskrps-48/?prefix=obj7 2023-05-24T21:28:21.584 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj7 2023-05-24T21:28:21.584 DEBUG:boto:Method: GET 2023-05-24T21:28:21.585 DEBUG:boto:Path: /qskrps-48/?prefix=obj7 2023-05-24T21:28:21.585 DEBUG:boto:Data: 2023-05-24T21:28:21.586 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.586 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:21.586 DEBUG:boto:Port: 8000 2023-05-24T21:28:21.587 DEBUG:boto:Params: {} 2023-05-24T21:28:21.587 DEBUG:boto:Token: None 2023-05-24T21:28:21.588 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:21 GMT /qskrps-48/ 2023-05-24T21:28:21.588 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:6IoPBa2Mi8L1eD/Ct15giFLjQ8o= 2023-05-24T21:28:21.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:6IoPBa2Mi8L1eD/Ct15giFLjQ8o=', 'Content-Length': '0'} 2023-05-24T21:28:21.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000810802e2e5cd0cdd-00646e8175-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.632 DEBUG:boto:b'qskrps-48obj71000falseobj72023-05-24T21:28:09.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:21.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-05-24T21:28:21.634 DEBUG:boto:path=/qskrps-48/obj8 2023-05-24T21:28:21.634 DEBUG:boto:auth_path=/qskrps-48/obj8 2023-05-24T21:28:21.635 DEBUG:boto:path=/qskrps-48/obj8?versionId=null 2023-05-24T21:28:21.635 DEBUG:boto:auth_path=/qskrps-48/obj8?versionId=null 2023-05-24T21:28:21.636 DEBUG:boto:Method: GET 2023-05-24T21:28:21.636 DEBUG:boto:Path: /qskrps-48/obj8?versionId=null 2023-05-24T21:28:21.636 DEBUG:boto:Data: 2023-05-24T21:28:21.637 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.637 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:21.637 DEBUG:boto:Port: 8001 2023-05-24T21:28:21.638 DEBUG:boto:Params: {} 2023-05-24T21:28:21.638 DEBUG:boto:Token: None 2023-05-24T21:28:21.639 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:21 GMT /qskrps-48/obj8?versionId=null 2023-05-24T21:28:21.639 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:7y5YdUdofRu/fH6U4bp3T9IZAQk= 2023-05-24T21:28:21.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:7y5YdUdofRu/fH6U4bp3T9IZAQk=', 'Content-Length': '0'} 2023-05-24T21:28:21.641 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001a2c47b5b97f391b-00646e8175-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.642 DEBUG:boto:path=/qskrps-48/obj8 2023-05-24T21:28:21.643 DEBUG:boto:auth_path=/qskrps-48/obj8 2023-05-24T21:28:21.643 DEBUG:boto:path=/qskrps-48/obj8?versionId=null 2023-05-24T21:28:21.644 DEBUG:boto:auth_path=/qskrps-48/obj8?versionId=null 2023-05-24T21:28:21.644 DEBUG:boto:Method: GET 2023-05-24T21:28:21.644 DEBUG:boto:Path: /qskrps-48/obj8?versionId=null 2023-05-24T21:28:21.645 DEBUG:boto:Data: 2023-05-24T21:28:21.645 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.646 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:21.646 DEBUG:boto:Port: 8000 2023-05-24T21:28:21.647 DEBUG:boto:Params: {} 2023-05-24T21:28:21.647 DEBUG:boto:Token: None 2023-05-24T21:28:21.648 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:21 GMT /qskrps-48/obj8?versionId=null 2023-05-24T21:28:21.648 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:7y5YdUdofRu/fH6U4bp3T9IZAQk= 2023-05-24T21:28:21.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:7y5YdUdofRu/fH6U4bp3T9IZAQk=', 'Content-Length': '0'} 2023-05-24T21:28:21.650 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009ef8ddc8cd7065b5-00646e8175-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.651 DEBUG:boto:path=/qskrps-48/obj8 2023-05-24T21:28:21.651 DEBUG:boto:auth_path=/qskrps-48/obj8 2023-05-24T21:28:21.652 DEBUG:boto:path=/qskrps-48/obj8?versionId=null 2023-05-24T21:28:21.652 DEBUG:boto:auth_path=/qskrps-48/obj8?versionId=null 2023-05-24T21:28:21.652 DEBUG:boto:Method: HEAD 2023-05-24T21:28:21.653 DEBUG:boto:Path: /qskrps-48/obj8?versionId=null 2023-05-24T21:28:21.653 DEBUG:boto:Data: 2023-05-24T21:28:21.653 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.654 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:21.654 DEBUG:boto:Port: 8001 2023-05-24T21:28:21.654 DEBUG:boto:Params: {} 2023-05-24T21:28:21.655 DEBUG:boto:Token: None 2023-05-24T21:28:21.655 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:21 GMT /qskrps-48/obj8?versionId=null 2023-05-24T21:28:21.656 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:UQaaTSyuc1IOPl1CLBxMj0Rm8Hs= 2023-05-24T21:28:21.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:UQaaTSyuc1IOPl1CLBxMj0Rm8Hs=', 'Content-Length': '0'} 2023-05-24T21:28:21.658 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006fd7717f135c86ac-00646e8175-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.658 DEBUG:boto:path=/qskrps-48/obj8 2023-05-24T21:28:21.659 DEBUG:boto:auth_path=/qskrps-48/obj8 2023-05-24T21:28:21.659 DEBUG:boto:path=/qskrps-48/obj8?versionId=null 2023-05-24T21:28:21.659 DEBUG:boto:auth_path=/qskrps-48/obj8?versionId=null 2023-05-24T21:28:21.660 DEBUG:boto:Method: HEAD 2023-05-24T21:28:21.660 DEBUG:boto:Path: /qskrps-48/obj8?versionId=null 2023-05-24T21:28:21.660 DEBUG:boto:Data: 2023-05-24T21:28:21.661 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.661 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:21.661 DEBUG:boto:Port: 8000 2023-05-24T21:28:21.662 DEBUG:boto:Params: {} 2023-05-24T21:28:21.662 DEBUG:boto:Token: None 2023-05-24T21:28:21.662 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:21 GMT /qskrps-48/obj8?versionId=null 2023-05-24T21:28:21.663 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:UQaaTSyuc1IOPl1CLBxMj0Rm8Hs= 2023-05-24T21:28:21.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:UQaaTSyuc1IOPl1CLBxMj0Rm8Hs=', 'Content-Length': '0'} 2023-05-24T21:28:21.666 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000b8df3ed7adb58a7-00646e8175-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-05-24T21:28:21.667 DEBUG:boto:path=/qskrps-48/obj8 2023-05-24T21:28:21.667 DEBUG:boto:auth_path=/qskrps-48/obj8 2023-05-24T21:28:21.668 DEBUG:boto:path=/qskrps-48/obj8?versionId=null 2023-05-24T21:28:21.668 DEBUG:boto:auth_path=/qskrps-48/obj8?versionId=null 2023-05-24T21:28:21.668 DEBUG:boto:Method: GET 2023-05-24T21:28:21.669 DEBUG:boto:Path: /qskrps-48/obj8?versionId=null 2023-05-24T21:28:21.669 DEBUG:boto:Data: 2023-05-24T21:28:21.669 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.670 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:21.670 DEBUG:boto:Port: 8001 2023-05-24T21:28:21.670 DEBUG:boto:Params: {} 2023-05-24T21:28:21.671 DEBUG:boto:Token: None 2023-05-24T21:28:21.671 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:21 GMT /qskrps-48/obj8?versionId=null 2023-05-24T21:28:21.672 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:7y5YdUdofRu/fH6U4bp3T9IZAQk= 2023-05-24T21:28:21.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:7y5YdUdofRu/fH6U4bp3T9IZAQk=', 'Content-Length': '0'} 2023-05-24T21:28:21.674 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000669a2a738c4b115f-00646e8175-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.674 DEBUG:boto:path=/qskrps-48/obj8 2023-05-24T21:28:21.675 DEBUG:boto:auth_path=/qskrps-48/obj8 2023-05-24T21:28:21.675 DEBUG:boto:path=/qskrps-48/obj8?versionId=null 2023-05-24T21:28:21.676 DEBUG:boto:auth_path=/qskrps-48/obj8?versionId=null 2023-05-24T21:28:21.676 DEBUG:boto:Method: GET 2023-05-24T21:28:21.677 DEBUG:boto:Path: /qskrps-48/obj8?versionId=null 2023-05-24T21:28:21.677 DEBUG:boto:Data: 2023-05-24T21:28:21.677 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.678 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:21.678 DEBUG:boto:Port: 8000 2023-05-24T21:28:21.679 DEBUG:boto:Params: {} 2023-05-24T21:28:21.679 DEBUG:boto:Token: None 2023-05-24T21:28:21.680 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:21 GMT /qskrps-48/obj8?versionId=null 2023-05-24T21:28:21.680 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:7y5YdUdofRu/fH6U4bp3T9IZAQk= 2023-05-24T21:28:21.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:7y5YdUdofRu/fH6U4bp3T9IZAQk=', 'Content-Length': '0'} 2023-05-24T21:28:21.683 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004d1fe34a1c87e718-00646e8175-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.683 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:28:21.684 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:28:21.684 DEBUG:boto:path=/qskrps-48/?prefix=obj8 2023-05-24T21:28:21.685 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj8 2023-05-24T21:28:21.685 DEBUG:boto:Method: GET 2023-05-24T21:28:21.685 DEBUG:boto:Path: /qskrps-48/?prefix=obj8 2023-05-24T21:28:21.686 DEBUG:boto:Data: 2023-05-24T21:28:21.686 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.686 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:21.687 DEBUG:boto:Port: 8001 2023-05-24T21:28:21.687 DEBUG:boto:Params: {} 2023-05-24T21:28:21.687 DEBUG:boto:Token: None 2023-05-24T21:28:21.688 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:21 GMT /qskrps-48/ 2023-05-24T21:28:21.688 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:6IoPBa2Mi8L1eD/Ct15giFLjQ8o= 2023-05-24T21:28:21.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:6IoPBa2Mi8L1eD/Ct15giFLjQ8o=', 'Content-Length': '0'} 2023-05-24T21:28:21.691 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4c883b75d60c6c3-00646e8175-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.692 DEBUG:boto:b'qskrps-48obj81000falseobj82023-05-24T21:28:09.670Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:21.692 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:28:21.693 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:28:21.693 DEBUG:boto:path=/qskrps-48/?prefix=obj8 2023-05-24T21:28:21.694 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj8 2023-05-24T21:28:21.694 DEBUG:boto:Method: GET 2023-05-24T21:28:21.694 DEBUG:boto:Path: /qskrps-48/?prefix=obj8 2023-05-24T21:28:21.695 DEBUG:boto:Data: 2023-05-24T21:28:21.695 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.696 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:21.696 DEBUG:boto:Port: 8000 2023-05-24T21:28:21.697 DEBUG:boto:Params: {} 2023-05-24T21:28:21.697 DEBUG:boto:Token: None 2023-05-24T21:28:21.697 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:21 GMT /qskrps-48/ 2023-05-24T21:28:21.698 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:6IoPBa2Mi8L1eD/Ct15giFLjQ8o= 2023-05-24T21:28:21.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:6IoPBa2Mi8L1eD/Ct15giFLjQ8o=', 'Content-Length': '0'} 2023-05-24T21:28:21.700 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d1e49314a379774-00646e8175-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.701 DEBUG:boto:b'qskrps-48obj81000falseobj82023-05-24T21:28:09.670Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:21.702 DEBUG:boto:path=/qskrps-48/obj8 2023-05-24T21:28:21.702 DEBUG:boto:auth_path=/qskrps-48/obj8 2023-05-24T21:28:21.702 DEBUG:boto:Method: HEAD 2023-05-24T21:28:21.703 DEBUG:boto:Path: /qskrps-48/obj8 2023-05-24T21:28:21.703 DEBUG:boto:Data: 2023-05-24T21:28:21.704 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.704 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:21.704 DEBUG:boto:Port: 8001 2023-05-24T21:28:21.705 DEBUG:boto:Params: {} 2023-05-24T21:28:21.705 DEBUG:boto:Token: None 2023-05-24T21:28:21.706 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:21 GMT /qskrps-48/obj8 2023-05-24T21:28:21.706 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:vw56+lO200uxikUO/lJ/qSX1fwQ= 2023-05-24T21:28:21.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:vw56+lO200uxikUO/lJ/qSX1fwQ=', 'Content-Length': '0'} 2023-05-24T21:28:21.709 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000415ff66780d57a12-00646e8175-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.709 DEBUG:boto:path=/qskrps-48/obj8 2023-05-24T21:28:21.710 DEBUG:boto:auth_path=/qskrps-48/obj8 2023-05-24T21:28:21.710 DEBUG:boto:Method: HEAD 2023-05-24T21:28:21.711 DEBUG:boto:Path: /qskrps-48/obj8 2023-05-24T21:28:21.711 DEBUG:boto:Data: 2023-05-24T21:28:21.711 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.712 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:21.712 DEBUG:boto:Port: 8000 2023-05-24T21:28:21.712 DEBUG:boto:Params: {} 2023-05-24T21:28:21.713 DEBUG:boto:Token: None 2023-05-24T21:28:21.713 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:21 GMT /qskrps-48/obj8 2023-05-24T21:28:21.713 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:vw56+lO200uxikUO/lJ/qSX1fwQ= 2023-05-24T21:28:21.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:vw56+lO200uxikUO/lJ/qSX1fwQ=', 'Content-Length': '0'} 2023-05-24T21:28:21.716 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000490f7f0a8c7e6a12-00646e8175-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-05-24T21:28:21.717 DEBUG:boto:path=/qskrps-48/obj8 2023-05-24T21:28:21.717 DEBUG:boto:auth_path=/qskrps-48/obj8 2023-05-24T21:28:21.718 DEBUG:boto:Method: GET 2023-05-24T21:28:21.718 DEBUG:boto:Path: /qskrps-48/obj8 2023-05-24T21:28:21.718 DEBUG:boto:Data: 2023-05-24T21:28:21.719 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.719 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:21.719 DEBUG:boto:Port: 8001 2023-05-24T21:28:21.720 DEBUG:boto:Params: {} 2023-05-24T21:28:21.720 DEBUG:boto:Token: None 2023-05-24T21:28:21.721 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:21 GMT /qskrps-48/obj8 2023-05-24T21:28:21.721 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:qKuEmrmC9iJ573pr5CqEBtkKqko= 2023-05-24T21:28:21.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qKuEmrmC9iJ573pr5CqEBtkKqko=', 'Content-Length': '0'} 2023-05-24T21:28:21.723 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d072e5dd8cda30d4-00646e8175-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.764 DEBUG:boto:path=/qskrps-48/obj8 2023-05-24T21:28:21.765 DEBUG:boto:auth_path=/qskrps-48/obj8 2023-05-24T21:28:21.765 DEBUG:boto:Method: GET 2023-05-24T21:28:21.766 DEBUG:boto:Path: /qskrps-48/obj8 2023-05-24T21:28:21.766 DEBUG:boto:Data: 2023-05-24T21:28:21.766 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.767 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:21.767 DEBUG:boto:Port: 8000 2023-05-24T21:28:21.768 DEBUG:boto:Params: {} 2023-05-24T21:28:21.768 DEBUG:boto:Token: None 2023-05-24T21:28:21.768 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:21 GMT /qskrps-48/obj8 2023-05-24T21:28:21.769 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:qKuEmrmC9iJ573pr5CqEBtkKqko= 2023-05-24T21:28:21.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qKuEmrmC9iJ573pr5CqEBtkKqko=', 'Content-Length': '0'} 2023-05-24T21:28:21.771 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003371cc029a4af849-00646e8175-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.772 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:28:21.773 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:28:21.773 DEBUG:boto:path=/qskrps-48/?prefix=obj8 2023-05-24T21:28:21.773 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj8 2023-05-24T21:28:21.774 DEBUG:boto:Method: GET 2023-05-24T21:28:21.774 DEBUG:boto:Path: /qskrps-48/?prefix=obj8 2023-05-24T21:28:21.775 DEBUG:boto:Data: 2023-05-24T21:28:21.775 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.775 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:21.776 DEBUG:boto:Port: 8001 2023-05-24T21:28:21.776 DEBUG:boto:Params: {} 2023-05-24T21:28:21.777 DEBUG:boto:Token: None 2023-05-24T21:28:21.777 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:21 GMT /qskrps-48/ 2023-05-24T21:28:21.778 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:6IoPBa2Mi8L1eD/Ct15giFLjQ8o= 2023-05-24T21:28:21.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:6IoPBa2Mi8L1eD/Ct15giFLjQ8o=', 'Content-Length': '0'} 2023-05-24T21:28:21.780 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f20ae02907dd9142-00646e8175-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.781 DEBUG:boto:b'qskrps-48obj81000falseobj82023-05-24T21:28:09.670Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:21.781 DEBUG:boto:path=/qskrps-48/ 2023-05-24T21:28:21.782 DEBUG:boto:auth_path=/qskrps-48/ 2023-05-24T21:28:21.782 DEBUG:boto:path=/qskrps-48/?prefix=obj8 2023-05-24T21:28:21.782 DEBUG:boto:auth_path=/qskrps-48/?prefix=obj8 2023-05-24T21:28:21.783 DEBUG:boto:Method: GET 2023-05-24T21:28:21.783 DEBUG:boto:Path: /qskrps-48/?prefix=obj8 2023-05-24T21:28:21.784 DEBUG:boto:Data: 2023-05-24T21:28:21.784 DEBUG:boto:Headers: {} 2023-05-24T21:28:21.784 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:21.785 DEBUG:boto:Port: 8000 2023-05-24T21:28:21.785 DEBUG:boto:Params: {} 2023-05-24T21:28:21.785 DEBUG:boto:Token: None 2023-05-24T21:28:21.786 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:21 GMT /qskrps-48/ 2023-05-24T21:28:21.786 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:6IoPBa2Mi8L1eD/Ct15giFLjQ8o= 2023-05-24T21:28:21.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:21 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:6IoPBa2Mi8L1eD/Ct15giFLjQ8o=', 'Content-Length': '0'} 2023-05-24T21:28:21.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000accaaf19aa33684c-00646e8175-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:21.790 DEBUG:boto:b'qskrps-48obj81000falseobj82023-05-24T21:28:09.670Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:21.790 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qskrps-48 zones={a2, a1} 2023-05-24T21:28:21.791 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=qskrps-49 2023-05-24T21:28:21.791 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:28:21.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:28:21.792 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-49 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:28:21.854 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:28:21.854 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:28:21.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:28:21.885+0000 7f0def999780 1 waiting for incremental sync to catch up: 2023-05-24T21:28:21.892 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [00000000003.25.1, 00000000002.13.1, 00000000003.31.1, 00000000002.25.1, 00000000003.62.1, 00000000002.26.1, 00000000002.24.1, 00000000003.25.1, 00000000002.17.1, 00000000002.32.1, 00000000002.63.1] 2023-05-24T21:28:21.893 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [00000000004.28.6, 00000000002.13.1, 00000000004.34.6, 00000000002.25.1, 00000000004.65.6, 00000000002.26.1, 00000000002.24.1, 00000000004.27.6, 00000000002.17.1, 00000000002.32.1, 00000000002.63.1] 2023-05-24T21:28:26.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:28:26.887+0000 7f0def999780 1 waiting for incremental sync to catch up: 2023-05-24T21:28:26.894 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [00000000003.25.1, 00000000002.13.1, 00000000003.31.1, 00000000002.25.1, 00000000003.62.1, 00000000002.26.1, 00000000002.24.1, 00000000003.25.1, 00000000002.17.1, 00000000002.32.1, 00000000002.63.1] 2023-05-24T21:28:26.895 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [00000000004.28.6, 00000000002.13.1, 00000000004.34.6, 00000000002.25.1, 00000000004.65.6, 00000000002.26.1, 00000000002.24.1, 00000000004.27.6, 00000000002.17.1, 00000000002.32.1, 00000000002.63.1] 2023-05-24T21:28:31.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:28:31.889+0000 7f0def999780 1 waiting for incremental sync to catch up: 2023-05-24T21:28:31.896 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [00000000003.25.1, 00000000002.13.1, 00000000003.31.1, 00000000002.25.1, 00000000003.62.1, 00000000002.26.1, 00000000002.24.1, 00000000003.25.1, 00000000002.17.1, 00000000002.32.1, 00000000002.63.1] 2023-05-24T21:28:31.896 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [00000000004.28.6, 00000000002.13.1, 00000000004.34.6, 00000000002.25.1, 00000000004.65.6, 00000000002.26.1, 00000000002.24.1, 00000000004.27.6, 00000000002.17.1, 00000000002.32.1, 00000000002.63.1] 2023-05-24T21:28:36.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:28:36.890+0000 7f0def999780 1 bucket sync caught up with source: 2023-05-24T21:28:36.897 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [00000000004.28.6, 00000000002.13.1, 00000000004.34.6, 00000000002.25.1, 00000000004.65.6, 00000000002.26.1, 00000000002.24.1, 00000000004.27.6, 00000000002.17.1, 00000000002.32.1, 00000000002.63.1] 2023-05-24T21:28:36.897 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [00000000004.28.6, 00000000002.13.1, 00000000004.34.6, 00000000002.25.1, 00000000004.65.6, 00000000002.26.1, 00000000002.24.1, 00000000004.27.6, 00000000002.17.1, 00000000002.32.1, 00000000002.63.1] 2023-05-24T21:28:36.897 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:28:36.890+0000 7f0def999780 0 bucket checkpoint complete 2023-05-24T21:28:36.920 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=qskrps-49 2023-05-24T21:28:36.920 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:28:36.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:28:36.921 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-49 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:28:36.982 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:28:36.983 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:28:37.025 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:28:37.018+0000 7f7740e2b780 1 bucket sync caught up with source: 2023-05-24T21:28:37.025 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [00000000004.370.1, 00000000002.349.1, 00000000004.541.1, 00000000002.350.1, 00000000004.565.1, 00000000002.371.1, 00000000002.1307.1, 00000000004.1289.1, 00000000002.208.1, 00000000002.542.1, 00000000002.566.1] 2023-05-24T21:28:37.025 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [00000000004.370.1, 00000000002.349.1, 00000000004.541.1, 00000000002.350.1, 00000000004.565.1, 00000000002.371.1, 00000000002.1307.1, 00000000004.1289.1, 00000000002.208.1, 00000000002.542.1, 00000000002.566.1] 2023-05-24T21:28:37.025 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:28:37.018+0000 7f7740e2b780 0 bucket checkpoint complete 2023-05-24T21:28:37.039 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qskrps-49 zones={a2, a1} 2023-05-24T21:28:37.039 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:37.039 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:37.040 DEBUG:boto:Method: HEAD 2023-05-24T21:28:37.040 DEBUG:boto:Path: /qskrps-49/ 2023-05-24T21:28:37.041 DEBUG:boto:Data: 2023-05-24T21:28:37.041 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.041 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.042 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.042 DEBUG:boto:Params: {} 2023-05-24T21:28:37.042 DEBUG:boto:Token: None 2023-05-24T21:28:37.043 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:37 GMT /qskrps-49/ 2023-05-24T21:28:37.043 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:cOE/FfG8ap7wFzYl4ABpoFe9Lv0= 2023-05-24T21:28:37.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:cOE/FfG8ap7wFzYl4ABpoFe9Lv0=', 'Content-Length': '0'} 2023-05-24T21:28:37.046 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000edcb670952896b4c-00646e8185-1625-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.046 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:37.047 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:37.047 DEBUG:boto:Method: HEAD 2023-05-24T21:28:37.047 DEBUG:boto:Path: /qskrps-49/ 2023-05-24T21:28:37.048 DEBUG:boto:Data: 2023-05-24T21:28:37.048 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.048 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.049 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.049 DEBUG:boto:Params: {} 2023-05-24T21:28:37.049 DEBUG:boto:Token: None 2023-05-24T21:28:37.050 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:37 GMT /qskrps-49/ 2023-05-24T21:28:37.050 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:cOE/FfG8ap7wFzYl4ABpoFe9Lv0= 2023-05-24T21:28:37.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:cOE/FfG8ap7wFzYl4ABpoFe9Lv0=', 'Content-Length': '0'} 2023-05-24T21:28:37.053 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c8bb0861bb6edd8-00646e8185-1627-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.053 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-24T21:28:37.054 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:37.054 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:37.054 DEBUG:boto:path=/qskrps-49/?versions&max-keys=999 2023-05-24T21:28:37.055 DEBUG:boto:auth_path=/qskrps-49/?versions&max-keys=999 2023-05-24T21:28:37.055 DEBUG:boto:Method: GET 2023-05-24T21:28:37.055 DEBUG:boto:Path: /qskrps-49/?versions&max-keys=999 2023-05-24T21:28:37.056 DEBUG:boto:Data: 2023-05-24T21:28:37.056 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.056 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.057 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.057 DEBUG:boto:Params: {} 2023-05-24T21:28:37.057 DEBUG:boto:Token: None 2023-05-24T21:28:37.058 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/?versions 2023-05-24T21:28:37.058 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:yD+bjvQxoi4OZuOKYAzc+UJdUUw= 2023-05-24T21:28:37.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:yD+bjvQxoi4OZuOKYAzc+UJdUUw=', 'Content-Length': '0'} 2023-05-24T21:28:37.061 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffed1cda5051ff4f-00646e8185-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.061 DEBUG:boto:b'qskrps-49999falseobj1nulltrue2023-05-24T21:27:06.420Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-24T21:27:06.438Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-24T21:27:06.495Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-24T21:27:06.514Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-05-24T21:28:09.687Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-05-24T21:28:09.703Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-05-24T21:28:09.762Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-05-24T21:28:09.781Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:37.062 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-05-24T21:28:37.063 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-05-24T21:28:37.063 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-05-24T21:28:37.063 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-05-24T21:28:37.064 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-05-24T21:28:37.064 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-05-24T21:28:37.064 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-05-24T21:28:37.064 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-05-24T21:28:37.065 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-24T21:28:37.065 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:37.065 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:37.066 DEBUG:boto:path=/qskrps-49/?versions&max-keys=999 2023-05-24T21:28:37.066 DEBUG:boto:auth_path=/qskrps-49/?versions&max-keys=999 2023-05-24T21:28:37.066 DEBUG:boto:Method: GET 2023-05-24T21:28:37.067 DEBUG:boto:Path: /qskrps-49/?versions&max-keys=999 2023-05-24T21:28:37.067 DEBUG:boto:Data: 2023-05-24T21:28:37.067 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.068 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.068 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.069 DEBUG:boto:Params: {} 2023-05-24T21:28:37.069 DEBUG:boto:Token: None 2023-05-24T21:28:37.069 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/?versions 2023-05-24T21:28:37.070 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:yD+bjvQxoi4OZuOKYAzc+UJdUUw= 2023-05-24T21:28:37.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:yD+bjvQxoi4OZuOKYAzc+UJdUUw=', 'Content-Length': '0'} 2023-05-24T21:28:37.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eaffb6c548756720-00646e8185-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.073 DEBUG:boto:b'qskrps-49999falseobj1nulltrue2023-05-24T21:27:06.420Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-24T21:27:06.438Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-24T21:27:06.495Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-24T21:27:06.514Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-05-24T21:28:09.687Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-05-24T21:28:09.703Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-05-24T21:28:09.762Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-05-24T21:28:09.781Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:37.073 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-05-24T21:28:37.074 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-05-24T21:28:37.074 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-05-24T21:28:37.074 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-05-24T21:28:37.075 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-05-24T21:28:37.075 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-05-24T21:28:37.075 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-05-24T21:28:37.076 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-05-24T21:28:37.076 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:37.076 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:37.077 DEBUG:boto:path=/qskrps-49/?versions&max-keys=999 2023-05-24T21:28:37.077 DEBUG:boto:auth_path=/qskrps-49/?versions&max-keys=999 2023-05-24T21:28:37.077 DEBUG:boto:Method: GET 2023-05-24T21:28:37.078 DEBUG:boto:Path: /qskrps-49/?versions&max-keys=999 2023-05-24T21:28:37.078 DEBUG:boto:Data: 2023-05-24T21:28:37.079 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.079 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.079 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.080 DEBUG:boto:Params: {} 2023-05-24T21:28:37.080 DEBUG:boto:Token: None 2023-05-24T21:28:37.080 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/?versions 2023-05-24T21:28:37.081 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:yD+bjvQxoi4OZuOKYAzc+UJdUUw= 2023-05-24T21:28:37.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:yD+bjvQxoi4OZuOKYAzc+UJdUUw=', 'Content-Length': '0'} 2023-05-24T21:28:37.083 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f61867e5a05faae-00646e8185-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.084 DEBUG:boto:b'qskrps-49999falseobj1nulltrue2023-05-24T21:27:06.420Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-24T21:27:06.438Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-24T21:27:06.495Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-24T21:27:06.514Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-05-24T21:28:09.687Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-05-24T21:28:09.703Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-05-24T21:28:09.762Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-05-24T21:28:09.781Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:37.085 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:37.085 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:37.085 DEBUG:boto:path=/qskrps-49/?versions&max-keys=999 2023-05-24T21:28:37.086 DEBUG:boto:auth_path=/qskrps-49/?versions&max-keys=999 2023-05-24T21:28:37.086 DEBUG:boto:Method: GET 2023-05-24T21:28:37.086 DEBUG:boto:Path: /qskrps-49/?versions&max-keys=999 2023-05-24T21:28:37.087 DEBUG:boto:Data: 2023-05-24T21:28:37.087 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.087 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.088 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.088 DEBUG:boto:Params: {} 2023-05-24T21:28:37.088 DEBUG:boto:Token: None 2023-05-24T21:28:37.089 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/?versions 2023-05-24T21:28:37.089 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:yD+bjvQxoi4OZuOKYAzc+UJdUUw= 2023-05-24T21:28:37.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:yD+bjvQxoi4OZuOKYAzc+UJdUUw=', 'Content-Length': '0'} 2023-05-24T21:28:37.092 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efdde0c8e02d9d47-00646e8185-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.093 DEBUG:boto:b'qskrps-49999falseobj1nulltrue2023-05-24T21:27:06.420Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-24T21:27:06.438Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-24T21:27:06.495Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-24T21:27:06.514Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-05-24T21:28:09.687Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-05-24T21:28:09.703Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-05-24T21:28:09.762Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-05-24T21:28:09.781Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:37.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-24T21:28:37.094 DEBUG:boto:path=/qskrps-49/obj1 2023-05-24T21:28:37.094 DEBUG:boto:auth_path=/qskrps-49/obj1 2023-05-24T21:28:37.095 DEBUG:boto:path=/qskrps-49/obj1?versionId=null 2023-05-24T21:28:37.095 DEBUG:boto:auth_path=/qskrps-49/obj1?versionId=null 2023-05-24T21:28:37.095 DEBUG:boto:Method: GET 2023-05-24T21:28:37.096 DEBUG:boto:Path: /qskrps-49/obj1?versionId=null 2023-05-24T21:28:37.096 DEBUG:boto:Data: 2023-05-24T21:28:37.097 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.097 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.097 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.098 DEBUG:boto:Params: {} 2023-05-24T21:28:37.098 DEBUG:boto:Token: None 2023-05-24T21:28:37.099 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj1?versionId=null 2023-05-24T21:28:37.099 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:LYzzaJ625B1GsZxyXtDN504S/q4= 2023-05-24T21:28:37.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:LYzzaJ625B1GsZxyXtDN504S/q4=', 'Content-Length': '0'} 2023-05-24T21:28:37.101 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009bc26cc583deece6-00646e8185-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.102 DEBUG:boto:path=/qskrps-49/obj1 2023-05-24T21:28:37.102 DEBUG:boto:auth_path=/qskrps-49/obj1 2023-05-24T21:28:37.103 DEBUG:boto:path=/qskrps-49/obj1?versionId=null 2023-05-24T21:28:37.103 DEBUG:boto:auth_path=/qskrps-49/obj1?versionId=null 2023-05-24T21:28:37.103 DEBUG:boto:Method: GET 2023-05-24T21:28:37.104 DEBUG:boto:Path: /qskrps-49/obj1?versionId=null 2023-05-24T21:28:37.104 DEBUG:boto:Data: 2023-05-24T21:28:37.105 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.105 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.105 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.106 DEBUG:boto:Params: {} 2023-05-24T21:28:37.106 DEBUG:boto:Token: None 2023-05-24T21:28:37.107 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj1?versionId=null 2023-05-24T21:28:37.107 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:LYzzaJ625B1GsZxyXtDN504S/q4= 2023-05-24T21:28:37.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:LYzzaJ625B1GsZxyXtDN504S/q4=', 'Content-Length': '0'} 2023-05-24T21:28:37.110 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006ca01616d195df50-00646e8185-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.152 DEBUG:boto:path=/qskrps-49/obj1 2023-05-24T21:28:37.153 DEBUG:boto:auth_path=/qskrps-49/obj1 2023-05-24T21:28:37.153 DEBUG:boto:path=/qskrps-49/obj1?versionId=null 2023-05-24T21:28:37.153 DEBUG:boto:auth_path=/qskrps-49/obj1?versionId=null 2023-05-24T21:28:37.154 DEBUG:boto:Method: HEAD 2023-05-24T21:28:37.154 DEBUG:boto:Path: /qskrps-49/obj1?versionId=null 2023-05-24T21:28:37.155 DEBUG:boto:Data: 2023-05-24T21:28:37.155 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.156 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.156 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.157 DEBUG:boto:Params: {} 2023-05-24T21:28:37.157 DEBUG:boto:Token: None 2023-05-24T21:28:37.157 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj1?versionId=null 2023-05-24T21:28:37.158 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:xuL2TkCGfIJQXDHOWMDTgvgf+Ek= 2023-05-24T21:28:37.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:xuL2TkCGfIJQXDHOWMDTgvgf+Ek=', 'Content-Length': '0'} 2023-05-24T21:28:37.160 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006ac17c88dea9d059-00646e8185-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.160 DEBUG:boto:path=/qskrps-49/obj1 2023-05-24T21:28:37.161 DEBUG:boto:auth_path=/qskrps-49/obj1 2023-05-24T21:28:37.161 DEBUG:boto:path=/qskrps-49/obj1?versionId=null 2023-05-24T21:28:37.161 DEBUG:boto:auth_path=/qskrps-49/obj1?versionId=null 2023-05-24T21:28:37.162 DEBUG:boto:Method: HEAD 2023-05-24T21:28:37.162 DEBUG:boto:Path: /qskrps-49/obj1?versionId=null 2023-05-24T21:28:37.163 DEBUG:boto:Data: 2023-05-24T21:28:37.163 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.163 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.164 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.164 DEBUG:boto:Params: {} 2023-05-24T21:28:37.164 DEBUG:boto:Token: None 2023-05-24T21:28:37.165 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj1?versionId=null 2023-05-24T21:28:37.165 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:xuL2TkCGfIJQXDHOWMDTgvgf+Ek= 2023-05-24T21:28:37.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:xuL2TkCGfIJQXDHOWMDTgvgf+Ek=', 'Content-Length': '0'} 2023-05-24T21:28:37.167 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006ca8e52c5619bc10-00646e8185-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-24T21:28:37.168 DEBUG:boto:path=/qskrps-49/obj1 2023-05-24T21:28:37.169 DEBUG:boto:auth_path=/qskrps-49/obj1 2023-05-24T21:28:37.169 DEBUG:boto:path=/qskrps-49/obj1?versionId=null 2023-05-24T21:28:37.169 DEBUG:boto:auth_path=/qskrps-49/obj1?versionId=null 2023-05-24T21:28:37.170 DEBUG:boto:Method: GET 2023-05-24T21:28:37.170 DEBUG:boto:Path: /qskrps-49/obj1?versionId=null 2023-05-24T21:28:37.171 DEBUG:boto:Data: 2023-05-24T21:28:37.171 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.171 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.172 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.172 DEBUG:boto:Params: {} 2023-05-24T21:28:37.173 DEBUG:boto:Token: None 2023-05-24T21:28:37.173 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj1?versionId=null 2023-05-24T21:28:37.174 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:LYzzaJ625B1GsZxyXtDN504S/q4= 2023-05-24T21:28:37.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:LYzzaJ625B1GsZxyXtDN504S/q4=', 'Content-Length': '0'} 2023-05-24T21:28:37.176 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000008734b4170d68fc0-00646e8185-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.176 DEBUG:boto:path=/qskrps-49/obj1 2023-05-24T21:28:37.177 DEBUG:boto:auth_path=/qskrps-49/obj1 2023-05-24T21:28:37.177 DEBUG:boto:path=/qskrps-49/obj1?versionId=null 2023-05-24T21:28:37.177 DEBUG:boto:auth_path=/qskrps-49/obj1?versionId=null 2023-05-24T21:28:37.178 DEBUG:boto:Method: GET 2023-05-24T21:28:37.178 DEBUG:boto:Path: /qskrps-49/obj1?versionId=null 2023-05-24T21:28:37.178 DEBUG:boto:Data: 2023-05-24T21:28:37.179 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.179 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.179 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.180 DEBUG:boto:Params: {} 2023-05-24T21:28:37.180 DEBUG:boto:Token: None 2023-05-24T21:28:37.181 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj1?versionId=null 2023-05-24T21:28:37.181 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:LYzzaJ625B1GsZxyXtDN504S/q4= 2023-05-24T21:28:37.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:LYzzaJ625B1GsZxyXtDN504S/q4=', 'Content-Length': '0'} 2023-05-24T21:28:37.183 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d4199cf16aba5c5b-00646e8185-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.184 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:37.184 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:37.185 DEBUG:boto:path=/qskrps-49/?prefix=obj1 2023-05-24T21:28:37.185 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj1 2023-05-24T21:28:37.185 DEBUG:boto:Method: GET 2023-05-24T21:28:37.186 DEBUG:boto:Path: /qskrps-49/?prefix=obj1 2023-05-24T21:28:37.186 DEBUG:boto:Data: 2023-05-24T21:28:37.186 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.187 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.187 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.187 DEBUG:boto:Params: {} 2023-05-24T21:28:37.188 DEBUG:boto:Token: None 2023-05-24T21:28:37.188 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/ 2023-05-24T21:28:37.189 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ= 2023-05-24T21:28:37.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ=', 'Content-Length': '0'} 2023-05-24T21:28:37.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0e3325a0473f25b-00646e8185-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.192 DEBUG:boto:b'qskrps-49obj11000falseobj12023-05-24T21:27:06.420Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:37.192 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:37.193 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:37.193 DEBUG:boto:path=/qskrps-49/?prefix=obj1 2023-05-24T21:28:37.193 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj1 2023-05-24T21:28:37.194 DEBUG:boto:Method: GET 2023-05-24T21:28:37.194 DEBUG:boto:Path: /qskrps-49/?prefix=obj1 2023-05-24T21:28:37.194 DEBUG:boto:Data: 2023-05-24T21:28:37.195 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.195 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.195 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.196 DEBUG:boto:Params: {} 2023-05-24T21:28:37.196 DEBUG:boto:Token: None 2023-05-24T21:28:37.197 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/ 2023-05-24T21:28:37.197 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ= 2023-05-24T21:28:37.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ=', 'Content-Length': '0'} 2023-05-24T21:28:37.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e918044aab271e4-00646e8185-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.200 DEBUG:boto:b'qskrps-49obj11000falseobj12023-05-24T21:27:06.420Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:37.201 DEBUG:boto:path=/qskrps-49/obj1 2023-05-24T21:28:37.201 DEBUG:boto:auth_path=/qskrps-49/obj1 2023-05-24T21:28:37.201 DEBUG:boto:Method: HEAD 2023-05-24T21:28:37.202 DEBUG:boto:Path: /qskrps-49/obj1 2023-05-24T21:28:37.202 DEBUG:boto:Data: 2023-05-24T21:28:37.202 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.203 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.203 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.203 DEBUG:boto:Params: {} 2023-05-24T21:28:37.204 DEBUG:boto:Token: None 2023-05-24T21:28:37.204 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj1 2023-05-24T21:28:37.204 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:3wWXbCytwQmdLGRJmDcZ/VHT9zQ= 2023-05-24T21:28:37.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:3wWXbCytwQmdLGRJmDcZ/VHT9zQ=', 'Content-Length': '0'} 2023-05-24T21:28:37.206 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cbe8bd7b8a4059d8-00646e8185-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.207 DEBUG:boto:path=/qskrps-49/obj1 2023-05-24T21:28:37.207 DEBUG:boto:auth_path=/qskrps-49/obj1 2023-05-24T21:28:37.208 DEBUG:boto:Method: HEAD 2023-05-24T21:28:37.208 DEBUG:boto:Path: /qskrps-49/obj1 2023-05-24T21:28:37.208 DEBUG:boto:Data: 2023-05-24T21:28:37.209 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.209 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.209 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.210 DEBUG:boto:Params: {} 2023-05-24T21:28:37.210 DEBUG:boto:Token: None 2023-05-24T21:28:37.210 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj1 2023-05-24T21:28:37.211 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:3wWXbCytwQmdLGRJmDcZ/VHT9zQ= 2023-05-24T21:28:37.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:3wWXbCytwQmdLGRJmDcZ/VHT9zQ=', 'Content-Length': '0'} 2023-05-24T21:28:37.213 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000035008b56997685d-00646e8185-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-24T21:28:37.214 DEBUG:boto:path=/qskrps-49/obj1 2023-05-24T21:28:37.214 DEBUG:boto:auth_path=/qskrps-49/obj1 2023-05-24T21:28:37.214 DEBUG:boto:Method: GET 2023-05-24T21:28:37.215 DEBUG:boto:Path: /qskrps-49/obj1 2023-05-24T21:28:37.215 DEBUG:boto:Data: 2023-05-24T21:28:37.215 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.216 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.216 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.216 DEBUG:boto:Params: {} 2023-05-24T21:28:37.217 DEBUG:boto:Token: None 2023-05-24T21:28:37.217 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj1 2023-05-24T21:28:37.218 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:tEkFaS/1LLCXdENkIuEDYkgnFaI= 2023-05-24T21:28:37.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:tEkFaS/1LLCXdENkIuEDYkgnFaI=', 'Content-Length': '0'} 2023-05-24T21:28:37.219 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d687d710865508cf-00646e8185-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.220 DEBUG:boto:path=/qskrps-49/obj1 2023-05-24T21:28:37.220 DEBUG:boto:auth_path=/qskrps-49/obj1 2023-05-24T21:28:37.221 DEBUG:boto:Method: GET 2023-05-24T21:28:37.221 DEBUG:boto:Path: /qskrps-49/obj1 2023-05-24T21:28:37.222 DEBUG:boto:Data: 2023-05-24T21:28:37.222 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.222 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.223 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.223 DEBUG:boto:Params: {} 2023-05-24T21:28:37.223 DEBUG:boto:Token: None 2023-05-24T21:28:37.224 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj1 2023-05-24T21:28:37.224 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:tEkFaS/1LLCXdENkIuEDYkgnFaI= 2023-05-24T21:28:37.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:tEkFaS/1LLCXdENkIuEDYkgnFaI=', 'Content-Length': '0'} 2023-05-24T21:28:37.226 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002665c6fdae3fe875-00646e8185-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.227 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:37.227 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:37.228 DEBUG:boto:path=/qskrps-49/?prefix=obj1 2023-05-24T21:28:37.228 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj1 2023-05-24T21:28:37.228 DEBUG:boto:Method: GET 2023-05-24T21:28:37.229 DEBUG:boto:Path: /qskrps-49/?prefix=obj1 2023-05-24T21:28:37.229 DEBUG:boto:Data: 2023-05-24T21:28:37.229 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.230 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.230 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.230 DEBUG:boto:Params: {} 2023-05-24T21:28:37.231 DEBUG:boto:Token: None 2023-05-24T21:28:37.231 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/ 2023-05-24T21:28:37.232 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ= 2023-05-24T21:28:37.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ=', 'Content-Length': '0'} 2023-05-24T21:28:37.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d6533f828db4a43-00646e8185-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.234 DEBUG:boto:b'qskrps-49obj11000falseobj12023-05-24T21:27:06.420Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:37.235 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:37.235 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:37.236 DEBUG:boto:path=/qskrps-49/?prefix=obj1 2023-05-24T21:28:37.236 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj1 2023-05-24T21:28:37.236 DEBUG:boto:Method: GET 2023-05-24T21:28:37.237 DEBUG:boto:Path: /qskrps-49/?prefix=obj1 2023-05-24T21:28:37.237 DEBUG:boto:Data: 2023-05-24T21:28:37.237 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.238 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.238 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.238 DEBUG:boto:Params: {} 2023-05-24T21:28:37.239 DEBUG:boto:Token: None 2023-05-24T21:28:37.239 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/ 2023-05-24T21:28:37.240 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ= 2023-05-24T21:28:37.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ=', 'Content-Length': '0'} 2023-05-24T21:28:37.242 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b76a3bae61425544-00646e8185-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.242 DEBUG:boto:b'qskrps-49obj11000falseobj12023-05-24T21:27:06.420Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:37.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-24T21:28:37.243 DEBUG:boto:path=/qskrps-49/obj2 2023-05-24T21:28:37.244 DEBUG:boto:auth_path=/qskrps-49/obj2 2023-05-24T21:28:37.244 DEBUG:boto:path=/qskrps-49/obj2?versionId=null 2023-05-24T21:28:37.244 DEBUG:boto:auth_path=/qskrps-49/obj2?versionId=null 2023-05-24T21:28:37.245 DEBUG:boto:Method: GET 2023-05-24T21:28:37.245 DEBUG:boto:Path: /qskrps-49/obj2?versionId=null 2023-05-24T21:28:37.246 DEBUG:boto:Data: 2023-05-24T21:28:37.246 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.246 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.247 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.247 DEBUG:boto:Params: {} 2023-05-24T21:28:37.247 DEBUG:boto:Token: None 2023-05-24T21:28:37.248 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj2?versionId=null 2023-05-24T21:28:37.248 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:fd0PJq8Ow9MZPAOlhJ5JcBqB2XU= 2023-05-24T21:28:37.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:fd0PJq8Ow9MZPAOlhJ5JcBqB2XU=', 'Content-Length': '0'} 2023-05-24T21:28:37.250 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000184478dd0da4616c-00646e8185-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.251 DEBUG:boto:path=/qskrps-49/obj2 2023-05-24T21:28:37.251 DEBUG:boto:auth_path=/qskrps-49/obj2 2023-05-24T21:28:37.252 DEBUG:boto:path=/qskrps-49/obj2?versionId=null 2023-05-24T21:28:37.252 DEBUG:boto:auth_path=/qskrps-49/obj2?versionId=null 2023-05-24T21:28:37.253 DEBUG:boto:Method: GET 2023-05-24T21:28:37.253 DEBUG:boto:Path: /qskrps-49/obj2?versionId=null 2023-05-24T21:28:37.253 DEBUG:boto:Data: 2023-05-24T21:28:37.254 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.254 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.254 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.255 DEBUG:boto:Params: {} 2023-05-24T21:28:37.255 DEBUG:boto:Token: None 2023-05-24T21:28:37.256 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj2?versionId=null 2023-05-24T21:28:37.256 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:fd0PJq8Ow9MZPAOlhJ5JcBqB2XU= 2023-05-24T21:28:37.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:fd0PJq8Ow9MZPAOlhJ5JcBqB2XU=', 'Content-Length': '0'} 2023-05-24T21:28:37.258 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f1f25db2f620e96-00646e8185-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.259 DEBUG:boto:path=/qskrps-49/obj2 2023-05-24T21:28:37.259 DEBUG:boto:auth_path=/qskrps-49/obj2 2023-05-24T21:28:37.259 DEBUG:boto:path=/qskrps-49/obj2?versionId=null 2023-05-24T21:28:37.260 DEBUG:boto:auth_path=/qskrps-49/obj2?versionId=null 2023-05-24T21:28:37.260 DEBUG:boto:Method: HEAD 2023-05-24T21:28:37.260 DEBUG:boto:Path: /qskrps-49/obj2?versionId=null 2023-05-24T21:28:37.261 DEBUG:boto:Data: 2023-05-24T21:28:37.261 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.261 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.262 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.262 DEBUG:boto:Params: {} 2023-05-24T21:28:37.262 DEBUG:boto:Token: None 2023-05-24T21:28:37.263 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj2?versionId=null 2023-05-24T21:28:37.263 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:pt6/tRGZtIjhVcp7Op9WT4HxSug= 2023-05-24T21:28:37.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:pt6/tRGZtIjhVcp7Op9WT4HxSug=', 'Content-Length': '0'} 2023-05-24T21:28:37.265 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1c34a6551893bf6-00646e8185-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.266 DEBUG:boto:path=/qskrps-49/obj2 2023-05-24T21:28:37.266 DEBUG:boto:auth_path=/qskrps-49/obj2 2023-05-24T21:28:37.266 DEBUG:boto:path=/qskrps-49/obj2?versionId=null 2023-05-24T21:28:37.267 DEBUG:boto:auth_path=/qskrps-49/obj2?versionId=null 2023-05-24T21:28:37.267 DEBUG:boto:Method: HEAD 2023-05-24T21:28:37.267 DEBUG:boto:Path: /qskrps-49/obj2?versionId=null 2023-05-24T21:28:37.268 DEBUG:boto:Data: 2023-05-24T21:28:37.268 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.268 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.269 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.269 DEBUG:boto:Params: {} 2023-05-24T21:28:37.269 DEBUG:boto:Token: None 2023-05-24T21:28:37.270 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj2?versionId=null 2023-05-24T21:28:37.270 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:pt6/tRGZtIjhVcp7Op9WT4HxSug= 2023-05-24T21:28:37.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:pt6/tRGZtIjhVcp7Op9WT4HxSug=', 'Content-Length': '0'} 2023-05-24T21:28:37.272 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cdc0d6992791b6c3-00646e8185-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-24T21:28:37.273 DEBUG:boto:path=/qskrps-49/obj2 2023-05-24T21:28:37.273 DEBUG:boto:auth_path=/qskrps-49/obj2 2023-05-24T21:28:37.273 DEBUG:boto:path=/qskrps-49/obj2?versionId=null 2023-05-24T21:28:37.274 DEBUG:boto:auth_path=/qskrps-49/obj2?versionId=null 2023-05-24T21:28:37.274 DEBUG:boto:Method: GET 2023-05-24T21:28:37.275 DEBUG:boto:Path: /qskrps-49/obj2?versionId=null 2023-05-24T21:28:37.275 DEBUG:boto:Data: 2023-05-24T21:28:37.275 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.276 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.276 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.276 DEBUG:boto:Params: {} 2023-05-24T21:28:37.277 DEBUG:boto:Token: None 2023-05-24T21:28:37.277 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj2?versionId=null 2023-05-24T21:28:37.278 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:fd0PJq8Ow9MZPAOlhJ5JcBqB2XU= 2023-05-24T21:28:37.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:fd0PJq8Ow9MZPAOlhJ5JcBqB2XU=', 'Content-Length': '0'} 2023-05-24T21:28:37.279 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000892c7f4eec049215-00646e8185-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.280 DEBUG:boto:path=/qskrps-49/obj2 2023-05-24T21:28:37.280 DEBUG:boto:auth_path=/qskrps-49/obj2 2023-05-24T21:28:37.281 DEBUG:boto:path=/qskrps-49/obj2?versionId=null 2023-05-24T21:28:37.281 DEBUG:boto:auth_path=/qskrps-49/obj2?versionId=null 2023-05-24T21:28:37.281 DEBUG:boto:Method: GET 2023-05-24T21:28:37.282 DEBUG:boto:Path: /qskrps-49/obj2?versionId=null 2023-05-24T21:28:37.282 DEBUG:boto:Data: 2023-05-24T21:28:37.283 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.283 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.283 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.284 DEBUG:boto:Params: {} 2023-05-24T21:28:37.284 DEBUG:boto:Token: None 2023-05-24T21:28:37.284 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj2?versionId=null 2023-05-24T21:28:37.285 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:fd0PJq8Ow9MZPAOlhJ5JcBqB2XU= 2023-05-24T21:28:37.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:fd0PJq8Ow9MZPAOlhJ5JcBqB2XU=', 'Content-Length': '0'} 2023-05-24T21:28:37.287 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a743c95289a0221b-00646e8185-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.287 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:37.288 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:37.288 DEBUG:boto:path=/qskrps-49/?prefix=obj2 2023-05-24T21:28:37.288 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj2 2023-05-24T21:28:37.289 DEBUG:boto:Method: GET 2023-05-24T21:28:37.289 DEBUG:boto:Path: /qskrps-49/?prefix=obj2 2023-05-24T21:28:37.289 DEBUG:boto:Data: 2023-05-24T21:28:37.290 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.290 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.290 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.291 DEBUG:boto:Params: {} 2023-05-24T21:28:37.291 DEBUG:boto:Token: None 2023-05-24T21:28:37.292 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/ 2023-05-24T21:28:37.292 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ= 2023-05-24T21:28:37.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ=', 'Content-Length': '0'} 2023-05-24T21:28:37.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000024954ecafa7d5db-00646e8185-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.295 DEBUG:boto:b'qskrps-49obj21000falseobj22023-05-24T21:27:06.438Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:37.295 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:37.296 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:37.296 DEBUG:boto:path=/qskrps-49/?prefix=obj2 2023-05-24T21:28:37.296 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj2 2023-05-24T21:28:37.297 DEBUG:boto:Method: GET 2023-05-24T21:28:37.297 DEBUG:boto:Path: /qskrps-49/?prefix=obj2 2023-05-24T21:28:37.298 DEBUG:boto:Data: 2023-05-24T21:28:37.298 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.298 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.299 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.299 DEBUG:boto:Params: {} 2023-05-24T21:28:37.299 DEBUG:boto:Token: None 2023-05-24T21:28:37.300 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/ 2023-05-24T21:28:37.300 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ= 2023-05-24T21:28:37.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ=', 'Content-Length': '0'} 2023-05-24T21:28:37.302 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae67ec98181eec05-00646e8185-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.303 DEBUG:boto:b'qskrps-49obj21000falseobj22023-05-24T21:27:06.438Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:37.303 DEBUG:boto:path=/qskrps-49/obj2 2023-05-24T21:28:37.304 DEBUG:boto:auth_path=/qskrps-49/obj2 2023-05-24T21:28:37.304 DEBUG:boto:Method: HEAD 2023-05-24T21:28:37.304 DEBUG:boto:Path: /qskrps-49/obj2 2023-05-24T21:28:37.305 DEBUG:boto:Data: 2023-05-24T21:28:37.305 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.305 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.306 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.306 DEBUG:boto:Params: {} 2023-05-24T21:28:37.306 DEBUG:boto:Token: None 2023-05-24T21:28:37.307 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj2 2023-05-24T21:28:37.307 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:JCt9KdNxj19/yeGq4ALaHTBqwKY= 2023-05-24T21:28:37.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:JCt9KdNxj19/yeGq4ALaHTBqwKY=', 'Content-Length': '0'} 2023-05-24T21:28:37.309 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f72de8bff163572d-00646e8185-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.310 DEBUG:boto:path=/qskrps-49/obj2 2023-05-24T21:28:37.310 DEBUG:boto:auth_path=/qskrps-49/obj2 2023-05-24T21:28:37.310 DEBUG:boto:Method: HEAD 2023-05-24T21:28:37.311 DEBUG:boto:Path: /qskrps-49/obj2 2023-05-24T21:28:37.311 DEBUG:boto:Data: 2023-05-24T21:28:37.311 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.312 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.312 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.312 DEBUG:boto:Params: {} 2023-05-24T21:28:37.313 DEBUG:boto:Token: None 2023-05-24T21:28:37.313 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj2 2023-05-24T21:28:37.314 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:JCt9KdNxj19/yeGq4ALaHTBqwKY= 2023-05-24T21:28:37.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:JCt9KdNxj19/yeGq4ALaHTBqwKY=', 'Content-Length': '0'} 2023-05-24T21:28:37.316 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000088d0cda2e59831ff-00646e8185-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-24T21:28:37.317 DEBUG:boto:path=/qskrps-49/obj2 2023-05-24T21:28:37.317 DEBUG:boto:auth_path=/qskrps-49/obj2 2023-05-24T21:28:37.317 DEBUG:boto:Method: GET 2023-05-24T21:28:37.318 DEBUG:boto:Path: /qskrps-49/obj2 2023-05-24T21:28:37.318 DEBUG:boto:Data: 2023-05-24T21:28:37.318 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.319 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.319 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.320 DEBUG:boto:Params: {} 2023-05-24T21:28:37.320 DEBUG:boto:Token: None 2023-05-24T21:28:37.320 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj2 2023-05-24T21:28:37.321 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:62G5nLjoU/JwxQM5DLLYDsrYDp4= 2023-05-24T21:28:37.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:62G5nLjoU/JwxQM5DLLYDsrYDp4=', 'Content-Length': '0'} 2023-05-24T21:28:37.322 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000246f539d7ff86cb2-00646e8185-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.323 DEBUG:boto:path=/qskrps-49/obj2 2023-05-24T21:28:37.323 DEBUG:boto:auth_path=/qskrps-49/obj2 2023-05-24T21:28:37.324 DEBUG:boto:Method: GET 2023-05-24T21:28:37.324 DEBUG:boto:Path: /qskrps-49/obj2 2023-05-24T21:28:37.325 DEBUG:boto:Data: 2023-05-24T21:28:37.325 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.325 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.326 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.326 DEBUG:boto:Params: {} 2023-05-24T21:28:37.326 DEBUG:boto:Token: None 2023-05-24T21:28:37.327 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj2 2023-05-24T21:28:37.327 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:62G5nLjoU/JwxQM5DLLYDsrYDp4= 2023-05-24T21:28:37.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:62G5nLjoU/JwxQM5DLLYDsrYDp4=', 'Content-Length': '0'} 2023-05-24T21:28:37.329 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000127fd734c415f13-00646e8185-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.330 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:37.330 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:37.331 DEBUG:boto:path=/qskrps-49/?prefix=obj2 2023-05-24T21:28:37.331 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj2 2023-05-24T21:28:37.331 DEBUG:boto:Method: GET 2023-05-24T21:28:37.332 DEBUG:boto:Path: /qskrps-49/?prefix=obj2 2023-05-24T21:28:37.332 DEBUG:boto:Data: 2023-05-24T21:28:37.332 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.333 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.333 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.333 DEBUG:boto:Params: {} 2023-05-24T21:28:37.334 DEBUG:boto:Token: None 2023-05-24T21:28:37.334 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/ 2023-05-24T21:28:37.335 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ= 2023-05-24T21:28:37.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ=', 'Content-Length': '0'} 2023-05-24T21:28:37.337 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d6fd6a99d2e1f85-00646e8185-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.337 DEBUG:boto:b'qskrps-49obj21000falseobj22023-05-24T21:27:06.438Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:37.338 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:37.338 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:37.339 DEBUG:boto:path=/qskrps-49/?prefix=obj2 2023-05-24T21:28:37.339 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj2 2023-05-24T21:28:37.340 DEBUG:boto:Method: GET 2023-05-24T21:28:37.340 DEBUG:boto:Path: /qskrps-49/?prefix=obj2 2023-05-24T21:28:37.340 DEBUG:boto:Data: 2023-05-24T21:28:37.341 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.341 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.341 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.342 DEBUG:boto:Params: {} 2023-05-24T21:28:37.342 DEBUG:boto:Token: None 2023-05-24T21:28:37.342 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/ 2023-05-24T21:28:37.343 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ= 2023-05-24T21:28:37.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ=', 'Content-Length': '0'} 2023-05-24T21:28:37.345 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048f61596042f50cb-00646e8185-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.346 DEBUG:boto:b'qskrps-49obj21000falseobj22023-05-24T21:27:06.438Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:37.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-24T21:28:37.347 DEBUG:boto:path=/qskrps-49/obj3 2023-05-24T21:28:37.347 DEBUG:boto:auth_path=/qskrps-49/obj3 2023-05-24T21:28:37.347 DEBUG:boto:path=/qskrps-49/obj3?versionId=null 2023-05-24T21:28:37.348 DEBUG:boto:auth_path=/qskrps-49/obj3?versionId=null 2023-05-24T21:28:37.348 DEBUG:boto:Method: GET 2023-05-24T21:28:37.348 DEBUG:boto:Path: /qskrps-49/obj3?versionId=null 2023-05-24T21:28:37.349 DEBUG:boto:Data: 2023-05-24T21:28:37.349 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.350 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.350 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.350 DEBUG:boto:Params: {} 2023-05-24T21:28:37.351 DEBUG:boto:Token: None 2023-05-24T21:28:37.351 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj3?versionId=null 2023-05-24T21:28:37.351 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:yh07+1GUhVzlQgkm0Xnrw4PLMZo= 2023-05-24T21:28:37.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:yh07+1GUhVzlQgkm0Xnrw4PLMZo=', 'Content-Length': '0'} 2023-05-24T21:28:37.353 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b005639aa3899be6-00646e8185-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.396 DEBUG:boto:path=/qskrps-49/obj3 2023-05-24T21:28:37.397 DEBUG:boto:auth_path=/qskrps-49/obj3 2023-05-24T21:28:37.397 DEBUG:boto:path=/qskrps-49/obj3?versionId=null 2023-05-24T21:28:37.397 DEBUG:boto:auth_path=/qskrps-49/obj3?versionId=null 2023-05-24T21:28:37.398 DEBUG:boto:Method: GET 2023-05-24T21:28:37.398 DEBUG:boto:Path: /qskrps-49/obj3?versionId=null 2023-05-24T21:28:37.398 DEBUG:boto:Data: 2023-05-24T21:28:37.399 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.399 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.399 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.400 DEBUG:boto:Params: {} 2023-05-24T21:28:37.400 DEBUG:boto:Token: None 2023-05-24T21:28:37.401 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj3?versionId=null 2023-05-24T21:28:37.401 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:yh07+1GUhVzlQgkm0Xnrw4PLMZo= 2023-05-24T21:28:37.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:yh07+1GUhVzlQgkm0Xnrw4PLMZo=', 'Content-Length': '0'} 2023-05-24T21:28:37.403 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000646526ea4828acdc-00646e8185-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.403 DEBUG:boto:path=/qskrps-49/obj3 2023-05-24T21:28:37.404 DEBUG:boto:auth_path=/qskrps-49/obj3 2023-05-24T21:28:37.404 DEBUG:boto:path=/qskrps-49/obj3?versionId=null 2023-05-24T21:28:37.405 DEBUG:boto:auth_path=/qskrps-49/obj3?versionId=null 2023-05-24T21:28:37.405 DEBUG:boto:Method: HEAD 2023-05-24T21:28:37.405 DEBUG:boto:Path: /qskrps-49/obj3?versionId=null 2023-05-24T21:28:37.406 DEBUG:boto:Data: 2023-05-24T21:28:37.406 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.406 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.407 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.407 DEBUG:boto:Params: {} 2023-05-24T21:28:37.407 DEBUG:boto:Token: None 2023-05-24T21:28:37.408 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj3?versionId=null 2023-05-24T21:28:37.408 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:fwYyRsCGZ7UfFGd9+nPJNxkdZBQ= 2023-05-24T21:28:37.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:fwYyRsCGZ7UfFGd9+nPJNxkdZBQ=', 'Content-Length': '0'} 2023-05-24T21:28:37.410 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009815233dc7275176-00646e8185-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.411 DEBUG:boto:path=/qskrps-49/obj3 2023-05-24T21:28:37.411 DEBUG:boto:auth_path=/qskrps-49/obj3 2023-05-24T21:28:37.411 DEBUG:boto:path=/qskrps-49/obj3?versionId=null 2023-05-24T21:28:37.412 DEBUG:boto:auth_path=/qskrps-49/obj3?versionId=null 2023-05-24T21:28:37.412 DEBUG:boto:Method: HEAD 2023-05-24T21:28:37.412 DEBUG:boto:Path: /qskrps-49/obj3?versionId=null 2023-05-24T21:28:37.413 DEBUG:boto:Data: 2023-05-24T21:28:37.413 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.413 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.414 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.414 DEBUG:boto:Params: {} 2023-05-24T21:28:37.414 DEBUG:boto:Token: None 2023-05-24T21:28:37.415 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj3?versionId=null 2023-05-24T21:28:37.415 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:fwYyRsCGZ7UfFGd9+nPJNxkdZBQ= 2023-05-24T21:28:37.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:fwYyRsCGZ7UfFGd9+nPJNxkdZBQ=', 'Content-Length': '0'} 2023-05-24T21:28:37.417 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb0b562f5e640547-00646e8185-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-24T21:28:37.418 DEBUG:boto:path=/qskrps-49/obj3 2023-05-24T21:28:37.418 DEBUG:boto:auth_path=/qskrps-49/obj3 2023-05-24T21:28:37.419 DEBUG:boto:path=/qskrps-49/obj3?versionId=null 2023-05-24T21:28:37.419 DEBUG:boto:auth_path=/qskrps-49/obj3?versionId=null 2023-05-24T21:28:37.420 DEBUG:boto:Method: GET 2023-05-24T21:28:37.420 DEBUG:boto:Path: /qskrps-49/obj3?versionId=null 2023-05-24T21:28:37.420 DEBUG:boto:Data: 2023-05-24T21:28:37.421 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.421 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.421 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.422 DEBUG:boto:Params: {} 2023-05-24T21:28:37.422 DEBUG:boto:Token: None 2023-05-24T21:28:37.423 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj3?versionId=null 2023-05-24T21:28:37.423 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:yh07+1GUhVzlQgkm0Xnrw4PLMZo= 2023-05-24T21:28:37.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:yh07+1GUhVzlQgkm0Xnrw4PLMZo=', 'Content-Length': '0'} 2023-05-24T21:28:37.425 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a665736c03665916-00646e8185-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.425 DEBUG:boto:path=/qskrps-49/obj3 2023-05-24T21:28:37.426 DEBUG:boto:auth_path=/qskrps-49/obj3 2023-05-24T21:28:37.426 DEBUG:boto:path=/qskrps-49/obj3?versionId=null 2023-05-24T21:28:37.426 DEBUG:boto:auth_path=/qskrps-49/obj3?versionId=null 2023-05-24T21:28:37.427 DEBUG:boto:Method: GET 2023-05-24T21:28:37.427 DEBUG:boto:Path: /qskrps-49/obj3?versionId=null 2023-05-24T21:28:37.428 DEBUG:boto:Data: 2023-05-24T21:28:37.428 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.428 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.429 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.429 DEBUG:boto:Params: {} 2023-05-24T21:28:37.429 DEBUG:boto:Token: None 2023-05-24T21:28:37.430 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj3?versionId=null 2023-05-24T21:28:37.430 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:yh07+1GUhVzlQgkm0Xnrw4PLMZo= 2023-05-24T21:28:37.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:yh07+1GUhVzlQgkm0Xnrw4PLMZo=', 'Content-Length': '0'} 2023-05-24T21:28:37.432 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b1bfe0329133699c-00646e8185-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.433 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:37.433 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:37.434 DEBUG:boto:path=/qskrps-49/?prefix=obj3 2023-05-24T21:28:37.434 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj3 2023-05-24T21:28:37.434 DEBUG:boto:Method: GET 2023-05-24T21:28:37.435 DEBUG:boto:Path: /qskrps-49/?prefix=obj3 2023-05-24T21:28:37.435 DEBUG:boto:Data: 2023-05-24T21:28:37.435 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.436 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.436 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.436 DEBUG:boto:Params: {} 2023-05-24T21:28:37.437 DEBUG:boto:Token: None 2023-05-24T21:28:37.437 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/ 2023-05-24T21:28:37.438 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ= 2023-05-24T21:28:37.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ=', 'Content-Length': '0'} 2023-05-24T21:28:37.440 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a831df4da3b1ee9d-00646e8185-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.440 DEBUG:boto:b'qskrps-49obj31000falseobj32023-05-24T21:27:06.495Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:37.441 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:37.441 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:37.442 DEBUG:boto:path=/qskrps-49/?prefix=obj3 2023-05-24T21:28:37.442 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj3 2023-05-24T21:28:37.442 DEBUG:boto:Method: GET 2023-05-24T21:28:37.443 DEBUG:boto:Path: /qskrps-49/?prefix=obj3 2023-05-24T21:28:37.443 DEBUG:boto:Data: 2023-05-24T21:28:37.444 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.444 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.444 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.445 DEBUG:boto:Params: {} 2023-05-24T21:28:37.445 DEBUG:boto:Token: None 2023-05-24T21:28:37.446 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/ 2023-05-24T21:28:37.446 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ= 2023-05-24T21:28:37.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ=', 'Content-Length': '0'} 2023-05-24T21:28:37.448 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4a7118e364acbc7-00646e8185-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.492 DEBUG:boto:b'qskrps-49obj31000falseobj32023-05-24T21:27:06.495Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:37.493 DEBUG:boto:path=/qskrps-49/obj3 2023-05-24T21:28:37.493 DEBUG:boto:auth_path=/qskrps-49/obj3 2023-05-24T21:28:37.493 DEBUG:boto:Method: HEAD 2023-05-24T21:28:37.494 DEBUG:boto:Path: /qskrps-49/obj3 2023-05-24T21:28:37.494 DEBUG:boto:Data: 2023-05-24T21:28:37.494 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.495 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.495 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.495 DEBUG:boto:Params: {} 2023-05-24T21:28:37.496 DEBUG:boto:Token: None 2023-05-24T21:28:37.496 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj3 2023-05-24T21:28:37.497 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:raF3yaATdEJjS8+66apI5NSD77k= 2023-05-24T21:28:37.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:raF3yaATdEJjS8+66apI5NSD77k=', 'Content-Length': '0'} 2023-05-24T21:28:37.498 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008682da2cd77a2be0-00646e8185-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.499 DEBUG:boto:path=/qskrps-49/obj3 2023-05-24T21:28:37.499 DEBUG:boto:auth_path=/qskrps-49/obj3 2023-05-24T21:28:37.500 DEBUG:boto:Method: HEAD 2023-05-24T21:28:37.500 DEBUG:boto:Path: /qskrps-49/obj3 2023-05-24T21:28:37.500 DEBUG:boto:Data: 2023-05-24T21:28:37.501 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.501 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.501 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.502 DEBUG:boto:Params: {} 2023-05-24T21:28:37.502 DEBUG:boto:Token: None 2023-05-24T21:28:37.502 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj3 2023-05-24T21:28:37.503 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:raF3yaATdEJjS8+66apI5NSD77k= 2023-05-24T21:28:37.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:raF3yaATdEJjS8+66apI5NSD77k=', 'Content-Length': '0'} 2023-05-24T21:28:37.505 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000071c90751c2dcd0a9-00646e8185-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-24T21:28:37.505 DEBUG:boto:path=/qskrps-49/obj3 2023-05-24T21:28:37.506 DEBUG:boto:auth_path=/qskrps-49/obj3 2023-05-24T21:28:37.506 DEBUG:boto:Method: GET 2023-05-24T21:28:37.507 DEBUG:boto:Path: /qskrps-49/obj3 2023-05-24T21:28:37.507 DEBUG:boto:Data: 2023-05-24T21:28:37.507 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.508 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.508 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.508 DEBUG:boto:Params: {} 2023-05-24T21:28:37.509 DEBUG:boto:Token: None 2023-05-24T21:28:37.509 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj3 2023-05-24T21:28:37.510 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:2VUH3+y+1n/y3WCHMQrT6jYnRZs= 2023-05-24T21:28:37.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:2VUH3+y+1n/y3WCHMQrT6jYnRZs=', 'Content-Length': '0'} 2023-05-24T21:28:37.511 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aad33984f7806bdb-00646e8185-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.512 DEBUG:boto:path=/qskrps-49/obj3 2023-05-24T21:28:37.513 DEBUG:boto:auth_path=/qskrps-49/obj3 2023-05-24T21:28:37.513 DEBUG:boto:Method: GET 2023-05-24T21:28:37.513 DEBUG:boto:Path: /qskrps-49/obj3 2023-05-24T21:28:37.514 DEBUG:boto:Data: 2023-05-24T21:28:37.514 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.514 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.515 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.515 DEBUG:boto:Params: {} 2023-05-24T21:28:37.515 DEBUG:boto:Token: None 2023-05-24T21:28:37.516 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj3 2023-05-24T21:28:37.516 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:2VUH3+y+1n/y3WCHMQrT6jYnRZs= 2023-05-24T21:28:37.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:2VUH3+y+1n/y3WCHMQrT6jYnRZs=', 'Content-Length': '0'} 2023-05-24T21:28:37.518 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000052f09fb41b48a1b3-00646e8185-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.519 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:37.519 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:37.520 DEBUG:boto:path=/qskrps-49/?prefix=obj3 2023-05-24T21:28:37.520 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj3 2023-05-24T21:28:37.520 DEBUG:boto:Method: GET 2023-05-24T21:28:37.521 DEBUG:boto:Path: /qskrps-49/?prefix=obj3 2023-05-24T21:28:37.521 DEBUG:boto:Data: 2023-05-24T21:28:37.521 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.522 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.522 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.522 DEBUG:boto:Params: {} 2023-05-24T21:28:37.523 DEBUG:boto:Token: None 2023-05-24T21:28:37.523 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/ 2023-05-24T21:28:37.524 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ= 2023-05-24T21:28:37.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ=', 'Content-Length': '0'} 2023-05-24T21:28:37.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d0828d3b141bfb7-00646e8185-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.527 DEBUG:boto:b'qskrps-49obj31000falseobj32023-05-24T21:27:06.495Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:37.527 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:37.528 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:37.528 DEBUG:boto:path=/qskrps-49/?prefix=obj3 2023-05-24T21:28:37.528 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj3 2023-05-24T21:28:37.529 DEBUG:boto:Method: GET 2023-05-24T21:28:37.529 DEBUG:boto:Path: /qskrps-49/?prefix=obj3 2023-05-24T21:28:37.529 DEBUG:boto:Data: 2023-05-24T21:28:37.530 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.530 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.530 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.531 DEBUG:boto:Params: {} 2023-05-24T21:28:37.531 DEBUG:boto:Token: None 2023-05-24T21:28:37.532 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/ 2023-05-24T21:28:37.532 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ= 2023-05-24T21:28:37.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ=', 'Content-Length': '0'} 2023-05-24T21:28:37.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6942447156599fd-00646e8185-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.535 DEBUG:boto:b'qskrps-49obj31000falseobj32023-05-24T21:27:06.495Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:37.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-24T21:28:37.536 DEBUG:boto:path=/qskrps-49/obj4 2023-05-24T21:28:37.536 DEBUG:boto:auth_path=/qskrps-49/obj4 2023-05-24T21:28:37.536 DEBUG:boto:path=/qskrps-49/obj4?versionId=null 2023-05-24T21:28:37.537 DEBUG:boto:auth_path=/qskrps-49/obj4?versionId=null 2023-05-24T21:28:37.537 DEBUG:boto:Method: GET 2023-05-24T21:28:37.538 DEBUG:boto:Path: /qskrps-49/obj4?versionId=null 2023-05-24T21:28:37.538 DEBUG:boto:Data: 2023-05-24T21:28:37.538 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.539 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.539 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.539 DEBUG:boto:Params: {} 2023-05-24T21:28:37.540 DEBUG:boto:Token: None 2023-05-24T21:28:37.540 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj4?versionId=null 2023-05-24T21:28:37.541 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:sSqZ/LYGvtiX+yaI0shYB5PZ/Sw= 2023-05-24T21:28:37.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:sSqZ/LYGvtiX+yaI0shYB5PZ/Sw=', 'Content-Length': '0'} 2023-05-24T21:28:37.542 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d55ec0c6aa0cb83a-00646e8185-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.543 DEBUG:boto:path=/qskrps-49/obj4 2023-05-24T21:28:37.543 DEBUG:boto:auth_path=/qskrps-49/obj4 2023-05-24T21:28:37.544 DEBUG:boto:path=/qskrps-49/obj4?versionId=null 2023-05-24T21:28:37.544 DEBUG:boto:auth_path=/qskrps-49/obj4?versionId=null 2023-05-24T21:28:37.545 DEBUG:boto:Method: GET 2023-05-24T21:28:37.545 DEBUG:boto:Path: /qskrps-49/obj4?versionId=null 2023-05-24T21:28:37.545 DEBUG:boto:Data: 2023-05-24T21:28:37.546 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.546 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.546 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.547 DEBUG:boto:Params: {} 2023-05-24T21:28:37.547 DEBUG:boto:Token: None 2023-05-24T21:28:37.547 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj4?versionId=null 2023-05-24T21:28:37.548 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:sSqZ/LYGvtiX+yaI0shYB5PZ/Sw= 2023-05-24T21:28:37.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:sSqZ/LYGvtiX+yaI0shYB5PZ/Sw=', 'Content-Length': '0'} 2023-05-24T21:28:37.550 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006f6d97934a4005ad-00646e8185-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.551 DEBUG:boto:path=/qskrps-49/obj4 2023-05-24T21:28:37.551 DEBUG:boto:auth_path=/qskrps-49/obj4 2023-05-24T21:28:37.551 DEBUG:boto:path=/qskrps-49/obj4?versionId=null 2023-05-24T21:28:37.552 DEBUG:boto:auth_path=/qskrps-49/obj4?versionId=null 2023-05-24T21:28:37.552 DEBUG:boto:Method: HEAD 2023-05-24T21:28:37.552 DEBUG:boto:Path: /qskrps-49/obj4?versionId=null 2023-05-24T21:28:37.553 DEBUG:boto:Data: 2023-05-24T21:28:37.553 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.553 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.554 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.554 DEBUG:boto:Params: {} 2023-05-24T21:28:37.554 DEBUG:boto:Token: None 2023-05-24T21:28:37.555 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj4?versionId=null 2023-05-24T21:28:37.555 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:TqaCIsolS/V3+nR8c/aAP629Jic= 2023-05-24T21:28:37.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:TqaCIsolS/V3+nR8c/aAP629Jic=', 'Content-Length': '0'} 2023-05-24T21:28:37.557 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000e4c96bd5346db98-00646e8185-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.558 DEBUG:boto:path=/qskrps-49/obj4 2023-05-24T21:28:37.558 DEBUG:boto:auth_path=/qskrps-49/obj4 2023-05-24T21:28:37.558 DEBUG:boto:path=/qskrps-49/obj4?versionId=null 2023-05-24T21:28:37.559 DEBUG:boto:auth_path=/qskrps-49/obj4?versionId=null 2023-05-24T21:28:37.559 DEBUG:boto:Method: HEAD 2023-05-24T21:28:37.559 DEBUG:boto:Path: /qskrps-49/obj4?versionId=null 2023-05-24T21:28:37.560 DEBUG:boto:Data: 2023-05-24T21:28:37.560 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.560 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.561 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.561 DEBUG:boto:Params: {} 2023-05-24T21:28:37.561 DEBUG:boto:Token: None 2023-05-24T21:28:37.562 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj4?versionId=null 2023-05-24T21:28:37.562 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:TqaCIsolS/V3+nR8c/aAP629Jic= 2023-05-24T21:28:37.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:TqaCIsolS/V3+nR8c/aAP629Jic=', 'Content-Length': '0'} 2023-05-24T21:28:37.564 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005f50276c915994df-00646e8185-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-24T21:28:37.565 DEBUG:boto:path=/qskrps-49/obj4 2023-05-24T21:28:37.566 DEBUG:boto:auth_path=/qskrps-49/obj4 2023-05-24T21:28:37.566 DEBUG:boto:path=/qskrps-49/obj4?versionId=null 2023-05-24T21:28:37.566 DEBUG:boto:auth_path=/qskrps-49/obj4?versionId=null 2023-05-24T21:28:37.567 DEBUG:boto:Method: GET 2023-05-24T21:28:37.567 DEBUG:boto:Path: /qskrps-49/obj4?versionId=null 2023-05-24T21:28:37.567 DEBUG:boto:Data: 2023-05-24T21:28:37.568 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.568 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.568 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.569 DEBUG:boto:Params: {} 2023-05-24T21:28:37.569 DEBUG:boto:Token: None 2023-05-24T21:28:37.570 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj4?versionId=null 2023-05-24T21:28:37.570 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:sSqZ/LYGvtiX+yaI0shYB5PZ/Sw= 2023-05-24T21:28:37.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:sSqZ/LYGvtiX+yaI0shYB5PZ/Sw=', 'Content-Length': '0'} 2023-05-24T21:28:37.572 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e9c891f4b6ae101c-00646e8185-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.572 DEBUG:boto:path=/qskrps-49/obj4 2023-05-24T21:28:37.573 DEBUG:boto:auth_path=/qskrps-49/obj4 2023-05-24T21:28:37.573 DEBUG:boto:path=/qskrps-49/obj4?versionId=null 2023-05-24T21:28:37.574 DEBUG:boto:auth_path=/qskrps-49/obj4?versionId=null 2023-05-24T21:28:37.574 DEBUG:boto:Method: GET 2023-05-24T21:28:37.574 DEBUG:boto:Path: /qskrps-49/obj4?versionId=null 2023-05-24T21:28:37.575 DEBUG:boto:Data: 2023-05-24T21:28:37.575 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.575 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.576 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.576 DEBUG:boto:Params: {} 2023-05-24T21:28:37.576 DEBUG:boto:Token: None 2023-05-24T21:28:37.577 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj4?versionId=null 2023-05-24T21:28:37.577 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:sSqZ/LYGvtiX+yaI0shYB5PZ/Sw= 2023-05-24T21:28:37.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:sSqZ/LYGvtiX+yaI0shYB5PZ/Sw=', 'Content-Length': '0'} 2023-05-24T21:28:37.579 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d1f61ca3a1da607-00646e8185-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.620 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:37.621 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:37.621 DEBUG:boto:path=/qskrps-49/?prefix=obj4 2023-05-24T21:28:37.621 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj4 2023-05-24T21:28:37.622 DEBUG:boto:Method: GET 2023-05-24T21:28:37.622 DEBUG:boto:Path: /qskrps-49/?prefix=obj4 2023-05-24T21:28:37.622 DEBUG:boto:Data: 2023-05-24T21:28:37.623 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.623 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.623 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.624 DEBUG:boto:Params: {} 2023-05-24T21:28:37.624 DEBUG:boto:Token: None 2023-05-24T21:28:37.625 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/ 2023-05-24T21:28:37.625 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ= 2023-05-24T21:28:37.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ=', 'Content-Length': '0'} 2023-05-24T21:28:37.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003dbf1efda61bb20e-00646e8185-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.628 DEBUG:boto:b'qskrps-49obj41000falseobj42023-05-24T21:27:06.514Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:37.628 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:37.629 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:37.629 DEBUG:boto:path=/qskrps-49/?prefix=obj4 2023-05-24T21:28:37.629 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj4 2023-05-24T21:28:37.630 DEBUG:boto:Method: GET 2023-05-24T21:28:37.630 DEBUG:boto:Path: /qskrps-49/?prefix=obj4 2023-05-24T21:28:37.630 DEBUG:boto:Data: 2023-05-24T21:28:37.631 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.631 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.632 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.632 DEBUG:boto:Params: {} 2023-05-24T21:28:37.632 DEBUG:boto:Token: None 2023-05-24T21:28:37.633 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/ 2023-05-24T21:28:37.633 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ= 2023-05-24T21:28:37.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ=', 'Content-Length': '0'} 2023-05-24T21:28:37.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008da9e8786515392f-00646e8185-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.636 DEBUG:boto:b'qskrps-49obj41000falseobj42023-05-24T21:27:06.514Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:37.637 DEBUG:boto:path=/qskrps-49/obj4 2023-05-24T21:28:37.637 DEBUG:boto:auth_path=/qskrps-49/obj4 2023-05-24T21:28:37.637 DEBUG:boto:Method: HEAD 2023-05-24T21:28:37.638 DEBUG:boto:Path: /qskrps-49/obj4 2023-05-24T21:28:37.638 DEBUG:boto:Data: 2023-05-24T21:28:37.638 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.639 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.639 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.639 DEBUG:boto:Params: {} 2023-05-24T21:28:37.640 DEBUG:boto:Token: None 2023-05-24T21:28:37.640 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj4 2023-05-24T21:28:37.641 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:XtqePR/2tf7+HjrR1J8x0knInKY= 2023-05-24T21:28:37.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:XtqePR/2tf7+HjrR1J8x0knInKY=', 'Content-Length': '0'} 2023-05-24T21:28:37.642 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005ddb0e67b9b2c8ee-00646e8185-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.643 DEBUG:boto:path=/qskrps-49/obj4 2023-05-24T21:28:37.643 DEBUG:boto:auth_path=/qskrps-49/obj4 2023-05-24T21:28:37.644 DEBUG:boto:Method: HEAD 2023-05-24T21:28:37.644 DEBUG:boto:Path: /qskrps-49/obj4 2023-05-24T21:28:37.644 DEBUG:boto:Data: 2023-05-24T21:28:37.645 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.645 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.645 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.646 DEBUG:boto:Params: {} 2023-05-24T21:28:37.646 DEBUG:boto:Token: None 2023-05-24T21:28:37.646 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj4 2023-05-24T21:28:37.647 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:XtqePR/2tf7+HjrR1J8x0knInKY= 2023-05-24T21:28:37.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:XtqePR/2tf7+HjrR1J8x0knInKY=', 'Content-Length': '0'} 2023-05-24T21:28:37.649 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009a6e7c1f2d76698e-00646e8185-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-24T21:28:37.650 DEBUG:boto:path=/qskrps-49/obj4 2023-05-24T21:28:37.650 DEBUG:boto:auth_path=/qskrps-49/obj4 2023-05-24T21:28:37.650 DEBUG:boto:Method: GET 2023-05-24T21:28:37.651 DEBUG:boto:Path: /qskrps-49/obj4 2023-05-24T21:28:37.651 DEBUG:boto:Data: 2023-05-24T21:28:37.651 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.652 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.652 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.652 DEBUG:boto:Params: {} 2023-05-24T21:28:37.653 DEBUG:boto:Token: None 2023-05-24T21:28:37.653 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj4 2023-05-24T21:28:37.654 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:zcyGtVXeLtBtVUcP8YxkVtc4clE= 2023-05-24T21:28:37.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:zcyGtVXeLtBtVUcP8YxkVtc4clE=', 'Content-Length': '0'} 2023-05-24T21:28:37.656 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000773540b83584e15a-00646e8185-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.656 DEBUG:boto:path=/qskrps-49/obj4 2023-05-24T21:28:37.657 DEBUG:boto:auth_path=/qskrps-49/obj4 2023-05-24T21:28:37.657 DEBUG:boto:Method: GET 2023-05-24T21:28:37.657 DEBUG:boto:Path: /qskrps-49/obj4 2023-05-24T21:28:37.658 DEBUG:boto:Data: 2023-05-24T21:28:37.658 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.658 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.659 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.659 DEBUG:boto:Params: {} 2023-05-24T21:28:37.660 DEBUG:boto:Token: None 2023-05-24T21:28:37.660 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj4 2023-05-24T21:28:37.660 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:zcyGtVXeLtBtVUcP8YxkVtc4clE= 2023-05-24T21:28:37.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:zcyGtVXeLtBtVUcP8YxkVtc4clE=', 'Content-Length': '0'} 2023-05-24T21:28:37.662 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:27:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f92c170b968d5d8b-00646e8185-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.663 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:37.663 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:37.664 DEBUG:boto:path=/qskrps-49/?prefix=obj4 2023-05-24T21:28:37.664 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj4 2023-05-24T21:28:37.664 DEBUG:boto:Method: GET 2023-05-24T21:28:37.665 DEBUG:boto:Path: /qskrps-49/?prefix=obj4 2023-05-24T21:28:37.665 DEBUG:boto:Data: 2023-05-24T21:28:37.665 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.666 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.666 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.667 DEBUG:boto:Params: {} 2023-05-24T21:28:37.667 DEBUG:boto:Token: None 2023-05-24T21:28:37.667 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/ 2023-05-24T21:28:37.668 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ= 2023-05-24T21:28:37.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ=', 'Content-Length': '0'} 2023-05-24T21:28:37.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007f165ba52d5df8f-00646e8185-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.671 DEBUG:boto:b'qskrps-49obj41000falseobj42023-05-24T21:27:06.514Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:37.671 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:37.672 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:37.672 DEBUG:boto:path=/qskrps-49/?prefix=obj4 2023-05-24T21:28:37.672 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj4 2023-05-24T21:28:37.673 DEBUG:boto:Method: GET 2023-05-24T21:28:37.673 DEBUG:boto:Path: /qskrps-49/?prefix=obj4 2023-05-24T21:28:37.673 DEBUG:boto:Data: 2023-05-24T21:28:37.674 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.674 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.674 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.675 DEBUG:boto:Params: {} 2023-05-24T21:28:37.675 DEBUG:boto:Token: None 2023-05-24T21:28:37.676 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/ 2023-05-24T21:28:37.676 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ= 2023-05-24T21:28:37.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ=', 'Content-Length': '0'} 2023-05-24T21:28:37.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000480bd5b261c76d98-00646e8185-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.679 DEBUG:boto:b'qskrps-49obj41000falseobj42023-05-24T21:27:06.514Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:37.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-05-24T21:28:37.680 DEBUG:boto:path=/qskrps-49/obj5 2023-05-24T21:28:37.680 DEBUG:boto:auth_path=/qskrps-49/obj5 2023-05-24T21:28:37.681 DEBUG:boto:path=/qskrps-49/obj5?versionId=null 2023-05-24T21:28:37.681 DEBUG:boto:auth_path=/qskrps-49/obj5?versionId=null 2023-05-24T21:28:37.681 DEBUG:boto:Method: GET 2023-05-24T21:28:37.682 DEBUG:boto:Path: /qskrps-49/obj5?versionId=null 2023-05-24T21:28:37.682 DEBUG:boto:Data: 2023-05-24T21:28:37.682 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.683 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.683 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.683 DEBUG:boto:Params: {} 2023-05-24T21:28:37.684 DEBUG:boto:Token: None 2023-05-24T21:28:37.684 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj5?versionId=null 2023-05-24T21:28:37.685 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:xXfNsAxdpB2fa+2LzwJLjL2QuZ4= 2023-05-24T21:28:37.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:xXfNsAxdpB2fa+2LzwJLjL2QuZ4=', 'Content-Length': '0'} 2023-05-24T21:28:37.687 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000048937a79724ee4a9-00646e8185-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.687 DEBUG:boto:path=/qskrps-49/obj5 2023-05-24T21:28:37.688 DEBUG:boto:auth_path=/qskrps-49/obj5 2023-05-24T21:28:37.688 DEBUG:boto:path=/qskrps-49/obj5?versionId=null 2023-05-24T21:28:37.688 DEBUG:boto:auth_path=/qskrps-49/obj5?versionId=null 2023-05-24T21:28:37.689 DEBUG:boto:Method: GET 2023-05-24T21:28:37.689 DEBUG:boto:Path: /qskrps-49/obj5?versionId=null 2023-05-24T21:28:37.689 DEBUG:boto:Data: 2023-05-24T21:28:37.690 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.690 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.690 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.691 DEBUG:boto:Params: {} 2023-05-24T21:28:37.691 DEBUG:boto:Token: None 2023-05-24T21:28:37.692 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj5?versionId=null 2023-05-24T21:28:37.692 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:xXfNsAxdpB2fa+2LzwJLjL2QuZ4= 2023-05-24T21:28:37.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:xXfNsAxdpB2fa+2LzwJLjL2QuZ4=', 'Content-Length': '0'} 2023-05-24T21:28:37.694 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000697c94273dd47cb7-00646e8185-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.695 DEBUG:boto:path=/qskrps-49/obj5 2023-05-24T21:28:37.695 DEBUG:boto:auth_path=/qskrps-49/obj5 2023-05-24T21:28:37.695 DEBUG:boto:path=/qskrps-49/obj5?versionId=null 2023-05-24T21:28:37.696 DEBUG:boto:auth_path=/qskrps-49/obj5?versionId=null 2023-05-24T21:28:37.696 DEBUG:boto:Method: HEAD 2023-05-24T21:28:37.696 DEBUG:boto:Path: /qskrps-49/obj5?versionId=null 2023-05-24T21:28:37.697 DEBUG:boto:Data: 2023-05-24T21:28:37.697 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.697 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.698 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.698 DEBUG:boto:Params: {} 2023-05-24T21:28:37.698 DEBUG:boto:Token: None 2023-05-24T21:28:37.699 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj5?versionId=null 2023-05-24T21:28:37.699 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:tW2JEadp5gGc2qkTeVtrl/kELtU= 2023-05-24T21:28:37.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:tW2JEadp5gGc2qkTeVtrl/kELtU=', 'Content-Length': '0'} 2023-05-24T21:28:37.701 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c17bb8ee3d6895bf-00646e8185-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.702 DEBUG:boto:path=/qskrps-49/obj5 2023-05-24T21:28:37.702 DEBUG:boto:auth_path=/qskrps-49/obj5 2023-05-24T21:28:37.702 DEBUG:boto:path=/qskrps-49/obj5?versionId=null 2023-05-24T21:28:37.703 DEBUG:boto:auth_path=/qskrps-49/obj5?versionId=null 2023-05-24T21:28:37.703 DEBUG:boto:Method: HEAD 2023-05-24T21:28:37.703 DEBUG:boto:Path: /qskrps-49/obj5?versionId=null 2023-05-24T21:28:37.704 DEBUG:boto:Data: 2023-05-24T21:28:37.704 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.704 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.705 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.705 DEBUG:boto:Params: {} 2023-05-24T21:28:37.705 DEBUG:boto:Token: None 2023-05-24T21:28:37.706 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj5?versionId=null 2023-05-24T21:28:37.706 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:tW2JEadp5gGc2qkTeVtrl/kELtU= 2023-05-24T21:28:37.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:tW2JEadp5gGc2qkTeVtrl/kELtU=', 'Content-Length': '0'} 2023-05-24T21:28:37.708 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b88a0cf47c8ba6a7-00646e8185-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-05-24T21:28:37.709 DEBUG:boto:path=/qskrps-49/obj5 2023-05-24T21:28:37.709 DEBUG:boto:auth_path=/qskrps-49/obj5 2023-05-24T21:28:37.710 DEBUG:boto:path=/qskrps-49/obj5?versionId=null 2023-05-24T21:28:37.710 DEBUG:boto:auth_path=/qskrps-49/obj5?versionId=null 2023-05-24T21:28:37.710 DEBUG:boto:Method: GET 2023-05-24T21:28:37.711 DEBUG:boto:Path: /qskrps-49/obj5?versionId=null 2023-05-24T21:28:37.711 DEBUG:boto:Data: 2023-05-24T21:28:37.711 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.712 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.712 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.713 DEBUG:boto:Params: {} 2023-05-24T21:28:37.713 DEBUG:boto:Token: None 2023-05-24T21:28:37.713 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj5?versionId=null 2023-05-24T21:28:37.714 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:xXfNsAxdpB2fa+2LzwJLjL2QuZ4= 2023-05-24T21:28:37.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:xXfNsAxdpB2fa+2LzwJLjL2QuZ4=', 'Content-Length': '0'} 2023-05-24T21:28:37.715 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008ab709b2c2e17eec-00646e8185-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.716 DEBUG:boto:path=/qskrps-49/obj5 2023-05-24T21:28:37.717 DEBUG:boto:auth_path=/qskrps-49/obj5 2023-05-24T21:28:37.717 DEBUG:boto:path=/qskrps-49/obj5?versionId=null 2023-05-24T21:28:37.717 DEBUG:boto:auth_path=/qskrps-49/obj5?versionId=null 2023-05-24T21:28:37.718 DEBUG:boto:Method: GET 2023-05-24T21:28:37.718 DEBUG:boto:Path: /qskrps-49/obj5?versionId=null 2023-05-24T21:28:37.718 DEBUG:boto:Data: 2023-05-24T21:28:37.719 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.719 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.719 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.720 DEBUG:boto:Params: {} 2023-05-24T21:28:37.720 DEBUG:boto:Token: None 2023-05-24T21:28:37.721 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj5?versionId=null 2023-05-24T21:28:37.721 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:xXfNsAxdpB2fa+2LzwJLjL2QuZ4= 2023-05-24T21:28:37.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:xXfNsAxdpB2fa+2LzwJLjL2QuZ4=', 'Content-Length': '0'} 2023-05-24T21:28:37.723 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000093cb20232a2b9641-00646e8185-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.723 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:37.724 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:37.724 DEBUG:boto:path=/qskrps-49/?prefix=obj5 2023-05-24T21:28:37.724 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj5 2023-05-24T21:28:37.725 DEBUG:boto:Method: GET 2023-05-24T21:28:37.725 DEBUG:boto:Path: /qskrps-49/?prefix=obj5 2023-05-24T21:28:37.725 DEBUG:boto:Data: 2023-05-24T21:28:37.726 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.726 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.727 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.727 DEBUG:boto:Params: {} 2023-05-24T21:28:37.727 DEBUG:boto:Token: None 2023-05-24T21:28:37.728 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/ 2023-05-24T21:28:37.728 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ= 2023-05-24T21:28:37.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ=', 'Content-Length': '0'} 2023-05-24T21:28:37.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc98757728313bbf-00646e8185-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.731 DEBUG:boto:b'qskrps-49obj51000falseobj52023-05-24T21:28:09.687Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:37.731 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:37.732 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:37.732 DEBUG:boto:path=/qskrps-49/?prefix=obj5 2023-05-24T21:28:37.733 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj5 2023-05-24T21:28:37.733 DEBUG:boto:Method: GET 2023-05-24T21:28:37.733 DEBUG:boto:Path: /qskrps-49/?prefix=obj5 2023-05-24T21:28:37.734 DEBUG:boto:Data: 2023-05-24T21:28:37.734 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.734 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.735 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.735 DEBUG:boto:Params: {} 2023-05-24T21:28:37.735 DEBUG:boto:Token: None 2023-05-24T21:28:37.736 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/ 2023-05-24T21:28:37.736 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ= 2023-05-24T21:28:37.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ=', 'Content-Length': '0'} 2023-05-24T21:28:37.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c0709fcb427c5b8-00646e8185-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.739 DEBUG:boto:b'qskrps-49obj51000falseobj52023-05-24T21:28:09.687Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:37.740 DEBUG:boto:path=/qskrps-49/obj5 2023-05-24T21:28:37.740 DEBUG:boto:auth_path=/qskrps-49/obj5 2023-05-24T21:28:37.740 DEBUG:boto:Method: HEAD 2023-05-24T21:28:37.741 DEBUG:boto:Path: /qskrps-49/obj5 2023-05-24T21:28:37.741 DEBUG:boto:Data: 2023-05-24T21:28:37.741 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.742 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.742 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.742 DEBUG:boto:Params: {} 2023-05-24T21:28:37.743 DEBUG:boto:Token: None 2023-05-24T21:28:37.743 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj5 2023-05-24T21:28:37.743 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:mOLJkfaovjByuPYVKHNCntr6OqQ= 2023-05-24T21:28:37.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:mOLJkfaovjByuPYVKHNCntr6OqQ=', 'Content-Length': '0'} 2023-05-24T21:28:37.745 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001d3adf091326b4e1-00646e8185-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.746 DEBUG:boto:path=/qskrps-49/obj5 2023-05-24T21:28:37.746 DEBUG:boto:auth_path=/qskrps-49/obj5 2023-05-24T21:28:37.746 DEBUG:boto:Method: HEAD 2023-05-24T21:28:37.747 DEBUG:boto:Path: /qskrps-49/obj5 2023-05-24T21:28:37.747 DEBUG:boto:Data: 2023-05-24T21:28:37.747 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.748 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.748 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.748 DEBUG:boto:Params: {} 2023-05-24T21:28:37.749 DEBUG:boto:Token: None 2023-05-24T21:28:37.749 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj5 2023-05-24T21:28:37.750 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:mOLJkfaovjByuPYVKHNCntr6OqQ= 2023-05-24T21:28:37.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:mOLJkfaovjByuPYVKHNCntr6OqQ=', 'Content-Length': '0'} 2023-05-24T21:28:37.751 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c4d398f6ac73b522-00646e8185-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-05-24T21:28:37.752 DEBUG:boto:path=/qskrps-49/obj5 2023-05-24T21:28:37.753 DEBUG:boto:auth_path=/qskrps-49/obj5 2023-05-24T21:28:37.753 DEBUG:boto:Method: GET 2023-05-24T21:28:37.753 DEBUG:boto:Path: /qskrps-49/obj5 2023-05-24T21:28:37.754 DEBUG:boto:Data: 2023-05-24T21:28:37.754 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.754 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.755 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.755 DEBUG:boto:Params: {} 2023-05-24T21:28:37.755 DEBUG:boto:Token: None 2023-05-24T21:28:37.756 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj5 2023-05-24T21:28:37.756 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:cjmPiEPGQ9M9BEHws0iTFov6nEw= 2023-05-24T21:28:37.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:cjmPiEPGQ9M9BEHws0iTFov6nEw=', 'Content-Length': '0'} 2023-05-24T21:28:37.758 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000084d0ec414d4be190-00646e8185-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.759 DEBUG:boto:path=/qskrps-49/obj5 2023-05-24T21:28:37.759 DEBUG:boto:auth_path=/qskrps-49/obj5 2023-05-24T21:28:37.759 DEBUG:boto:Method: GET 2023-05-24T21:28:37.760 DEBUG:boto:Path: /qskrps-49/obj5 2023-05-24T21:28:37.760 DEBUG:boto:Data: 2023-05-24T21:28:37.761 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.761 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.761 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.762 DEBUG:boto:Params: {} 2023-05-24T21:28:37.762 DEBUG:boto:Token: None 2023-05-24T21:28:37.762 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj5 2023-05-24T21:28:37.763 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:cjmPiEPGQ9M9BEHws0iTFov6nEw= 2023-05-24T21:28:37.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:cjmPiEPGQ9M9BEHws0iTFov6nEw=', 'Content-Length': '0'} 2023-05-24T21:28:37.764 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005f545b9fb946c4a1-00646e8185-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.765 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:37.766 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:37.766 DEBUG:boto:path=/qskrps-49/?prefix=obj5 2023-05-24T21:28:37.766 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj5 2023-05-24T21:28:37.767 DEBUG:boto:Method: GET 2023-05-24T21:28:37.767 DEBUG:boto:Path: /qskrps-49/?prefix=obj5 2023-05-24T21:28:37.767 DEBUG:boto:Data: 2023-05-24T21:28:37.768 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.768 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.768 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.769 DEBUG:boto:Params: {} 2023-05-24T21:28:37.769 DEBUG:boto:Token: None 2023-05-24T21:28:37.770 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/ 2023-05-24T21:28:37.770 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ= 2023-05-24T21:28:37.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ=', 'Content-Length': '0'} 2023-05-24T21:28:37.772 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a2c72b7ef8dffbe-00646e8185-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.773 DEBUG:boto:b'qskrps-49obj51000falseobj52023-05-24T21:28:09.687Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:37.773 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:37.774 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:37.774 DEBUG:boto:path=/qskrps-49/?prefix=obj5 2023-05-24T21:28:37.775 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj5 2023-05-24T21:28:37.775 DEBUG:boto:Method: GET 2023-05-24T21:28:37.775 DEBUG:boto:Path: /qskrps-49/?prefix=obj5 2023-05-24T21:28:37.776 DEBUG:boto:Data: 2023-05-24T21:28:37.776 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.776 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.777 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.777 DEBUG:boto:Params: {} 2023-05-24T21:28:37.777 DEBUG:boto:Token: None 2023-05-24T21:28:37.778 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/ 2023-05-24T21:28:37.778 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ= 2023-05-24T21:28:37.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ=', 'Content-Length': '0'} 2023-05-24T21:28:37.781 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005827761dacdadfb1-00646e8185-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.824 DEBUG:boto:b'qskrps-49obj51000falseobj52023-05-24T21:28:09.687Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:37.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-05-24T21:28:37.825 DEBUG:boto:path=/qskrps-49/obj6 2023-05-24T21:28:37.825 DEBUG:boto:auth_path=/qskrps-49/obj6 2023-05-24T21:28:37.826 DEBUG:boto:path=/qskrps-49/obj6?versionId=null 2023-05-24T21:28:37.826 DEBUG:boto:auth_path=/qskrps-49/obj6?versionId=null 2023-05-24T21:28:37.827 DEBUG:boto:Method: GET 2023-05-24T21:28:37.827 DEBUG:boto:Path: /qskrps-49/obj6?versionId=null 2023-05-24T21:28:37.827 DEBUG:boto:Data: 2023-05-24T21:28:37.828 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.828 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.828 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.829 DEBUG:boto:Params: {} 2023-05-24T21:28:37.829 DEBUG:boto:Token: None 2023-05-24T21:28:37.829 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj6?versionId=null 2023-05-24T21:28:37.830 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:m2Shs2LWenlyO8Y3AQIZgcgbV00= 2023-05-24T21:28:37.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:m2Shs2LWenlyO8Y3AQIZgcgbV00=', 'Content-Length': '0'} 2023-05-24T21:28:37.832 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bac4161053fb9f08-00646e8185-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.832 DEBUG:boto:path=/qskrps-49/obj6 2023-05-24T21:28:37.833 DEBUG:boto:auth_path=/qskrps-49/obj6 2023-05-24T21:28:37.833 DEBUG:boto:path=/qskrps-49/obj6?versionId=null 2023-05-24T21:28:37.834 DEBUG:boto:auth_path=/qskrps-49/obj6?versionId=null 2023-05-24T21:28:37.834 DEBUG:boto:Method: GET 2023-05-24T21:28:37.834 DEBUG:boto:Path: /qskrps-49/obj6?versionId=null 2023-05-24T21:28:37.835 DEBUG:boto:Data: 2023-05-24T21:28:37.835 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.835 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.836 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.836 DEBUG:boto:Params: {} 2023-05-24T21:28:37.836 DEBUG:boto:Token: None 2023-05-24T21:28:37.837 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj6?versionId=null 2023-05-24T21:28:37.837 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:m2Shs2LWenlyO8Y3AQIZgcgbV00= 2023-05-24T21:28:37.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:m2Shs2LWenlyO8Y3AQIZgcgbV00=', 'Content-Length': '0'} 2023-05-24T21:28:37.839 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008058e6a017b8d971-00646e8185-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.840 DEBUG:boto:path=/qskrps-49/obj6 2023-05-24T21:28:37.840 DEBUG:boto:auth_path=/qskrps-49/obj6 2023-05-24T21:28:37.841 DEBUG:boto:path=/qskrps-49/obj6?versionId=null 2023-05-24T21:28:37.841 DEBUG:boto:auth_path=/qskrps-49/obj6?versionId=null 2023-05-24T21:28:37.841 DEBUG:boto:Method: HEAD 2023-05-24T21:28:37.842 DEBUG:boto:Path: /qskrps-49/obj6?versionId=null 2023-05-24T21:28:37.842 DEBUG:boto:Data: 2023-05-24T21:28:37.842 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.843 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.843 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.843 DEBUG:boto:Params: {} 2023-05-24T21:28:37.844 DEBUG:boto:Token: None 2023-05-24T21:28:37.844 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj6?versionId=null 2023-05-24T21:28:37.845 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:m3Cps7ZiwquPD07PbMr2VgayQSM= 2023-05-24T21:28:37.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:m3Cps7ZiwquPD07PbMr2VgayQSM=', 'Content-Length': '0'} 2023-05-24T21:28:37.847 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000227f70e0af9a2913-00646e8185-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.847 DEBUG:boto:path=/qskrps-49/obj6 2023-05-24T21:28:37.848 DEBUG:boto:auth_path=/qskrps-49/obj6 2023-05-24T21:28:37.848 DEBUG:boto:path=/qskrps-49/obj6?versionId=null 2023-05-24T21:28:37.848 DEBUG:boto:auth_path=/qskrps-49/obj6?versionId=null 2023-05-24T21:28:37.849 DEBUG:boto:Method: HEAD 2023-05-24T21:28:37.849 DEBUG:boto:Path: /qskrps-49/obj6?versionId=null 2023-05-24T21:28:37.849 DEBUG:boto:Data: 2023-05-24T21:28:37.850 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.850 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.850 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.851 DEBUG:boto:Params: {} 2023-05-24T21:28:37.851 DEBUG:boto:Token: None 2023-05-24T21:28:37.851 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj6?versionId=null 2023-05-24T21:28:37.852 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:m3Cps7ZiwquPD07PbMr2VgayQSM= 2023-05-24T21:28:37.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:m3Cps7ZiwquPD07PbMr2VgayQSM=', 'Content-Length': '0'} 2023-05-24T21:28:37.854 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007bb1ff70d72c70c9-00646e8185-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-05-24T21:28:37.855 DEBUG:boto:path=/qskrps-49/obj6 2023-05-24T21:28:37.855 DEBUG:boto:auth_path=/qskrps-49/obj6 2023-05-24T21:28:37.856 DEBUG:boto:path=/qskrps-49/obj6?versionId=null 2023-05-24T21:28:37.856 DEBUG:boto:auth_path=/qskrps-49/obj6?versionId=null 2023-05-24T21:28:37.856 DEBUG:boto:Method: GET 2023-05-24T21:28:37.857 DEBUG:boto:Path: /qskrps-49/obj6?versionId=null 2023-05-24T21:28:37.857 DEBUG:boto:Data: 2023-05-24T21:28:37.857 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.858 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.858 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.858 DEBUG:boto:Params: {} 2023-05-24T21:28:37.859 DEBUG:boto:Token: None 2023-05-24T21:28:37.859 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj6?versionId=null 2023-05-24T21:28:37.860 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:m2Shs2LWenlyO8Y3AQIZgcgbV00= 2023-05-24T21:28:37.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:m2Shs2LWenlyO8Y3AQIZgcgbV00=', 'Content-Length': '0'} 2023-05-24T21:28:37.862 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000195db8d2caabc1a9-00646e8185-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.862 DEBUG:boto:path=/qskrps-49/obj6 2023-05-24T21:28:37.863 DEBUG:boto:auth_path=/qskrps-49/obj6 2023-05-24T21:28:37.863 DEBUG:boto:path=/qskrps-49/obj6?versionId=null 2023-05-24T21:28:37.863 DEBUG:boto:auth_path=/qskrps-49/obj6?versionId=null 2023-05-24T21:28:37.864 DEBUG:boto:Method: GET 2023-05-24T21:28:37.864 DEBUG:boto:Path: /qskrps-49/obj6?versionId=null 2023-05-24T21:28:37.864 DEBUG:boto:Data: 2023-05-24T21:28:37.865 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.865 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.865 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.866 DEBUG:boto:Params: {} 2023-05-24T21:28:37.866 DEBUG:boto:Token: None 2023-05-24T21:28:37.867 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj6?versionId=null 2023-05-24T21:28:37.867 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:m2Shs2LWenlyO8Y3AQIZgcgbV00= 2023-05-24T21:28:37.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:m2Shs2LWenlyO8Y3AQIZgcgbV00=', 'Content-Length': '0'} 2023-05-24T21:28:37.869 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000081deb237ff947cf1-00646e8185-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.870 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:37.870 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:37.871 DEBUG:boto:path=/qskrps-49/?prefix=obj6 2023-05-24T21:28:37.871 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj6 2023-05-24T21:28:37.871 DEBUG:boto:Method: GET 2023-05-24T21:28:37.872 DEBUG:boto:Path: /qskrps-49/?prefix=obj6 2023-05-24T21:28:37.872 DEBUG:boto:Data: 2023-05-24T21:28:37.872 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.873 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.873 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.873 DEBUG:boto:Params: {} 2023-05-24T21:28:37.874 DEBUG:boto:Token: None 2023-05-24T21:28:37.874 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/ 2023-05-24T21:28:37.875 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ= 2023-05-24T21:28:37.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ=', 'Content-Length': '0'} 2023-05-24T21:28:37.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a371a65a671eb27-00646e8185-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.878 DEBUG:boto:b'qskrps-49obj61000falseobj62023-05-24T21:28:09.703Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:37.878 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:37.878 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:37.879 DEBUG:boto:path=/qskrps-49/?prefix=obj6 2023-05-24T21:28:37.879 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj6 2023-05-24T21:28:37.880 DEBUG:boto:Method: GET 2023-05-24T21:28:37.880 DEBUG:boto:Path: /qskrps-49/?prefix=obj6 2023-05-24T21:28:37.880 DEBUG:boto:Data: 2023-05-24T21:28:37.881 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.881 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.881 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.882 DEBUG:boto:Params: {} 2023-05-24T21:28:37.882 DEBUG:boto:Token: None 2023-05-24T21:28:37.883 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/ 2023-05-24T21:28:37.883 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ= 2023-05-24T21:28:37.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ=', 'Content-Length': '0'} 2023-05-24T21:28:37.885 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f639d188c31a343-00646e8185-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.886 DEBUG:boto:b'qskrps-49obj61000falseobj62023-05-24T21:28:09.703Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:37.886 DEBUG:boto:path=/qskrps-49/obj6 2023-05-24T21:28:37.887 DEBUG:boto:auth_path=/qskrps-49/obj6 2023-05-24T21:28:37.887 DEBUG:boto:Method: HEAD 2023-05-24T21:28:37.888 DEBUG:boto:Path: /qskrps-49/obj6 2023-05-24T21:28:37.888 DEBUG:boto:Data: 2023-05-24T21:28:37.888 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.889 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.889 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.889 DEBUG:boto:Params: {} 2023-05-24T21:28:37.890 DEBUG:boto:Token: None 2023-05-24T21:28:37.890 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj6 2023-05-24T21:28:37.890 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:2Ip3mej8NaYf7InK87wHsbOvcY8= 2023-05-24T21:28:37.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:2Ip3mej8NaYf7InK87wHsbOvcY8=', 'Content-Length': '0'} 2023-05-24T21:28:37.892 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001b85afe75da8ad4c-00646e8185-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.893 DEBUG:boto:path=/qskrps-49/obj6 2023-05-24T21:28:37.893 DEBUG:boto:auth_path=/qskrps-49/obj6 2023-05-24T21:28:37.893 DEBUG:boto:Method: HEAD 2023-05-24T21:28:37.894 DEBUG:boto:Path: /qskrps-49/obj6 2023-05-24T21:28:37.894 DEBUG:boto:Data: 2023-05-24T21:28:37.894 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.895 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.895 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.895 DEBUG:boto:Params: {} 2023-05-24T21:28:37.896 DEBUG:boto:Token: None 2023-05-24T21:28:37.896 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj6 2023-05-24T21:28:37.896 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:2Ip3mej8NaYf7InK87wHsbOvcY8= 2023-05-24T21:28:37.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:2Ip3mej8NaYf7InK87wHsbOvcY8=', 'Content-Length': '0'} 2023-05-24T21:28:37.898 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005eb3cef8e2e4a6b6-00646e8185-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-05-24T21:28:37.899 DEBUG:boto:path=/qskrps-49/obj6 2023-05-24T21:28:37.900 DEBUG:boto:auth_path=/qskrps-49/obj6 2023-05-24T21:28:37.900 DEBUG:boto:Method: GET 2023-05-24T21:28:37.900 DEBUG:boto:Path: /qskrps-49/obj6 2023-05-24T21:28:37.901 DEBUG:boto:Data: 2023-05-24T21:28:37.901 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.901 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.902 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.902 DEBUG:boto:Params: {} 2023-05-24T21:28:37.903 DEBUG:boto:Token: None 2023-05-24T21:28:37.903 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj6 2023-05-24T21:28:37.903 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:p5zK6b1zdmwvSJ8ufG77dsaaZI8= 2023-05-24T21:28:37.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:p5zK6b1zdmwvSJ8ufG77dsaaZI8=', 'Content-Length': '0'} 2023-05-24T21:28:37.905 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a1042c8b2e02b387-00646e8185-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.948 DEBUG:boto:path=/qskrps-49/obj6 2023-05-24T21:28:37.949 DEBUG:boto:auth_path=/qskrps-49/obj6 2023-05-24T21:28:37.949 DEBUG:boto:Method: GET 2023-05-24T21:28:37.949 DEBUG:boto:Path: /qskrps-49/obj6 2023-05-24T21:28:37.950 DEBUG:boto:Data: 2023-05-24T21:28:37.950 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.950 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.951 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.951 DEBUG:boto:Params: {} 2023-05-24T21:28:37.951 DEBUG:boto:Token: None 2023-05-24T21:28:37.952 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj6 2023-05-24T21:28:37.952 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:p5zK6b1zdmwvSJ8ufG77dsaaZI8= 2023-05-24T21:28:37.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:p5zK6b1zdmwvSJ8ufG77dsaaZI8=', 'Content-Length': '0'} 2023-05-24T21:28:37.954 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001ae9f3b27f55265a-00646e8185-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.955 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:37.955 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:37.955 DEBUG:boto:path=/qskrps-49/?prefix=obj6 2023-05-24T21:28:37.956 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj6 2023-05-24T21:28:37.956 DEBUG:boto:Method: GET 2023-05-24T21:28:37.957 DEBUG:boto:Path: /qskrps-49/?prefix=obj6 2023-05-24T21:28:37.957 DEBUG:boto:Data: 2023-05-24T21:28:37.957 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.958 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.958 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.958 DEBUG:boto:Params: {} 2023-05-24T21:28:37.959 DEBUG:boto:Token: None 2023-05-24T21:28:37.959 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/ 2023-05-24T21:28:37.960 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ= 2023-05-24T21:28:37.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ=', 'Content-Length': '0'} 2023-05-24T21:28:37.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000723e1495ca7c72c9-00646e8185-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.962 DEBUG:boto:b'qskrps-49obj61000falseobj62023-05-24T21:28:09.703Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:37.963 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:37.963 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:37.964 DEBUG:boto:path=/qskrps-49/?prefix=obj6 2023-05-24T21:28:37.964 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj6 2023-05-24T21:28:37.964 DEBUG:boto:Method: GET 2023-05-24T21:28:37.965 DEBUG:boto:Path: /qskrps-49/?prefix=obj6 2023-05-24T21:28:37.965 DEBUG:boto:Data: 2023-05-24T21:28:37.965 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.966 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.966 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.967 DEBUG:boto:Params: {} 2023-05-24T21:28:37.967 DEBUG:boto:Token: None 2023-05-24T21:28:37.967 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/ 2023-05-24T21:28:37.968 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ= 2023-05-24T21:28:37.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PDmprbJH+PsZRCrlNNCAWBKqeSQ=', 'Content-Length': '0'} 2023-05-24T21:28:37.970 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058f6dce58a66acb0-00646e8185-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.971 DEBUG:boto:b'qskrps-49obj61000falseobj62023-05-24T21:28:09.703Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:37.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-05-24T21:28:37.972 DEBUG:boto:path=/qskrps-49/obj7 2023-05-24T21:28:37.972 DEBUG:boto:auth_path=/qskrps-49/obj7 2023-05-24T21:28:37.972 DEBUG:boto:path=/qskrps-49/obj7?versionId=null 2023-05-24T21:28:37.973 DEBUG:boto:auth_path=/qskrps-49/obj7?versionId=null 2023-05-24T21:28:37.973 DEBUG:boto:Method: GET 2023-05-24T21:28:37.973 DEBUG:boto:Path: /qskrps-49/obj7?versionId=null 2023-05-24T21:28:37.974 DEBUG:boto:Data: 2023-05-24T21:28:37.974 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.974 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.975 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.975 DEBUG:boto:Params: {} 2023-05-24T21:28:37.975 DEBUG:boto:Token: None 2023-05-24T21:28:37.976 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj7?versionId=null 2023-05-24T21:28:37.976 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:q+X/pu1o88shtIx5GEGRZiyhiW4= 2023-05-24T21:28:37.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:q+X/pu1o88shtIx5GEGRZiyhiW4=', 'Content-Length': '0'} 2023-05-24T21:28:37.978 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c5eb4a511d0e49ea-00646e8185-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.979 DEBUG:boto:path=/qskrps-49/obj7 2023-05-24T21:28:37.979 DEBUG:boto:auth_path=/qskrps-49/obj7 2023-05-24T21:28:37.980 DEBUG:boto:path=/qskrps-49/obj7?versionId=null 2023-05-24T21:28:37.980 DEBUG:boto:auth_path=/qskrps-49/obj7?versionId=null 2023-05-24T21:28:37.981 DEBUG:boto:Method: GET 2023-05-24T21:28:37.981 DEBUG:boto:Path: /qskrps-49/obj7?versionId=null 2023-05-24T21:28:37.982 DEBUG:boto:Data: 2023-05-24T21:28:37.982 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.982 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.983 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.983 DEBUG:boto:Params: {} 2023-05-24T21:28:37.983 DEBUG:boto:Token: None 2023-05-24T21:28:37.984 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj7?versionId=null 2023-05-24T21:28:37.984 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:q+X/pu1o88shtIx5GEGRZiyhiW4= 2023-05-24T21:28:37.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:q+X/pu1o88shtIx5GEGRZiyhiW4=', 'Content-Length': '0'} 2023-05-24T21:28:37.987 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000e59233742c9a343-00646e8185-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.987 DEBUG:boto:path=/qskrps-49/obj7 2023-05-24T21:28:37.988 DEBUG:boto:auth_path=/qskrps-49/obj7 2023-05-24T21:28:37.988 DEBUG:boto:path=/qskrps-49/obj7?versionId=null 2023-05-24T21:28:37.988 DEBUG:boto:auth_path=/qskrps-49/obj7?versionId=null 2023-05-24T21:28:37.989 DEBUG:boto:Method: HEAD 2023-05-24T21:28:37.989 DEBUG:boto:Path: /qskrps-49/obj7?versionId=null 2023-05-24T21:28:37.989 DEBUG:boto:Data: 2023-05-24T21:28:37.990 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.990 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:37.990 DEBUG:boto:Port: 8001 2023-05-24T21:28:37.991 DEBUG:boto:Params: {} 2023-05-24T21:28:37.991 DEBUG:boto:Token: None 2023-05-24T21:28:37.992 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj7?versionId=null 2023-05-24T21:28:37.992 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:heH/6YPs/A6H9lolWmIoG2XfNJo= 2023-05-24T21:28:37.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:heH/6YPs/A6H9lolWmIoG2XfNJo=', 'Content-Length': '0'} 2023-05-24T21:28:37.994 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001c726e13c7686c9c-00646e8185-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:37.994 DEBUG:boto:path=/qskrps-49/obj7 2023-05-24T21:28:37.995 DEBUG:boto:auth_path=/qskrps-49/obj7 2023-05-24T21:28:37.995 DEBUG:boto:path=/qskrps-49/obj7?versionId=null 2023-05-24T21:28:37.995 DEBUG:boto:auth_path=/qskrps-49/obj7?versionId=null 2023-05-24T21:28:37.996 DEBUG:boto:Method: HEAD 2023-05-24T21:28:37.996 DEBUG:boto:Path: /qskrps-49/obj7?versionId=null 2023-05-24T21:28:37.997 DEBUG:boto:Data: 2023-05-24T21:28:37.997 DEBUG:boto:Headers: {} 2023-05-24T21:28:37.997 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:37.998 DEBUG:boto:Port: 8000 2023-05-24T21:28:37.998 DEBUG:boto:Params: {} 2023-05-24T21:28:37.999 DEBUG:boto:Token: None 2023-05-24T21:28:37.999 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:37 GMT /qskrps-49/obj7?versionId=null 2023-05-24T21:28:37.999 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:heH/6YPs/A6H9lolWmIoG2XfNJo= 2023-05-24T21:28:38.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:37 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:heH/6YPs/A6H9lolWmIoG2XfNJo=', 'Content-Length': '0'} 2023-05-24T21:28:38.001 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000455e16d5b2e327f6-00646e8185-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:38.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-05-24T21:28:38.002 DEBUG:boto:path=/qskrps-49/obj7 2023-05-24T21:28:38.003 DEBUG:boto:auth_path=/qskrps-49/obj7 2023-05-24T21:28:38.003 DEBUG:boto:path=/qskrps-49/obj7?versionId=null 2023-05-24T21:28:38.003 DEBUG:boto:auth_path=/qskrps-49/obj7?versionId=null 2023-05-24T21:28:38.004 DEBUG:boto:Method: GET 2023-05-24T21:28:38.004 DEBUG:boto:Path: /qskrps-49/obj7?versionId=null 2023-05-24T21:28:38.005 DEBUG:boto:Data: 2023-05-24T21:28:38.005 DEBUG:boto:Headers: {} 2023-05-24T21:28:38.005 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:38.006 DEBUG:boto:Port: 8001 2023-05-24T21:28:38.006 DEBUG:boto:Params: {} 2023-05-24T21:28:38.006 DEBUG:boto:Token: None 2023-05-24T21:28:38.007 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:38 GMT /qskrps-49/obj7?versionId=null 2023-05-24T21:28:38.007 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:lQQ3b/FGPOO2HxE35nGYTw6zD3A= 2023-05-24T21:28:38.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:lQQ3b/FGPOO2HxE35nGYTw6zD3A=', 'Content-Length': '0'} 2023-05-24T21:28:38.009 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000066d7af0998ec26fa-00646e8186-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:38.010 DEBUG:boto:path=/qskrps-49/obj7 2023-05-24T21:28:38.011 DEBUG:boto:auth_path=/qskrps-49/obj7 2023-05-24T21:28:38.011 DEBUG:boto:path=/qskrps-49/obj7?versionId=null 2023-05-24T21:28:38.011 DEBUG:boto:auth_path=/qskrps-49/obj7?versionId=null 2023-05-24T21:28:38.012 DEBUG:boto:Method: GET 2023-05-24T21:28:38.012 DEBUG:boto:Path: /qskrps-49/obj7?versionId=null 2023-05-24T21:28:38.012 DEBUG:boto:Data: 2023-05-24T21:28:38.013 DEBUG:boto:Headers: {} 2023-05-24T21:28:38.013 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:38.013 DEBUG:boto:Port: 8000 2023-05-24T21:28:38.014 DEBUG:boto:Params: {} 2023-05-24T21:28:38.014 DEBUG:boto:Token: None 2023-05-24T21:28:38.015 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:38 GMT /qskrps-49/obj7?versionId=null 2023-05-24T21:28:38.015 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:lQQ3b/FGPOO2HxE35nGYTw6zD3A= 2023-05-24T21:28:38.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:lQQ3b/FGPOO2HxE35nGYTw6zD3A=', 'Content-Length': '0'} 2023-05-24T21:28:38.017 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000862a5cf975d4e1c8-00646e8186-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:38.018 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:38.018 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:38.019 DEBUG:boto:path=/qskrps-49/?prefix=obj7 2023-05-24T21:28:38.019 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj7 2023-05-24T21:28:38.020 DEBUG:boto:Method: GET 2023-05-24T21:28:38.020 DEBUG:boto:Path: /qskrps-49/?prefix=obj7 2023-05-24T21:28:38.020 DEBUG:boto:Data: 2023-05-24T21:28:38.021 DEBUG:boto:Headers: {} 2023-05-24T21:28:38.021 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:38.021 DEBUG:boto:Port: 8001 2023-05-24T21:28:38.022 DEBUG:boto:Params: {} 2023-05-24T21:28:38.022 DEBUG:boto:Token: None 2023-05-24T21:28:38.023 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:38 GMT /qskrps-49/ 2023-05-24T21:28:38.023 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:d3k2gXaWyo3VP8MFgxwaBg4JBqU= 2023-05-24T21:28:38.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:d3k2gXaWyo3VP8MFgxwaBg4JBqU=', 'Content-Length': '0'} 2023-05-24T21:28:38.026 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7e4782b918ccf7b-00646e8186-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:38.026 DEBUG:boto:b'qskrps-49obj71000falseobj72023-05-24T21:28:09.762Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:38.027 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:38.027 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:38.028 DEBUG:boto:path=/qskrps-49/?prefix=obj7 2023-05-24T21:28:38.028 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj7 2023-05-24T21:28:38.028 DEBUG:boto:Method: GET 2023-05-24T21:28:38.029 DEBUG:boto:Path: /qskrps-49/?prefix=obj7 2023-05-24T21:28:38.029 DEBUG:boto:Data: 2023-05-24T21:28:38.030 DEBUG:boto:Headers: {} 2023-05-24T21:28:38.030 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:38.030 DEBUG:boto:Port: 8000 2023-05-24T21:28:38.031 DEBUG:boto:Params: {} 2023-05-24T21:28:38.031 DEBUG:boto:Token: None 2023-05-24T21:28:38.032 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:38 GMT /qskrps-49/ 2023-05-24T21:28:38.032 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:d3k2gXaWyo3VP8MFgxwaBg4JBqU= 2023-05-24T21:28:38.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:d3k2gXaWyo3VP8MFgxwaBg4JBqU=', 'Content-Length': '0'} 2023-05-24T21:28:38.035 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1c0ee44fbb5e8a5-00646e8186-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:38.035 DEBUG:boto:b'qskrps-49obj71000falseobj72023-05-24T21:28:09.762Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:38.036 DEBUG:boto:path=/qskrps-49/obj7 2023-05-24T21:28:38.036 DEBUG:boto:auth_path=/qskrps-49/obj7 2023-05-24T21:28:38.037 DEBUG:boto:Method: HEAD 2023-05-24T21:28:38.037 DEBUG:boto:Path: /qskrps-49/obj7 2023-05-24T21:28:38.037 DEBUG:boto:Data: 2023-05-24T21:28:38.038 DEBUG:boto:Headers: {} 2023-05-24T21:28:38.038 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:38.038 DEBUG:boto:Port: 8001 2023-05-24T21:28:38.039 DEBUG:boto:Params: {} 2023-05-24T21:28:38.039 DEBUG:boto:Token: None 2023-05-24T21:28:38.039 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:38 GMT /qskrps-49/obj7 2023-05-24T21:28:38.040 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:6to+0jilO3QTdWY0S21EVxzuTMU= 2023-05-24T21:28:38.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:6to+0jilO3QTdWY0S21EVxzuTMU=', 'Content-Length': '0'} 2023-05-24T21:28:38.042 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000393d874b88057c55-00646e8186-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:38.043 DEBUG:boto:path=/qskrps-49/obj7 2023-05-24T21:28:38.043 DEBUG:boto:auth_path=/qskrps-49/obj7 2023-05-24T21:28:38.043 DEBUG:boto:Method: HEAD 2023-05-24T21:28:38.044 DEBUG:boto:Path: /qskrps-49/obj7 2023-05-24T21:28:38.044 DEBUG:boto:Data: 2023-05-24T21:28:38.044 DEBUG:boto:Headers: {} 2023-05-24T21:28:38.045 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:38.045 DEBUG:boto:Port: 8000 2023-05-24T21:28:38.045 DEBUG:boto:Params: {} 2023-05-24T21:28:38.046 DEBUG:boto:Token: None 2023-05-24T21:28:38.046 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:38 GMT /qskrps-49/obj7 2023-05-24T21:28:38.047 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:6to+0jilO3QTdWY0S21EVxzuTMU= 2023-05-24T21:28:38.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:6to+0jilO3QTdWY0S21EVxzuTMU=', 'Content-Length': '0'} 2023-05-24T21:28:38.049 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd8a9b6c379d4911-00646e8186-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:38.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-05-24T21:28:38.050 DEBUG:boto:path=/qskrps-49/obj7 2023-05-24T21:28:38.051 DEBUG:boto:auth_path=/qskrps-49/obj7 2023-05-24T21:28:38.051 DEBUG:boto:Method: GET 2023-05-24T21:28:38.051 DEBUG:boto:Path: /qskrps-49/obj7 2023-05-24T21:28:38.052 DEBUG:boto:Data: 2023-05-24T21:28:38.052 DEBUG:boto:Headers: {} 2023-05-24T21:28:38.052 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:38.053 DEBUG:boto:Port: 8001 2023-05-24T21:28:38.053 DEBUG:boto:Params: {} 2023-05-24T21:28:38.054 DEBUG:boto:Token: None 2023-05-24T21:28:38.054 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:38 GMT /qskrps-49/obj7 2023-05-24T21:28:38.054 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:LUm2vunxMw1cCeCoR06x+jFMG6Q= 2023-05-24T21:28:38.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:LUm2vunxMw1cCeCoR06x+jFMG6Q=', 'Content-Length': '0'} 2023-05-24T21:28:38.057 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000023541b824064211c-00646e8186-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:38.057 DEBUG:boto:path=/qskrps-49/obj7 2023-05-24T21:28:38.058 DEBUG:boto:auth_path=/qskrps-49/obj7 2023-05-24T21:28:38.058 DEBUG:boto:Method: GET 2023-05-24T21:28:38.058 DEBUG:boto:Path: /qskrps-49/obj7 2023-05-24T21:28:38.059 DEBUG:boto:Data: 2023-05-24T21:28:38.059 DEBUG:boto:Headers: {} 2023-05-24T21:28:38.060 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:38.060 DEBUG:boto:Port: 8000 2023-05-24T21:28:38.060 DEBUG:boto:Params: {} 2023-05-24T21:28:38.061 DEBUG:boto:Token: None 2023-05-24T21:28:38.061 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:38 GMT /qskrps-49/obj7 2023-05-24T21:28:38.062 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:LUm2vunxMw1cCeCoR06x+jFMG6Q= 2023-05-24T21:28:38.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:LUm2vunxMw1cCeCoR06x+jFMG6Q=', 'Content-Length': '0'} 2023-05-24T21:28:38.063 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009c742022a6059f24-00646e8186-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:38.104 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:38.105 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:38.105 DEBUG:boto:path=/qskrps-49/?prefix=obj7 2023-05-24T21:28:38.105 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj7 2023-05-24T21:28:38.106 DEBUG:boto:Method: GET 2023-05-24T21:28:38.106 DEBUG:boto:Path: /qskrps-49/?prefix=obj7 2023-05-24T21:28:38.107 DEBUG:boto:Data: 2023-05-24T21:28:38.107 DEBUG:boto:Headers: {} 2023-05-24T21:28:38.107 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:38.108 DEBUG:boto:Port: 8001 2023-05-24T21:28:38.108 DEBUG:boto:Params: {} 2023-05-24T21:28:38.108 DEBUG:boto:Token: None 2023-05-24T21:28:38.109 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:38 GMT /qskrps-49/ 2023-05-24T21:28:38.109 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:d3k2gXaWyo3VP8MFgxwaBg4JBqU= 2023-05-24T21:28:38.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:d3k2gXaWyo3VP8MFgxwaBg4JBqU=', 'Content-Length': '0'} 2023-05-24T21:28:38.112 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000763d4fcc0d90367e-00646e8186-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:38.112 DEBUG:boto:b'qskrps-49obj71000falseobj72023-05-24T21:28:09.762Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:38.113 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:38.114 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:38.114 DEBUG:boto:path=/qskrps-49/?prefix=obj7 2023-05-24T21:28:38.114 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj7 2023-05-24T21:28:38.115 DEBUG:boto:Method: GET 2023-05-24T21:28:38.115 DEBUG:boto:Path: /qskrps-49/?prefix=obj7 2023-05-24T21:28:38.115 DEBUG:boto:Data: 2023-05-24T21:28:38.116 DEBUG:boto:Headers: {} 2023-05-24T21:28:38.116 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:38.116 DEBUG:boto:Port: 8000 2023-05-24T21:28:38.117 DEBUG:boto:Params: {} 2023-05-24T21:28:38.117 DEBUG:boto:Token: None 2023-05-24T21:28:38.118 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:38 GMT /qskrps-49/ 2023-05-24T21:28:38.118 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:d3k2gXaWyo3VP8MFgxwaBg4JBqU= 2023-05-24T21:28:38.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:d3k2gXaWyo3VP8MFgxwaBg4JBqU=', 'Content-Length': '0'} 2023-05-24T21:28:38.121 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000112aa96dcb095c0-00646e8186-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:38.122 DEBUG:boto:b'qskrps-49obj71000falseobj72023-05-24T21:28:09.762Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:38.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-05-24T21:28:38.123 DEBUG:boto:path=/qskrps-49/obj8 2023-05-24T21:28:38.123 DEBUG:boto:auth_path=/qskrps-49/obj8 2023-05-24T21:28:38.123 DEBUG:boto:path=/qskrps-49/obj8?versionId=null 2023-05-24T21:28:38.124 DEBUG:boto:auth_path=/qskrps-49/obj8?versionId=null 2023-05-24T21:28:38.124 DEBUG:boto:Method: GET 2023-05-24T21:28:38.125 DEBUG:boto:Path: /qskrps-49/obj8?versionId=null 2023-05-24T21:28:38.125 DEBUG:boto:Data: 2023-05-24T21:28:38.125 DEBUG:boto:Headers: {} 2023-05-24T21:28:38.126 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:38.126 DEBUG:boto:Port: 8001 2023-05-24T21:28:38.126 DEBUG:boto:Params: {} 2023-05-24T21:28:38.127 DEBUG:boto:Token: None 2023-05-24T21:28:38.127 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:38 GMT /qskrps-49/obj8?versionId=null 2023-05-24T21:28:38.128 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:CEFn1rAl2OiPeejlfn1+dLgdNPU= 2023-05-24T21:28:38.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:CEFn1rAl2OiPeejlfn1+dLgdNPU=', 'Content-Length': '0'} 2023-05-24T21:28:38.129 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e4e5439d7be2e0f3-00646e8186-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:38.130 DEBUG:boto:path=/qskrps-49/obj8 2023-05-24T21:28:38.131 DEBUG:boto:auth_path=/qskrps-49/obj8 2023-05-24T21:28:38.131 DEBUG:boto:path=/qskrps-49/obj8?versionId=null 2023-05-24T21:28:38.131 DEBUG:boto:auth_path=/qskrps-49/obj8?versionId=null 2023-05-24T21:28:38.132 DEBUG:boto:Method: GET 2023-05-24T21:28:38.132 DEBUG:boto:Path: /qskrps-49/obj8?versionId=null 2023-05-24T21:28:38.132 DEBUG:boto:Data: 2023-05-24T21:28:38.133 DEBUG:boto:Headers: {} 2023-05-24T21:28:38.133 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:38.134 DEBUG:boto:Port: 8000 2023-05-24T21:28:38.134 DEBUG:boto:Params: {} 2023-05-24T21:28:38.134 DEBUG:boto:Token: None 2023-05-24T21:28:38.135 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:38 GMT /qskrps-49/obj8?versionId=null 2023-05-24T21:28:38.135 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:CEFn1rAl2OiPeejlfn1+dLgdNPU= 2023-05-24T21:28:38.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:CEFn1rAl2OiPeejlfn1+dLgdNPU=', 'Content-Length': '0'} 2023-05-24T21:28:38.138 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e4040dbd7c37ad4c-00646e8186-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:38.138 DEBUG:boto:path=/qskrps-49/obj8 2023-05-24T21:28:38.139 DEBUG:boto:auth_path=/qskrps-49/obj8 2023-05-24T21:28:38.139 DEBUG:boto:path=/qskrps-49/obj8?versionId=null 2023-05-24T21:28:38.139 DEBUG:boto:auth_path=/qskrps-49/obj8?versionId=null 2023-05-24T21:28:38.140 DEBUG:boto:Method: HEAD 2023-05-24T21:28:38.140 DEBUG:boto:Path: /qskrps-49/obj8?versionId=null 2023-05-24T21:28:38.140 DEBUG:boto:Data: 2023-05-24T21:28:38.141 DEBUG:boto:Headers: {} 2023-05-24T21:28:38.141 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:38.141 DEBUG:boto:Port: 8001 2023-05-24T21:28:38.142 DEBUG:boto:Params: {} 2023-05-24T21:28:38.142 DEBUG:boto:Token: None 2023-05-24T21:28:38.143 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:38 GMT /qskrps-49/obj8?versionId=null 2023-05-24T21:28:38.143 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:dWgpsM2QIMPX8GzrIzex3OeMY98= 2023-05-24T21:28:38.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:dWgpsM2QIMPX8GzrIzex3OeMY98=', 'Content-Length': '0'} 2023-05-24T21:28:38.145 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000ebf8484442c2aee-00646e8186-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:38.145 DEBUG:boto:path=/qskrps-49/obj8 2023-05-24T21:28:38.146 DEBUG:boto:auth_path=/qskrps-49/obj8 2023-05-24T21:28:38.146 DEBUG:boto:path=/qskrps-49/obj8?versionId=null 2023-05-24T21:28:38.146 DEBUG:boto:auth_path=/qskrps-49/obj8?versionId=null 2023-05-24T21:28:38.147 DEBUG:boto:Method: HEAD 2023-05-24T21:28:38.147 DEBUG:boto:Path: /qskrps-49/obj8?versionId=null 2023-05-24T21:28:38.147 DEBUG:boto:Data: 2023-05-24T21:28:38.148 DEBUG:boto:Headers: {} 2023-05-24T21:28:38.148 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:38.149 DEBUG:boto:Port: 8000 2023-05-24T21:28:38.149 DEBUG:boto:Params: {} 2023-05-24T21:28:38.149 DEBUG:boto:Token: None 2023-05-24T21:28:38.150 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:38 GMT /qskrps-49/obj8?versionId=null 2023-05-24T21:28:38.150 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:dWgpsM2QIMPX8GzrIzex3OeMY98= 2023-05-24T21:28:38.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:dWgpsM2QIMPX8GzrIzex3OeMY98=', 'Content-Length': '0'} 2023-05-24T21:28:38.152 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000476a1db9f1e4159d-00646e8186-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:38.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-05-24T21:28:38.153 DEBUG:boto:path=/qskrps-49/obj8 2023-05-24T21:28:38.153 DEBUG:boto:auth_path=/qskrps-49/obj8 2023-05-24T21:28:38.154 DEBUG:boto:path=/qskrps-49/obj8?versionId=null 2023-05-24T21:28:38.154 DEBUG:boto:auth_path=/qskrps-49/obj8?versionId=null 2023-05-24T21:28:38.154 DEBUG:boto:Method: GET 2023-05-24T21:28:38.155 DEBUG:boto:Path: /qskrps-49/obj8?versionId=null 2023-05-24T21:28:38.155 DEBUG:boto:Data: 2023-05-24T21:28:38.156 DEBUG:boto:Headers: {} 2023-05-24T21:28:38.156 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:38.156 DEBUG:boto:Port: 8001 2023-05-24T21:28:38.157 DEBUG:boto:Params: {} 2023-05-24T21:28:38.157 DEBUG:boto:Token: None 2023-05-24T21:28:38.158 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:38 GMT /qskrps-49/obj8?versionId=null 2023-05-24T21:28:38.158 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:CEFn1rAl2OiPeejlfn1+dLgdNPU= 2023-05-24T21:28:38.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:CEFn1rAl2OiPeejlfn1+dLgdNPU=', 'Content-Length': '0'} 2023-05-24T21:28:38.160 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d7ac8eaba4e6af6f-00646e8186-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:38.160 DEBUG:boto:path=/qskrps-49/obj8 2023-05-24T21:28:38.161 DEBUG:boto:auth_path=/qskrps-49/obj8 2023-05-24T21:28:38.161 DEBUG:boto:path=/qskrps-49/obj8?versionId=null 2023-05-24T21:28:38.162 DEBUG:boto:auth_path=/qskrps-49/obj8?versionId=null 2023-05-24T21:28:38.162 DEBUG:boto:Method: GET 2023-05-24T21:28:38.162 DEBUG:boto:Path: /qskrps-49/obj8?versionId=null 2023-05-24T21:28:38.163 DEBUG:boto:Data: 2023-05-24T21:28:38.163 DEBUG:boto:Headers: {} 2023-05-24T21:28:38.163 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:38.164 DEBUG:boto:Port: 8000 2023-05-24T21:28:38.164 DEBUG:boto:Params: {} 2023-05-24T21:28:38.165 DEBUG:boto:Token: None 2023-05-24T21:28:38.165 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:38 GMT /qskrps-49/obj8?versionId=null 2023-05-24T21:28:38.165 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:CEFn1rAl2OiPeejlfn1+dLgdNPU= 2023-05-24T21:28:38.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:CEFn1rAl2OiPeejlfn1+dLgdNPU=', 'Content-Length': '0'} 2023-05-24T21:28:38.167 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c5c9cb64ef6cd590-00646e8186-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:38.168 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:38.168 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:38.169 DEBUG:boto:path=/qskrps-49/?prefix=obj8 2023-05-24T21:28:38.169 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj8 2023-05-24T21:28:38.170 DEBUG:boto:Method: GET 2023-05-24T21:28:38.170 DEBUG:boto:Path: /qskrps-49/?prefix=obj8 2023-05-24T21:28:38.170 DEBUG:boto:Data: 2023-05-24T21:28:38.171 DEBUG:boto:Headers: {} 2023-05-24T21:28:38.171 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:38.172 DEBUG:boto:Port: 8001 2023-05-24T21:28:38.172 DEBUG:boto:Params: {} 2023-05-24T21:28:38.172 DEBUG:boto:Token: None 2023-05-24T21:28:38.173 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:38 GMT /qskrps-49/ 2023-05-24T21:28:38.173 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:d3k2gXaWyo3VP8MFgxwaBg4JBqU= 2023-05-24T21:28:38.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:d3k2gXaWyo3VP8MFgxwaBg4JBqU=', 'Content-Length': '0'} 2023-05-24T21:28:38.176 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087e31142f97a9d4b-00646e8186-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:38.176 DEBUG:boto:b'qskrps-49obj81000falseobj82023-05-24T21:28:09.781Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:38.177 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:38.177 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:38.178 DEBUG:boto:path=/qskrps-49/?prefix=obj8 2023-05-24T21:28:38.178 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj8 2023-05-24T21:28:38.178 DEBUG:boto:Method: GET 2023-05-24T21:28:38.179 DEBUG:boto:Path: /qskrps-49/?prefix=obj8 2023-05-24T21:28:38.179 DEBUG:boto:Data: 2023-05-24T21:28:38.179 DEBUG:boto:Headers: {} 2023-05-24T21:28:38.180 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:38.180 DEBUG:boto:Port: 8000 2023-05-24T21:28:38.181 DEBUG:boto:Params: {} 2023-05-24T21:28:38.181 DEBUG:boto:Token: None 2023-05-24T21:28:38.181 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:38 GMT /qskrps-49/ 2023-05-24T21:28:38.182 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:d3k2gXaWyo3VP8MFgxwaBg4JBqU= 2023-05-24T21:28:38.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:d3k2gXaWyo3VP8MFgxwaBg4JBqU=', 'Content-Length': '0'} 2023-05-24T21:28:38.184 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005718baea54cd6738-00646e8186-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:38.185 DEBUG:boto:b'qskrps-49obj81000falseobj82023-05-24T21:28:09.781Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:38.185 DEBUG:boto:path=/qskrps-49/obj8 2023-05-24T21:28:38.186 DEBUG:boto:auth_path=/qskrps-49/obj8 2023-05-24T21:28:38.186 DEBUG:boto:Method: HEAD 2023-05-24T21:28:38.186 DEBUG:boto:Path: /qskrps-49/obj8 2023-05-24T21:28:38.187 DEBUG:boto:Data: 2023-05-24T21:28:38.187 DEBUG:boto:Headers: {} 2023-05-24T21:28:38.187 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:38.188 DEBUG:boto:Port: 8001 2023-05-24T21:28:38.188 DEBUG:boto:Params: {} 2023-05-24T21:28:38.189 DEBUG:boto:Token: None 2023-05-24T21:28:38.189 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:38 GMT /qskrps-49/obj8 2023-05-24T21:28:38.189 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:0DbSqR0yG/Q37J3ZqfoM8+UcS34= 2023-05-24T21:28:38.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:0DbSqR0yG/Q37J3ZqfoM8+UcS34=', 'Content-Length': '0'} 2023-05-24T21:28:38.192 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e836c8f628afb96f-00646e8186-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:38.192 DEBUG:boto:path=/qskrps-49/obj8 2023-05-24T21:28:38.193 DEBUG:boto:auth_path=/qskrps-49/obj8 2023-05-24T21:28:38.193 DEBUG:boto:Method: HEAD 2023-05-24T21:28:38.193 DEBUG:boto:Path: /qskrps-49/obj8 2023-05-24T21:28:38.194 DEBUG:boto:Data: 2023-05-24T21:28:38.194 DEBUG:boto:Headers: {} 2023-05-24T21:28:38.194 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:38.195 DEBUG:boto:Port: 8000 2023-05-24T21:28:38.195 DEBUG:boto:Params: {} 2023-05-24T21:28:38.195 DEBUG:boto:Token: None 2023-05-24T21:28:38.196 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:28:38 GMT /qskrps-49/obj8 2023-05-24T21:28:38.196 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:0DbSqR0yG/Q37J3ZqfoM8+UcS34= 2023-05-24T21:28:38.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:0DbSqR0yG/Q37J3ZqfoM8+UcS34=', 'Content-Length': '0'} 2023-05-24T21:28:38.198 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000675385fd9bb7260-00646e8186-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:38.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-05-24T21:28:38.199 DEBUG:boto:path=/qskrps-49/obj8 2023-05-24T21:28:38.200 DEBUG:boto:auth_path=/qskrps-49/obj8 2023-05-24T21:28:38.200 DEBUG:boto:Method: GET 2023-05-24T21:28:38.200 DEBUG:boto:Path: /qskrps-49/obj8 2023-05-24T21:28:38.201 DEBUG:boto:Data: 2023-05-24T21:28:38.201 DEBUG:boto:Headers: {} 2023-05-24T21:28:38.202 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:38.202 DEBUG:boto:Port: 8001 2023-05-24T21:28:38.202 DEBUG:boto:Params: {} 2023-05-24T21:28:38.203 DEBUG:boto:Token: None 2023-05-24T21:28:38.203 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:38 GMT /qskrps-49/obj8 2023-05-24T21:28:38.204 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:IpK5BnZMzPrkRiK5RUFFdWTEO+M= 2023-05-24T21:28:38.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:IpK5BnZMzPrkRiK5RUFFdWTEO+M=', 'Content-Length': '0'} 2023-05-24T21:28:38.206 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bcfcd86303d404a8-00646e8186-1625-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:38.206 DEBUG:boto:path=/qskrps-49/obj8 2023-05-24T21:28:38.207 DEBUG:boto:auth_path=/qskrps-49/obj8 2023-05-24T21:28:38.207 DEBUG:boto:Method: GET 2023-05-24T21:28:38.207 DEBUG:boto:Path: /qskrps-49/obj8 2023-05-24T21:28:38.208 DEBUG:boto:Data: 2023-05-24T21:28:38.208 DEBUG:boto:Headers: {} 2023-05-24T21:28:38.209 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:38.209 DEBUG:boto:Port: 8000 2023-05-24T21:28:38.209 DEBUG:boto:Params: {} 2023-05-24T21:28:38.210 DEBUG:boto:Token: None 2023-05-24T21:28:38.210 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:38 GMT /qskrps-49/obj8 2023-05-24T21:28:38.211 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:IpK5BnZMzPrkRiK5RUFFdWTEO+M= 2023-05-24T21:28:38.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:IpK5BnZMzPrkRiK5RUFFdWTEO+M=', 'Content-Length': '0'} 2023-05-24T21:28:38.212 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cac2c9c4c99140c4-00646e8186-1627-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:28:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:38.256 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:38.257 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:38.257 DEBUG:boto:path=/qskrps-49/?prefix=obj8 2023-05-24T21:28:38.257 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj8 2023-05-24T21:28:38.258 DEBUG:boto:Method: GET 2023-05-24T21:28:38.258 DEBUG:boto:Path: /qskrps-49/?prefix=obj8 2023-05-24T21:28:38.259 DEBUG:boto:Data: 2023-05-24T21:28:38.259 DEBUG:boto:Headers: {} 2023-05-24T21:28:38.259 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:38.260 DEBUG:boto:Port: 8001 2023-05-24T21:28:38.260 DEBUG:boto:Params: {} 2023-05-24T21:28:38.260 DEBUG:boto:Token: None 2023-05-24T21:28:38.261 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:38 GMT /qskrps-49/ 2023-05-24T21:28:38.261 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:d3k2gXaWyo3VP8MFgxwaBg4JBqU= 2023-05-24T21:28:38.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:d3k2gXaWyo3VP8MFgxwaBg4JBqU=', 'Content-Length': '0'} 2023-05-24T21:28:38.264 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1e0b56ca7ce5ea1-00646e8186-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:38.265 DEBUG:boto:b'qskrps-49obj81000falseobj82023-05-24T21:28:09.781Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:38.265 DEBUG:boto:path=/qskrps-49/ 2023-05-24T21:28:38.266 DEBUG:boto:auth_path=/qskrps-49/ 2023-05-24T21:28:38.266 DEBUG:boto:path=/qskrps-49/?prefix=obj8 2023-05-24T21:28:38.266 DEBUG:boto:auth_path=/qskrps-49/?prefix=obj8 2023-05-24T21:28:38.267 DEBUG:boto:Method: GET 2023-05-24T21:28:38.267 DEBUG:boto:Path: /qskrps-49/?prefix=obj8 2023-05-24T21:28:38.267 DEBUG:boto:Data: 2023-05-24T21:28:38.268 DEBUG:boto:Headers: {} 2023-05-24T21:28:38.268 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:38.269 DEBUG:boto:Port: 8000 2023-05-24T21:28:38.269 DEBUG:boto:Params: {} 2023-05-24T21:28:38.269 DEBUG:boto:Token: None 2023-05-24T21:28:38.270 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:38 GMT /qskrps-49/ 2023-05-24T21:28:38.270 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:d3k2gXaWyo3VP8MFgxwaBg4JBqU= 2023-05-24T21:28:38.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:38 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:d3k2gXaWyo3VP8MFgxwaBg4JBqU=', 'Content-Length': '0'} 2023-05-24T21:28:38.273 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf3a92c5e00d2449-00646e8186-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:38.273 DEBUG:boto:b'qskrps-49obj81000falseobj82023-05-24T21:28:09.781Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:28:38.274 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qskrps-49 zones={a2, a1} 2023-05-24T21:28:38.274 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-05-24T21:28:38.275 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:28:38.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:28:38.275 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:28:38.323 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:28:38.324 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:28:38.392 INFO:teuthology.orchestra.run.smithi071.stdout:[ 2023-05-24T21:28:38.392 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:28:38.392 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000007274", 2023-05-24T21:28:38.393 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:28:10.071170Z" 2023-05-24T21:28:38.393 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:38.393 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:28:38.393 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000006576", 2023-05-24T21:28:38.393 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:28:17.787190Z" 2023-05-24T21:28:38.394 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:38.394 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:28:38.394 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000004595", 2023-05-24T21:28:38.394 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:28:17.843765Z" 2023-05-24T21:28:38.394 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:38.394 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:28:38.394 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000011076", 2023-05-24T21:28:38.395 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:28:17.836135Z" 2023-05-24T21:28:38.395 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:28:38.395 INFO:teuthology.orchestra.run.smithi071.stdout:] 2023-05-24T21:28:38.396 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000007274', 1: '00000000000000000000:00000000000000006576', 2: '00000000000000000000:00000000000000004595', 3: '00000000000000000000:00000000000000011076'} 2023-05-24T21:28:38.396 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-05-24T21:28:38.397 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:28:38.397 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:28:38.398 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:28:38.447 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:28:38.447 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:28:38.519 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:28:38.520 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:28:38.520 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:28:38.520 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:28:38.520 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:28:38.520 INFO:teuthology.orchestra.run.smithi071.stdout: "instance_id": 13190874283549403954 2023-05-24T21:28:38.521 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:38.521 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:28:38.521 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:28:38.521 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:28:38.521 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:28:38.521 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:28:38.522 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000007274", 2023-05-24T21:28:38.522 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:28:38.522 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:28:38.522 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:28:38.522 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:28:10.071170Z" 2023-05-24T21:28:38.522 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:28:38.523 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:38.523 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:28:38.523 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:28:38.523 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:28:38.524 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:28:38.524 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000006576", 2023-05-24T21:28:38.524 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:28:38.524 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:28:38.524 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:28:38.524 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:28:17.787190Z" 2023-05-24T21:28:38.525 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:28:38.525 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:38.525 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:28:38.525 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:28:38.525 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:28:38.525 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:28:38.526 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000004595", 2023-05-24T21:28:38.526 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:28:38.526 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:28:38.526 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:28:38.526 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:28:17.843765Z" 2023-05-24T21:28:38.526 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:28:38.526 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:38.527 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:28:38.527 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:28:38.527 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:28:38.527 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:28:38.527 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000011076", 2023-05-24T21:28:38.527 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:28:38.527 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:28:38.528 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:28:38.528 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:28:17.836135Z" 2023-05-24T21:28:38.528 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:28:38.528 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:28:38.528 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:28:38.528 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:38.528 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:28:38.529 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 0, 2023-05-24T21:28:38.529 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 0 2023-05-24T21:28:38.529 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:38.529 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:28:38Z" 2023-05-24T21:28:38.530 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:28:38.531 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13190874283549403954 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007274", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:28:10.071170Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006576", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:28:17.787190Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004595", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:28:17.843765Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011076", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:28:17.836135Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-05-24T21:28:38Z" } 2023-05-24T21:28:38.531 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007274', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:28:10.071170Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006576', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:28:17.787190Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004595', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:28:17.843765Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011076', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:28:17.836135Z'}}] 2023-05-24T21:28:38.532 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000007274', 1: '00000000000000000000:00000000000000006576', 2: '00000000000000000000:00000000000000004595', 3: '00000000000000000000:00000000000000011076'} 2023-05-24T21:28:38.532 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000007274', 1: '00000000000000000000:00000000000000006576', 2: '00000000000000000000:00000000000000004595', 3: '00000000000000000000:00000000000000011076'} 2023-05-24T21:28:38.532 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-05-24T21:28:38.533 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-05-24T21:28:38.533 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:28:38.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:28:38.534 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:28:38.582 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:28:38.582 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:28:38.647 INFO:teuthology.orchestra.run.smithi071.stdout:[ 2023-05-24T21:28:38.648 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:28:38.648 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000005020", 2023-05-24T21:28:38.648 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:28:29.282631Z" 2023-05-24T21:28:38.648 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:38.649 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:28:38.649 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000004597", 2023-05-24T21:28:38.649 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:28:29.284232Z" 2023-05-24T21:28:38.649 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:38.649 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:28:38.649 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000003185", 2023-05-24T21:28:38.650 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:28:29.286006Z" 2023-05-24T21:28:38.650 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:38.650 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:28:38.650 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000008961", 2023-05-24T21:28:38.650 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:28:29.290546Z" 2023-05-24T21:28:38.650 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:28:38.651 INFO:teuthology.orchestra.run.smithi071.stdout:] 2023-05-24T21:28:38.652 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000005020', 1: '00000000000000000000:00000000000000004597', 2: '00000000000000000000:00000000000000003185', 3: '00000000000000000000:00000000000000008961'} 2023-05-24T21:28:38.652 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-05-24T21:28:38.653 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:28:38.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:28:38.653 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:28:38.704 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:28:38.704 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:28:38.767 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:28:38.768 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:28:38.768 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:28:38.768 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:28:38.768 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:28:38.768 INFO:teuthology.orchestra.run.smithi071.stdout: "instance_id": 6585121019900467362 2023-05-24T21:28:38.769 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:38.769 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:28:38.769 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:28:38.769 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:28:38.769 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:28:38.769 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:28:38.769 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000004174", 2023-05-24T21:28:38.770 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:28:38.770 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:28:38.770 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:28:38.770 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:28:07.496880Z" 2023-05-24T21:28:38.770 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:28:38.770 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:38.771 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:28:38.771 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:28:38.771 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:28:38.771 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:28:38.771 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000003749", 2023-05-24T21:28:38.771 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:28:38.771 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:28:38.772 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:28:38.772 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:28:07.498036Z" 2023-05-24T21:28:38.772 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:28:38.772 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:38.772 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:28:38.772 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:28:38.772 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:28:38.773 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:28:38.773 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000002621", 2023-05-24T21:28:38.773 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:28:38.773 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:28:38.773 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:28:38.773 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:28:07.494266Z" 2023-05-24T21:28:38.773 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:28:38.774 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:38.774 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:28:38.774 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:28:38.774 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:28:38.774 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:28:38.774 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000008961", 2023-05-24T21:28:38.774 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:28:38.775 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:28:38.775 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:28:38.775 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:28:29.290546Z" 2023-05-24T21:28:38.775 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:28:38.775 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:28:38.775 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:28:38.775 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:38.776 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:28:38.776 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 0, 2023-05-24T21:28:38.776 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 0 2023-05-24T21:28:38.776 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:38.776 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:28:38Z" 2023-05-24T21:28:38.776 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:28:38.777 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6585121019900467362 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004174", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:28:07.496880Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003749", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:28:07.498036Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002621", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:28:07.494266Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008961", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:28:29.290546Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-05-24T21:28:38Z" } 2023-05-24T21:28:38.778 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004174', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:28:07.496880Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003749', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:28:07.498036Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002621', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:28:07.494266Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008961', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:28:29.290546Z'}}] 2023-05-24T21:28:38.778 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000005020', 1: '00000000000000000000:00000000000000004597', 2: '00000000000000000000:00000000000000003185', 3: '00000000000000000000:00000000000000008961'} 2023-05-24T21:28:38.779 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000004174', 1: '00000000000000000000:00000000000000003749', 2: '00000000000000000000:00000000000000002621', 3: '00000000000000000000:00000000000000008961'} 2023-05-24T21:28:38.779 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=0 master=00000000000000000000:00000000000000005020 target=00000000000000000000:00000000000000004174, shard=1 master=00000000000000000000:00000000000000004597 target=00000000000000000000:00000000000000003749, shard=2 master=00000000000000000000:00000000000000003185 target=00000000000000000000:00000000000000002621 2023-05-24T21:28:43.780 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:28:43.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:28:43.782 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:28:43.840 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:28:43.840 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:28:43.915 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:28:43.915 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:28:43.916 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:28:43.916 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:28:43.916 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:28:43.916 INFO:teuthology.orchestra.run.smithi071.stdout: "instance_id": 6585121019900467362 2023-05-24T21:28:43.916 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:43.916 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:28:43.917 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:28:43.917 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:28:43.917 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:28:43.917 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:28:43.917 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000005020", 2023-05-24T21:28:43.917 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:28:43.918 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:28:43.918 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:28:43.918 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:28:29.282631Z" 2023-05-24T21:28:43.918 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:28:43.918 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:43.918 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:28:43.918 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:28:43.919 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:28:43.919 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:28:43.919 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000004597", 2023-05-24T21:28:43.919 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:28:43.919 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:28:43.919 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:28:43.920 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:28:29.284232Z" 2023-05-24T21:28:43.920 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:28:43.920 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:43.920 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:28:43.920 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:28:43.921 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:28:43.921 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:28:43.921 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000003185", 2023-05-24T21:28:43.921 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:28:43.921 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:28:43.921 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:28:43.922 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:28:29.286006Z" 2023-05-24T21:28:43.922 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:28:43.922 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:43.922 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:28:43.922 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:28:43.922 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:28:43.922 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "incremental-sync", 2023-05-24T21:28:43.923 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "00000000000000000000:00000000000000008961", 2023-05-24T21:28:43.923 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:28:43.923 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:28:43.923 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:28:43.923 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:28:29.290546Z" 2023-05-24T21:28:43.923 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:28:43.923 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:28:43.924 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:28:43.924 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:43.924 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:28:43.924 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 0, 2023-05-24T21:28:43.924 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 0 2023-05-24T21:28:43.924 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:43.924 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:28:43Z" 2023-05-24T21:28:43.925 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:28:43.926 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6585121019900467362 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005020", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:28:29.282631Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004597", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:28:29.284232Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003185", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:28:29.286006Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008961", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:28:29.290546Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-05-24T21:28:43Z" } 2023-05-24T21:28:43.926 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005020', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:28:29.282631Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004597', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:28:29.284232Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003185', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:28:29.286006Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008961', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:28:29.290546Z'}}] 2023-05-24T21:28:43.927 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000005020', 1: '00000000000000000000:00000000000000004597', 2: '00000000000000000000:00000000000000003185', 3: '00000000000000000000:00000000000000008961'} 2023-05-24T21:28:43.927 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000005020', 1: '00000000000000000000:00000000000000004597', 2: '00000000000000000000:00000000000000003185', 3: '00000000000000000000:00000000000000008961'} 2023-05-24T21:28:43.927 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-05-24T21:28:43.928 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable_enable ... ok 2023-05-24T21:28:43.929 INFO:rgw_multi.tests:create bucket zone=a1 name=qskrps-50 2023-05-24T21:28:43.930 DEBUG:boto:path=/qskrps-50/ 2023-05-24T21:28:43.930 DEBUG:boto:auth_path=/qskrps-50/ 2023-05-24T21:28:43.930 DEBUG:boto:Method: PUT 2023-05-24T21:28:43.931 DEBUG:boto:Path: /qskrps-50/ 2023-05-24T21:28:43.931 DEBUG:boto:Data: 2023-05-24T21:28:43.931 DEBUG:boto:Headers: {} 2023-05-24T21:28:43.932 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:43.932 DEBUG:boto:Port: 8000 2023-05-24T21:28:43.932 DEBUG:boto:Params: {} 2023-05-24T21:28:43.933 DEBUG:boto:Token: None 2023-05-24T21:28:43.933 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:28:43 GMT /qskrps-50/ 2023-05-24T21:28:43.934 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:m3mFj1gCPUb6jl0YKG2Hes0NUUY= 2023-05-24T21:28:43.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:43 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:m3mFj1gCPUb6jl0YKG2Hes0NUUY=', 'Content-Length': '0'} 2023-05-24T21:28:43.960 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009328a1a054763c62-00646e818b-1627-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:28:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:43.961 INFO:rgw_multi.tests:create bucket zone=a2 name=qskrps-51 2023-05-24T21:28:43.961 DEBUG:boto:path=/qskrps-51/ 2023-05-24T21:28:43.962 DEBUG:boto:auth_path=/qskrps-51/ 2023-05-24T21:28:43.962 DEBUG:boto:Method: PUT 2023-05-24T21:28:43.962 DEBUG:boto:Path: /qskrps-51/ 2023-05-24T21:28:43.963 DEBUG:boto:Data: 2023-05-24T21:28:43.963 DEBUG:boto:Headers: {} 2023-05-24T21:28:43.963 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:28:43.964 DEBUG:boto:Port: 8001 2023-05-24T21:28:43.964 DEBUG:boto:Params: {} 2023-05-24T21:28:43.964 DEBUG:boto:Token: None 2023-05-24T21:28:43.965 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:28:43 GMT /qskrps-51/ 2023-05-24T21:28:43.965 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:lBzjoWngf585VJOoKIWjOE9DtSc= 2023-05-24T21:28:43.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:43 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:lBzjoWngf585VJOoKIWjOE9DtSc=', 'Content-Length': '0'} 2023-05-24T21:28:44.003 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000082ceddc82b4bc587-00646e818b-1625-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:28:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:44.004 DEBUG:boto:path=/qskrps-50/MULTIPART 2023-05-24T21:28:44.004 DEBUG:boto:auth_path=/qskrps-50/MULTIPART 2023-05-24T21:28:44.005 DEBUG:boto:path=/qskrps-50/MULTIPART?uploads 2023-05-24T21:28:44.005 DEBUG:boto:auth_path=/qskrps-50/MULTIPART?uploads 2023-05-24T21:28:44.005 DEBUG:boto:Method: POST 2023-05-24T21:28:44.006 DEBUG:boto:Path: /qskrps-50/MULTIPART?uploads 2023-05-24T21:28:44.006 DEBUG:boto:Data: 2023-05-24T21:28:44.006 DEBUG:boto:Headers: {} 2023-05-24T21:28:44.007 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:44.007 DEBUG:boto:Port: 8000 2023-05-24T21:28:44.007 DEBUG:boto:Params: {} 2023-05-24T21:28:44.008 DEBUG:boto:Token: None 2023-05-24T21:28:44.008 DEBUG:boto:StringToSign: POST Wed, 24 May 2023 21:28:44 GMT /qskrps-50/MULTIPART?uploads 2023-05-24T21:28:44.008 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:3hhHfItdBo8ksy8st6AmWF0wtvM= 2023-05-24T21:28:44.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:44 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:3hhHfItdBo8ksy8st6AmWF0wtvM=', 'Content-Length': '0'} 2023-05-24T21:28:46.054 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000dbba8f1f7b2def7a-00646e818c-1627-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '249'), ('Date', 'Wed, 24 May 2023 21:28:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:46.055 DEBUG:boto:b'qskrps-50MULTIPART2~folYx6DFokH7KL_Zv1-78LP-DsDDzvr' 2023-05-24T21:28:46.096 DEBUG:boto:path=/qskrps-50/MULTIPART 2023-05-24T21:28:46.130 DEBUG:boto:auth_path=/qskrps-50/MULTIPART 2023-05-24T21:28:46.130 DEBUG:boto:path=/qskrps-50/MULTIPART?uploadId=2~folYx6DFokH7KL_Zv1-78LP-DsDDzvr&partNumber=1 2023-05-24T21:28:46.131 DEBUG:boto:auth_path=/qskrps-50/MULTIPART?uploadId=2~folYx6DFokH7KL_Zv1-78LP-DsDDzvr&partNumber=1 2023-05-24T21:28:46.131 DEBUG:boto:Method: PUT 2023-05-24T21:28:46.131 DEBUG:boto:Path: /qskrps-50/MULTIPART?uploadId=2~folYx6DFokH7KL_Zv1-78LP-DsDDzvr&partNumber=1 2023-05-24T21:28:46.132 DEBUG:boto:Data: 2023-05-24T21:28:46.132 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-05-24T21:28:46.132 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:46.133 DEBUG:boto:Port: 8000 2023-05-24T21:28:46.133 DEBUG:boto:Params: {} 2023-05-24T21:28:46.133 DEBUG:boto:Token: None 2023-05-24T21:28:46.134 DEBUG:boto:StringToSign: PUT ebKBBg0ze5srhMzzkK3PdA== application/octet-stream Wed, 24 May 2023 21:28:46 GMT /qskrps-50/MULTIPART?partNumber=1&uploadId=2~folYx6DFokH7KL_Zv1-78LP-DsDDzvr 2023-05-24T21:28:46.134 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:NZU7b8yp9d627//66DxRUEmqfE4= 2023-05-24T21:28:46.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:28:46 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:NZU7b8yp9d627//66DxRUEmqfE4='} 2023-05-24T21:28:46.181 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"79b281060d337b9b2b84ccf390adcf74"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002c1f2d34b5259f3a-00646e818e-1627-a1'), ('Date', 'Wed, 24 May 2023 21:28:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:46.217 DEBUG:boto:path=/qskrps-50/MULTIPART 2023-05-24T21:28:46.217 DEBUG:boto:auth_path=/qskrps-50/MULTIPART 2023-05-24T21:28:46.218 DEBUG:boto:path=/qskrps-50/MULTIPART?uploadId=2~folYx6DFokH7KL_Zv1-78LP-DsDDzvr&partNumber=2 2023-05-24T21:28:46.218 DEBUG:boto:auth_path=/qskrps-50/MULTIPART?uploadId=2~folYx6DFokH7KL_Zv1-78LP-DsDDzvr&partNumber=2 2023-05-24T21:28:46.218 DEBUG:boto:Method: PUT 2023-05-24T21:28:46.219 DEBUG:boto:Path: /qskrps-50/MULTIPART?uploadId=2~folYx6DFokH7KL_Zv1-78LP-DsDDzvr&partNumber=2 2023-05-24T21:28:46.219 DEBUG:boto:Data: 2023-05-24T21:28:46.220 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-05-24T21:28:46.220 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:46.220 DEBUG:boto:Port: 8000 2023-05-24T21:28:46.221 DEBUG:boto:Params: {} 2023-05-24T21:28:46.221 DEBUG:boto:Token: None 2023-05-24T21:28:46.221 DEBUG:boto:StringToSign: PUT dIQ6OrGTo4m87YmUAtmdXw== application/octet-stream Wed, 24 May 2023 21:28:46 GMT /qskrps-50/MULTIPART?partNumber=2&uploadId=2~folYx6DFokH7KL_Zv1-78LP-DsDDzvr 2023-05-24T21:28:46.222 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:J7Bge+3WngKr2Y3s+7h9Sp2YBJk= 2023-05-24T21:28:46.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:28:46 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:J7Bge+3WngKr2Y3s+7h9Sp2YBJk='} 2023-05-24T21:28:46.268 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"74843a3ab193a389bced899402d99d5f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c3b662a26734c210-00646e818e-1627-a1'), ('Date', 'Wed, 24 May 2023 21:28:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:46.286 DEBUG:boto:path=/qskrps-50/MULTIPART 2023-05-24T21:28:46.286 DEBUG:boto:auth_path=/qskrps-50/MULTIPART 2023-05-24T21:28:46.287 DEBUG:boto:path=/qskrps-50/MULTIPART?uploadId=2~folYx6DFokH7KL_Zv1-78LP-DsDDzvr&partNumber=3 2023-05-24T21:28:46.287 DEBUG:boto:auth_path=/qskrps-50/MULTIPART?uploadId=2~folYx6DFokH7KL_Zv1-78LP-DsDDzvr&partNumber=3 2023-05-24T21:28:46.287 DEBUG:boto:Method: PUT 2023-05-24T21:28:46.288 DEBUG:boto:Path: /qskrps-50/MULTIPART?uploadId=2~folYx6DFokH7KL_Zv1-78LP-DsDDzvr&partNumber=3 2023-05-24T21:28:46.288 DEBUG:boto:Data: 2023-05-24T21:28:46.288 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-05-24T21:28:46.289 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:46.289 DEBUG:boto:Port: 8000 2023-05-24T21:28:46.289 DEBUG:boto:Params: {} 2023-05-24T21:28:46.290 DEBUG:boto:Token: None 2023-05-24T21:28:46.290 DEBUG:boto:StringToSign: PUT e4RW4edMN49Fhh9TYZ51tg== application/octet-stream Wed, 24 May 2023 21:28:46 GMT /qskrps-50/MULTIPART?partNumber=3&uploadId=2~folYx6DFokH7KL_Zv1-78LP-DsDDzvr 2023-05-24T21:28:46.291 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:UDGqSSjf+/2VbfeL9/hKZQNEp0Y= 2023-05-24T21:28:46.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:28:46 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:UDGqSSjf+/2VbfeL9/hKZQNEp0Y='} 2023-05-24T21:28:46.332 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"7b8456e1e74c378f45861f53619e75b6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e92ffbd1f5bf0207-00646e818e-1627-a1'), ('Date', 'Wed, 24 May 2023 21:28:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:46.350 DEBUG:boto:path=/qskrps-50/MULTIPART 2023-05-24T21:28:46.350 DEBUG:boto:auth_path=/qskrps-50/MULTIPART 2023-05-24T21:28:46.351 DEBUG:boto:path=/qskrps-50/MULTIPART?uploadId=2~folYx6DFokH7KL_Zv1-78LP-DsDDzvr&partNumber=4 2023-05-24T21:28:46.351 DEBUG:boto:auth_path=/qskrps-50/MULTIPART?uploadId=2~folYx6DFokH7KL_Zv1-78LP-DsDDzvr&partNumber=4 2023-05-24T21:28:46.351 DEBUG:boto:Method: PUT 2023-05-24T21:28:46.352 DEBUG:boto:Path: /qskrps-50/MULTIPART?uploadId=2~folYx6DFokH7KL_Zv1-78LP-DsDDzvr&partNumber=4 2023-05-24T21:28:46.352 DEBUG:boto:Data: 2023-05-24T21:28:46.352 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-05-24T21:28:46.353 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:46.353 DEBUG:boto:Port: 8000 2023-05-24T21:28:46.353 DEBUG:boto:Params: {} 2023-05-24T21:28:46.354 DEBUG:boto:Token: None 2023-05-24T21:28:46.354 DEBUG:boto:StringToSign: PUT l+jDi4yPcIlVvEiQwDQZTQ== application/octet-stream Wed, 24 May 2023 21:28:46 GMT /qskrps-50/MULTIPART?partNumber=4&uploadId=2~folYx6DFokH7KL_Zv1-78LP-DsDDzvr 2023-05-24T21:28:46.354 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:8LwWSr8bnC25obWh5zZGUAmoxG4= 2023-05-24T21:28:46.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:28:46 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:8LwWSr8bnC25obWh5zZGUAmoxG4='} 2023-05-24T21:28:46.393 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"97e8c38b8c8f708955bc4890c034194d"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000329ab4d7cb4385a4-00646e818e-1627-a1'), ('Date', 'Wed, 24 May 2023 21:28:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:46.394 DEBUG:boto:path=/qskrps-50/MULTIPART 2023-05-24T21:28:46.394 DEBUG:boto:auth_path=/qskrps-50/MULTIPART 2023-05-24T21:28:46.395 DEBUG:boto:path=/qskrps-50/MULTIPART?uploadId=2~folYx6DFokH7KL_Zv1-78LP-DsDDzvr 2023-05-24T21:28:46.395 DEBUG:boto:auth_path=/qskrps-50/MULTIPART?uploadId=2~folYx6DFokH7KL_Zv1-78LP-DsDDzvr 2023-05-24T21:28:46.395 DEBUG:boto:Method: GET 2023-05-24T21:28:46.396 DEBUG:boto:Path: /qskrps-50/MULTIPART?uploadId=2~folYx6DFokH7KL_Zv1-78LP-DsDDzvr 2023-05-24T21:28:46.396 DEBUG:boto:Data: 2023-05-24T21:28:46.396 DEBUG:boto:Headers: {} 2023-05-24T21:28:46.397 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:46.397 DEBUG:boto:Port: 8000 2023-05-24T21:28:46.397 DEBUG:boto:Params: {} 2023-05-24T21:28:46.398 DEBUG:boto:Token: None 2023-05-24T21:28:46.398 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:28:46 GMT /qskrps-50/MULTIPART?uploadId=2~folYx6DFokH7KL_Zv1-78LP-DsDDzvr 2023-05-24T21:28:46.399 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:P5+DOHQlvOQs4C7pzO5td5ylq80= 2023-05-24T21:28:46.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:46 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:P5+DOHQlvOQs4C7pzO5td5ylq80=', 'Content-Length': '0'} 2023-05-24T21:28:46.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba445c65530d7217-00646e818e-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:28:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:46.444 DEBUG:boto:path=/qskrps-50/MULTIPART 2023-05-24T21:28:46.445 DEBUG:boto:auth_path=/qskrps-50/MULTIPART 2023-05-24T21:28:46.445 DEBUG:boto:path=/qskrps-50/MULTIPART?uploadId=2~folYx6DFokH7KL_Zv1-78LP-DsDDzvr 2023-05-24T21:28:46.445 DEBUG:boto:auth_path=/qskrps-50/MULTIPART?uploadId=2~folYx6DFokH7KL_Zv1-78LP-DsDDzvr 2023-05-24T21:28:46.446 DEBUG:boto:Method: POST 2023-05-24T21:28:46.446 DEBUG:boto:Path: /qskrps-50/MULTIPART?uploadId=2~folYx6DFokH7KL_Zv1-78LP-DsDDzvr 2023-05-24T21:28:46.446 DEBUG:boto:Data: 1 "79b281060d337b9b2b84ccf390adcf74" 2 "74843a3ab193a389bced899402d99d5f" 3 "7b8456e1e74c378f45861f53619e75b6" 4 "97e8c38b8c8f708955bc4890c034194d" 2023-05-24T21:28:46.447 DEBUG:boto:Headers: {'Content-Type': 'text/xml'} 2023-05-24T21:28:46.447 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:28:46.447 DEBUG:boto:Port: 8000 2023-05-24T21:28:46.448 DEBUG:boto:Params: {} 2023-05-24T21:28:46.448 DEBUG:boto:Token: None 2023-05-24T21:28:46.449 DEBUG:boto:StringToSign: POST text/xml Wed, 24 May 2023 21:28:46 GMT /qskrps-50/MULTIPART?uploadId=2~folYx6DFokH7KL_Zv1-78LP-DsDDzvr 2023-05-24T21:28:46.449 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:Jzs0ze/uQVgIeKhMgvzeDRMkytM= 2023-05-24T21:28:46.449 DEBUG:boto:Final headers: {'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:28:46 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Jzs0ze/uQVgIeKhMgvzeDRMkytM=', 'Content-Length': '460'} 2023-05-24T21:28:46.461 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b5b2a89371eafcb3-00646e818e-1627-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '291'), ('Date', 'Wed, 24 May 2023 21:28:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:28:46.461 DEBUG:boto:http://smithi071.front.sepia.ceph.com:8000/qskrps-50/MULTIPARTqskrps-50MULTIPART 2023-05-24T21:28:46.462 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:28:46.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:28:46.463 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:28:46.511 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:28:46.511 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:28:46.542 INFO:teuthology.orchestra.run.smithi071.stderr:No --period given, using current period=5c04936b-70ea-4415-b4eb-3aad3dc612e5 2023-05-24T21:28:46.554 INFO:teuthology.orchestra.run.smithi071.stdout:[ 2023-05-24T21:28:46.555 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:28:46.555 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963723.986757_570.1", 2023-05-24T21:28:46.555 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:28:43.986757Z" 2023-05-24T21:28:46.555 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:46.555 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:28:46.555 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:28:46.556 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:06:32.333844Z" 2023-05-24T21:28:46.556 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:46.556 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:28:46.556 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:28:46.556 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:28:46.556 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:46.557 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:28:46.557 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:28:46.557 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:28:46.557 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:28:46.557 INFO:teuthology.orchestra.run.smithi071.stdout:] 2023-05-24T21:28:46.558 DEBUG:rgw_multi.tests:master meta markers={0: '1_1684963723.986757_570.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:28:46.559 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-24T21:28:46.559 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:28:46.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:28:46.560 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:28:46.609 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:28:46.609 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:28:46.649 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:28:46.643+0000 7f230ec70780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:28:46.649 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:28:46.644+0000 7f230ec70780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:28:46.662 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:28:46.662 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:28:46.663 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:28:46.663 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:28:46.663 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:28:46.663 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:28:46.663 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:28:46.663 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:46.664 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:28:46.664 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:28:46.664 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:28:46.664 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:28:46.664 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:28:46.664 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963723.986757_570.1", 2023-05-24T21:28:46.664 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:28:46.665 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:28:46.665 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:28:46.665 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:28:43.986757Z", 2023-05-24T21:28:46.665 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:28:46.665 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:28:46.665 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:46.665 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:28:46.666 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:28:46.666 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:28:46.666 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:28:46.666 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:28:46.666 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:28:46.666 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:28:46.666 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:28:46.667 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:28:46.667 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:28:46.667 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:28:46.667 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:46.667 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:28:46.667 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:28:46.667 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:28:46.668 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:28:46.668 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:28:46.668 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:28:46.668 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:28:46.668 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:28:46.668 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:28:46.669 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:28:46.669 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:28:46.669 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:46.669 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:28:46.669 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:28:46.669 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:28:46.669 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:28:46.670 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:28:46.670 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:28:46.670 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:28:46.670 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:28:46.670 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:28:46.670 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:28:46.670 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:28:46.671 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:28:46.671 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:28:46.671 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:46.671 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:28:46.671 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:28:46.671 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:28:46.672 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:28:46.672 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:28:46Z" 2023-05-24T21:28:46.672 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:28:46.673 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684963723.986757_570.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:28:43.986757Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:28:46Z" } 2023-05-24T21:28:46.673 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963723.986757_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:28:43.986757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:28:46.674 DEBUG:rgw_multi.tests:log_status={0: '1_1684963723.986757_570.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:28:46.674 DEBUG:rgw_multi.tests:sync_status={0: '1_1684963723.986757_570.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:28:46.675 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-24T21:28:46.675 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=qskrps-50 2023-05-24T21:28:46.676 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-50', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:28:46.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-50', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:28:46.677 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-50 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:28:46.732 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:28:46.733 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:28:46.775 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:28:46.770+0000 7ff2cd1dc780 1 waiting to reach incremental sync.. 2023-05-24T21:28:51.776 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:28:51.771+0000 7ff2cd1dc780 1 waiting to reach incremental sync.. 2023-05-24T21:28:56.778 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:28:56.772+0000 7ff2cd1dc780 1 bucket sync caught up with source: 2023-05-24T21:28:56.779 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000011.34.6, , , , , ] 2023-05-24T21:28:56.779 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000011.34.6, , , , , ] 2023-05-24T21:28:56.779 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:28:56.772+0000 7ff2cd1dc780 0 bucket checkpoint complete 2023-05-24T21:28:56.798 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=qskrps-50 2023-05-24T21:28:56.798 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-50', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:28:56.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-50', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:28:56.799 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-50 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:28:56.862 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:28:56.862 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:28:56.914 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:28:56.909+0000 7f6c61ed0780 1 waiting to reach incremental sync.. 2023-05-24T21:29:01.916 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:29:01.910+0000 7f6c61ed0780 1 bucket sync caught up with source: 2023-05-24T21:29:01.916 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000001.532.6, , , , , ] 2023-05-24T21:29:01.917 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000001.532.6, , , , , ] 2023-05-24T21:29:01.917 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:29:01.910+0000 7f6c61ed0780 0 bucket checkpoint complete 2023-05-24T21:29:01.935 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qskrps-50 zones={a2, a1} 2023-05-24T21:29:01.936 DEBUG:boto:path=/qskrps-50/ 2023-05-24T21:29:01.936 DEBUG:boto:auth_path=/qskrps-50/ 2023-05-24T21:29:01.936 DEBUG:boto:Method: HEAD 2023-05-24T21:29:01.937 DEBUG:boto:Path: /qskrps-50/ 2023-05-24T21:29:01.937 DEBUG:boto:Data: 2023-05-24T21:29:01.938 DEBUG:boto:Headers: {} 2023-05-24T21:29:01.938 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:29:01.938 DEBUG:boto:Port: 8001 2023-05-24T21:29:01.939 DEBUG:boto:Params: {} 2023-05-24T21:29:01.939 DEBUG:boto:Token: None 2023-05-24T21:29:01.939 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:29:01 GMT /qskrps-50/ 2023-05-24T21:29:01.940 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:CBqavOuX7LF3O53tj2hl4QdMg8Q= 2023-05-24T21:29:01.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:29:01 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:CBqavOuX7LF3O53tj2hl4QdMg8Q=', 'Content-Length': '0'} 2023-05-24T21:29:01.943 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '20971520'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cbcbe9b7c18dd712-00646e819d-1625-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:29:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:29:01.943 DEBUG:boto:path=/qskrps-50/ 2023-05-24T21:29:01.944 DEBUG:boto:auth_path=/qskrps-50/ 2023-05-24T21:29:01.944 DEBUG:boto:Method: HEAD 2023-05-24T21:29:01.944 DEBUG:boto:Path: /qskrps-50/ 2023-05-24T21:29:01.945 DEBUG:boto:Data: 2023-05-24T21:29:01.945 DEBUG:boto:Headers: {} 2023-05-24T21:29:01.946 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:29:01.946 DEBUG:boto:Port: 8000 2023-05-24T21:29:01.946 DEBUG:boto:Params: {} 2023-05-24T21:29:01.947 DEBUG:boto:Token: None 2023-05-24T21:29:01.947 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:29:01 GMT /qskrps-50/ 2023-05-24T21:29:01.947 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:CBqavOuX7LF3O53tj2hl4QdMg8Q= 2023-05-24T21:29:01.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:29:01 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:CBqavOuX7LF3O53tj2hl4QdMg8Q=', 'Content-Length': '0'} 2023-05-24T21:29:01.950 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '20971520'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a8a1e2c195d74a98-00646e819d-1627-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:29:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:29:01.950 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-24T21:29:01.951 DEBUG:boto:path=/qskrps-50/ 2023-05-24T21:29:01.951 DEBUG:boto:auth_path=/qskrps-50/ 2023-05-24T21:29:01.951 DEBUG:boto:path=/qskrps-50/?versions&max-keys=999 2023-05-24T21:29:01.952 DEBUG:boto:auth_path=/qskrps-50/?versions&max-keys=999 2023-05-24T21:29:01.952 DEBUG:boto:Method: GET 2023-05-24T21:29:01.952 DEBUG:boto:Path: /qskrps-50/?versions&max-keys=999 2023-05-24T21:29:01.953 DEBUG:boto:Data: 2023-05-24T21:29:01.953 DEBUG:boto:Headers: {} 2023-05-24T21:29:01.953 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:29:01.954 DEBUG:boto:Port: 8001 2023-05-24T21:29:01.954 DEBUG:boto:Params: {} 2023-05-24T21:29:01.955 DEBUG:boto:Token: None 2023-05-24T21:29:01.955 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:29:01 GMT /qskrps-50/?versions 2023-05-24T21:29:01.955 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4n8WTbZkMGvmTrH6EAUxz3jsdEY= 2023-05-24T21:29:01.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:29:01 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4n8WTbZkMGvmTrH6EAUxz3jsdEY=', 'Content-Length': '0'} 2023-05-24T21:29:01.958 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005098746b4fa602d7-00646e819d-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:29:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:29:01.959 DEBUG:boto:b'qskrps-50999falseMULTIPARTnulltrue2023-05-24T21:28:46.448Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:29:01.959 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2023-05-24T21:29:01.960 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-24T21:29:01.960 DEBUG:boto:path=/qskrps-50/ 2023-05-24T21:29:01.960 DEBUG:boto:auth_path=/qskrps-50/ 2023-05-24T21:29:01.961 DEBUG:boto:path=/qskrps-50/?versions&max-keys=999 2023-05-24T21:29:01.961 DEBUG:boto:auth_path=/qskrps-50/?versions&max-keys=999 2023-05-24T21:29:01.962 DEBUG:boto:Method: GET 2023-05-24T21:29:01.962 DEBUG:boto:Path: /qskrps-50/?versions&max-keys=999 2023-05-24T21:29:01.962 DEBUG:boto:Data: 2023-05-24T21:29:01.963 DEBUG:boto:Headers: {} 2023-05-24T21:29:01.963 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:29:01.963 DEBUG:boto:Port: 8000 2023-05-24T21:29:01.964 DEBUG:boto:Params: {} 2023-05-24T21:29:01.964 DEBUG:boto:Token: None 2023-05-24T21:29:01.964 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:29:01 GMT /qskrps-50/?versions 2023-05-24T21:29:01.965 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4n8WTbZkMGvmTrH6EAUxz3jsdEY= 2023-05-24T21:29:01.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:29:01 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4n8WTbZkMGvmTrH6EAUxz3jsdEY=', 'Content-Length': '0'} 2023-05-24T21:29:01.968 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044e26a6c45e95478-00646e819d-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:29:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:29:01.968 DEBUG:boto:b'qskrps-50999falseMULTIPARTnulltrue2023-05-24T21:28:46.448Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:29:01.969 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2023-05-24T21:29:01.969 DEBUG:boto:path=/qskrps-50/ 2023-05-24T21:29:01.970 DEBUG:boto:auth_path=/qskrps-50/ 2023-05-24T21:29:01.970 DEBUG:boto:path=/qskrps-50/?versions&max-keys=999 2023-05-24T21:29:01.970 DEBUG:boto:auth_path=/qskrps-50/?versions&max-keys=999 2023-05-24T21:29:01.971 DEBUG:boto:Method: GET 2023-05-24T21:29:01.971 DEBUG:boto:Path: /qskrps-50/?versions&max-keys=999 2023-05-24T21:29:01.971 DEBUG:boto:Data: 2023-05-24T21:29:01.972 DEBUG:boto:Headers: {} 2023-05-24T21:29:01.972 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:29:01.972 DEBUG:boto:Port: 8001 2023-05-24T21:29:01.973 DEBUG:boto:Params: {} 2023-05-24T21:29:01.973 DEBUG:boto:Token: None 2023-05-24T21:29:01.974 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:29:01 GMT /qskrps-50/?versions 2023-05-24T21:29:01.974 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4n8WTbZkMGvmTrH6EAUxz3jsdEY= 2023-05-24T21:29:01.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:29:01 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4n8WTbZkMGvmTrH6EAUxz3jsdEY=', 'Content-Length': '0'} 2023-05-24T21:29:01.976 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002459645440302ec-00646e819d-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:29:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:29:01.977 DEBUG:boto:b'qskrps-50999falseMULTIPARTnulltrue2023-05-24T21:28:46.448Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:29:01.978 DEBUG:boto:path=/qskrps-50/ 2023-05-24T21:29:01.978 DEBUG:boto:auth_path=/qskrps-50/ 2023-05-24T21:29:01.978 DEBUG:boto:path=/qskrps-50/?versions&max-keys=999 2023-05-24T21:29:01.979 DEBUG:boto:auth_path=/qskrps-50/?versions&max-keys=999 2023-05-24T21:29:01.979 DEBUG:boto:Method: GET 2023-05-24T21:29:01.979 DEBUG:boto:Path: /qskrps-50/?versions&max-keys=999 2023-05-24T21:29:01.980 DEBUG:boto:Data: 2023-05-24T21:29:01.980 DEBUG:boto:Headers: {} 2023-05-24T21:29:01.981 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:29:01.981 DEBUG:boto:Port: 8000 2023-05-24T21:29:01.981 DEBUG:boto:Params: {} 2023-05-24T21:29:01.982 DEBUG:boto:Token: None 2023-05-24T21:29:01.982 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:29:01 GMT /qskrps-50/?versions 2023-05-24T21:29:01.982 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:4n8WTbZkMGvmTrH6EAUxz3jsdEY= 2023-05-24T21:29:01.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:29:01 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4n8WTbZkMGvmTrH6EAUxz3jsdEY=', 'Content-Length': '0'} 2023-05-24T21:29:01.985 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bde8ad4175ed696b-00646e819d-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:29:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:29:01.986 DEBUG:boto:b'qskrps-50999falseMULTIPARTnulltrue2023-05-24T21:28:46.448Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:29:01.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2023-05-24T21:29:01.987 DEBUG:boto:path=/qskrps-50/MULTIPART 2023-05-24T21:29:01.987 DEBUG:boto:auth_path=/qskrps-50/MULTIPART 2023-05-24T21:29:01.987 DEBUG:boto:path=/qskrps-50/MULTIPART?versionId=null 2023-05-24T21:29:01.988 DEBUG:boto:auth_path=/qskrps-50/MULTIPART?versionId=null 2023-05-24T21:29:01.988 DEBUG:boto:Method: GET 2023-05-24T21:29:01.989 DEBUG:boto:Path: /qskrps-50/MULTIPART?versionId=null 2023-05-24T21:29:01.989 DEBUG:boto:Data: 2023-05-24T21:29:01.989 DEBUG:boto:Headers: {} 2023-05-24T21:29:01.990 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:29:01.990 DEBUG:boto:Port: 8001 2023-05-24T21:29:01.990 DEBUG:boto:Params: {} 2023-05-24T21:29:01.991 DEBUG:boto:Token: None 2023-05-24T21:29:01.991 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:29:01 GMT /qskrps-50/MULTIPART?versionId=null 2023-05-24T21:29:01.992 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:OWjzzyGoNUTkrURxIPSiXD7kRGM= 2023-05-24T21:29:01.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:29:01 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:OWjzzyGoNUTkrURxIPSiXD7kRGM=', 'Content-Length': '0'} 2023-05-24T21:29:01.995 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx0000031c789b4ff2623ad-00646e819d-1625-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Wed, 24 May 2023 21:29:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:29:02.065 DEBUG:boto:path=/qskrps-50/MULTIPART 2023-05-24T21:29:02.066 DEBUG:boto:auth_path=/qskrps-50/MULTIPART 2023-05-24T21:29:02.067 DEBUG:boto:path=/qskrps-50/MULTIPART?versionId=null 2023-05-24T21:29:02.067 DEBUG:boto:auth_path=/qskrps-50/MULTIPART?versionId=null 2023-05-24T21:29:02.067 DEBUG:boto:Method: GET 2023-05-24T21:29:02.068 DEBUG:boto:Path: /qskrps-50/MULTIPART?versionId=null 2023-05-24T21:29:02.068 DEBUG:boto:Data: 2023-05-24T21:29:02.068 DEBUG:boto:Headers: {} 2023-05-24T21:29:02.069 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:29:02.069 DEBUG:boto:Port: 8000 2023-05-24T21:29:02.070 DEBUG:boto:Params: {} 2023-05-24T21:29:02.070 DEBUG:boto:Token: None 2023-05-24T21:29:02.070 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:29:02 GMT /qskrps-50/MULTIPART?versionId=null 2023-05-24T21:29:02.071 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:z0RScy+Y/hbzOFfwnnFraIcozjw= 2023-05-24T21:29:02.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:29:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:z0RScy+Y/hbzOFfwnnFraIcozjw=', 'Content-Length': '0'} 2023-05-24T21:29:02.077 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000460b54b18b9e9e40-00646e819e-1627-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Wed, 24 May 2023 21:29:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:29:02.170 DEBUG:boto:path=/qskrps-50/MULTIPART 2023-05-24T21:29:02.171 DEBUG:boto:auth_path=/qskrps-50/MULTIPART 2023-05-24T21:29:02.171 DEBUG:boto:path=/qskrps-50/MULTIPART?versionId=null 2023-05-24T21:29:02.172 DEBUG:boto:auth_path=/qskrps-50/MULTIPART?versionId=null 2023-05-24T21:29:02.172 DEBUG:boto:Method: HEAD 2023-05-24T21:29:02.172 DEBUG:boto:Path: /qskrps-50/MULTIPART?versionId=null 2023-05-24T21:29:02.173 DEBUG:boto:Data: 2023-05-24T21:29:02.173 DEBUG:boto:Headers: {} 2023-05-24T21:29:02.173 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:29:02.174 DEBUG:boto:Port: 8001 2023-05-24T21:29:02.174 DEBUG:boto:Params: {} 2023-05-24T21:29:02.175 DEBUG:boto:Token: None 2023-05-24T21:29:02.175 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:29:02 GMT /qskrps-50/MULTIPART?versionId=null 2023-05-24T21:29:02.176 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:pVSK+G+wHfUXKjHKE9Dsh9E27uU= 2023-05-24T21:29:02.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:29:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:pVSK+G+wHfUXKjHKE9Dsh9E27uU=', 'Content-Length': '0'} 2023-05-24T21:29:02.178 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000e11b22a8f497f73d-00646e819e-1625-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Wed, 24 May 2023 21:29:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:29:02.179 DEBUG:boto:path=/qskrps-50/MULTIPART 2023-05-24T21:29:02.179 DEBUG:boto:auth_path=/qskrps-50/MULTIPART 2023-05-24T21:29:02.180 DEBUG:boto:path=/qskrps-50/MULTIPART?versionId=null 2023-05-24T21:29:02.180 DEBUG:boto:auth_path=/qskrps-50/MULTIPART?versionId=null 2023-05-24T21:29:02.180 DEBUG:boto:Method: HEAD 2023-05-24T21:29:02.181 DEBUG:boto:Path: /qskrps-50/MULTIPART?versionId=null 2023-05-24T21:29:02.181 DEBUG:boto:Data: 2023-05-24T21:29:02.181 DEBUG:boto:Headers: {} 2023-05-24T21:29:02.182 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:29:02.182 DEBUG:boto:Port: 8000 2023-05-24T21:29:02.182 DEBUG:boto:Params: {} 2023-05-24T21:29:02.183 DEBUG:boto:Token: None 2023-05-24T21:29:02.183 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:29:02 GMT /qskrps-50/MULTIPART?versionId=null 2023-05-24T21:29:02.184 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:pVSK+G+wHfUXKjHKE9Dsh9E27uU= 2023-05-24T21:29:02.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:29:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:pVSK+G+wHfUXKjHKE9Dsh9E27uU=', 'Content-Length': '0'} 2023-05-24T21:29:02.186 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000002499a342b2f4e18f-00646e819e-1627-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Wed, 24 May 2023 21:29:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:29:02.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2023-05-24T21:29:02.187 DEBUG:boto:path=/qskrps-50/MULTIPART 2023-05-24T21:29:02.187 DEBUG:boto:auth_path=/qskrps-50/MULTIPART 2023-05-24T21:29:02.187 DEBUG:boto:path=/qskrps-50/MULTIPART?versionId=null 2023-05-24T21:29:02.188 DEBUG:boto:auth_path=/qskrps-50/MULTIPART?versionId=null 2023-05-24T21:29:02.188 DEBUG:boto:Method: GET 2023-05-24T21:29:02.188 DEBUG:boto:Path: /qskrps-50/MULTIPART?versionId=null 2023-05-24T21:29:02.189 DEBUG:boto:Data: 2023-05-24T21:29:02.189 DEBUG:boto:Headers: {} 2023-05-24T21:29:02.190 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:29:02.190 DEBUG:boto:Port: 8001 2023-05-24T21:29:02.190 DEBUG:boto:Params: {} 2023-05-24T21:29:02.191 DEBUG:boto:Token: None 2023-05-24T21:29:02.191 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:29:02 GMT /qskrps-50/MULTIPART?versionId=null 2023-05-24T21:29:02.192 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:z0RScy+Y/hbzOFfwnnFraIcozjw= 2023-05-24T21:29:02.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:29:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:z0RScy+Y/hbzOFfwnnFraIcozjw=', 'Content-Length': '0'} 2023-05-24T21:29:02.194 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx0000081f441020de1216b-00646e819e-1625-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Wed, 24 May 2023 21:29:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:29:02.285 DEBUG:boto:path=/qskrps-50/MULTIPART 2023-05-24T21:29:02.286 DEBUG:boto:auth_path=/qskrps-50/MULTIPART 2023-05-24T21:29:02.286 DEBUG:boto:path=/qskrps-50/MULTIPART?versionId=null 2023-05-24T21:29:02.286 DEBUG:boto:auth_path=/qskrps-50/MULTIPART?versionId=null 2023-05-24T21:29:02.287 DEBUG:boto:Method: GET 2023-05-24T21:29:02.287 DEBUG:boto:Path: /qskrps-50/MULTIPART?versionId=null 2023-05-24T21:29:02.287 DEBUG:boto:Data: 2023-05-24T21:29:02.288 DEBUG:boto:Headers: {} 2023-05-24T21:29:02.288 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:29:02.289 DEBUG:boto:Port: 8000 2023-05-24T21:29:02.289 DEBUG:boto:Params: {} 2023-05-24T21:29:02.289 DEBUG:boto:Token: None 2023-05-24T21:29:02.290 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:29:02 GMT /qskrps-50/MULTIPART?versionId=null 2023-05-24T21:29:02.290 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:z0RScy+Y/hbzOFfwnnFraIcozjw= 2023-05-24T21:29:02.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:29:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:z0RScy+Y/hbzOFfwnnFraIcozjw=', 'Content-Length': '0'} 2023-05-24T21:29:02.296 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000003bb6a1d5b79c0a68-00646e819e-1627-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Wed, 24 May 2023 21:29:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:29:02.393 DEBUG:boto:path=/qskrps-50/ 2023-05-24T21:29:02.393 DEBUG:boto:auth_path=/qskrps-50/ 2023-05-24T21:29:02.394 DEBUG:boto:path=/qskrps-50/?prefix=MULTIPART 2023-05-24T21:29:02.394 DEBUG:boto:auth_path=/qskrps-50/?prefix=MULTIPART 2023-05-24T21:29:02.395 DEBUG:boto:Method: GET 2023-05-24T21:29:02.395 DEBUG:boto:Path: /qskrps-50/?prefix=MULTIPART 2023-05-24T21:29:02.395 DEBUG:boto:Data: 2023-05-24T21:29:02.396 DEBUG:boto:Headers: {} 2023-05-24T21:29:02.396 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:29:02.396 DEBUG:boto:Port: 8001 2023-05-24T21:29:02.397 DEBUG:boto:Params: {} 2023-05-24T21:29:02.397 DEBUG:boto:Token: None 2023-05-24T21:29:02.398 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:29:02 GMT /qskrps-50/ 2023-05-24T21:29:02.398 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ndEWCPYJ90CMBvkBz6uziXOZTPo= 2023-05-24T21:29:02.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:29:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ndEWCPYJ90CMBvkBz6uziXOZTPo=', 'Content-Length': '0'} 2023-05-24T21:29:02.401 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000388355f38af4fbb-00646e819e-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:29:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:29:02.402 DEBUG:boto:b'qskrps-50MULTIPART1000falseMULTIPART2023-05-24T21:28:46.448Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:29:02.402 DEBUG:boto:path=/qskrps-50/ 2023-05-24T21:29:02.403 DEBUG:boto:auth_path=/qskrps-50/ 2023-05-24T21:29:02.403 DEBUG:boto:path=/qskrps-50/?prefix=MULTIPART 2023-05-24T21:29:02.403 DEBUG:boto:auth_path=/qskrps-50/?prefix=MULTIPART 2023-05-24T21:29:02.404 DEBUG:boto:Method: GET 2023-05-24T21:29:02.404 DEBUG:boto:Path: /qskrps-50/?prefix=MULTIPART 2023-05-24T21:29:02.405 DEBUG:boto:Data: 2023-05-24T21:29:02.405 DEBUG:boto:Headers: {} 2023-05-24T21:29:02.405 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:29:02.406 DEBUG:boto:Port: 8000 2023-05-24T21:29:02.406 DEBUG:boto:Params: {} 2023-05-24T21:29:02.407 DEBUG:boto:Token: None 2023-05-24T21:29:02.407 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:29:02 GMT /qskrps-50/ 2023-05-24T21:29:02.407 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ndEWCPYJ90CMBvkBz6uziXOZTPo= 2023-05-24T21:29:02.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:29:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ndEWCPYJ90CMBvkBz6uziXOZTPo=', 'Content-Length': '0'} 2023-05-24T21:29:02.410 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa5618004b2dabe8-00646e819e-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:29:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:29:02.410 DEBUG:boto:b'qskrps-50MULTIPART1000falseMULTIPART2023-05-24T21:28:46.448Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:29:02.411 DEBUG:boto:path=/qskrps-50/MULTIPART 2023-05-24T21:29:02.411 DEBUG:boto:auth_path=/qskrps-50/MULTIPART 2023-05-24T21:29:02.412 DEBUG:boto:Method: HEAD 2023-05-24T21:29:02.412 DEBUG:boto:Path: /qskrps-50/MULTIPART 2023-05-24T21:29:02.412 DEBUG:boto:Data: 2023-05-24T21:29:02.413 DEBUG:boto:Headers: {} 2023-05-24T21:29:02.413 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:29:02.413 DEBUG:boto:Port: 8001 2023-05-24T21:29:02.414 DEBUG:boto:Params: {} 2023-05-24T21:29:02.414 DEBUG:boto:Token: None 2023-05-24T21:29:02.415 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:29:02 GMT /qskrps-50/MULTIPART 2023-05-24T21:29:02.415 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:WsDC5ikHH7FF+71zlhmX7bt3xEU= 2023-05-24T21:29:02.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:29:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WsDC5ikHH7FF+71zlhmX7bt3xEU=', 'Content-Length': '0'} 2023-05-24T21:29:02.417 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000824e948060b6be33-00646e819e-1625-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Wed, 24 May 2023 21:29:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:29:02.417 DEBUG:boto:path=/qskrps-50/MULTIPART 2023-05-24T21:29:02.418 DEBUG:boto:auth_path=/qskrps-50/MULTIPART 2023-05-24T21:29:02.418 DEBUG:boto:Method: HEAD 2023-05-24T21:29:02.419 DEBUG:boto:Path: /qskrps-50/MULTIPART 2023-05-24T21:29:02.419 DEBUG:boto:Data: 2023-05-24T21:29:02.419 DEBUG:boto:Headers: {} 2023-05-24T21:29:02.420 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:29:02.420 DEBUG:boto:Port: 8000 2023-05-24T21:29:02.420 DEBUG:boto:Params: {} 2023-05-24T21:29:02.421 DEBUG:boto:Token: None 2023-05-24T21:29:02.421 DEBUG:boto:StringToSign: HEAD Wed, 24 May 2023 21:29:02 GMT /qskrps-50/MULTIPART 2023-05-24T21:29:02.421 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:WsDC5ikHH7FF+71zlhmX7bt3xEU= 2023-05-24T21:29:02.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:29:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WsDC5ikHH7FF+71zlhmX7bt3xEU=', 'Content-Length': '0'} 2023-05-24T21:29:02.423 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000cff12b2f84d33003-00646e819e-1627-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Wed, 24 May 2023 21:29:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:29:02.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2023-05-24T21:29:02.424 DEBUG:boto:path=/qskrps-50/MULTIPART 2023-05-24T21:29:02.425 DEBUG:boto:auth_path=/qskrps-50/MULTIPART 2023-05-24T21:29:02.425 DEBUG:boto:Method: GET 2023-05-24T21:29:02.425 DEBUG:boto:Path: /qskrps-50/MULTIPART 2023-05-24T21:29:02.426 DEBUG:boto:Data: 2023-05-24T21:29:02.426 DEBUG:boto:Headers: {} 2023-05-24T21:29:02.427 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:29:02.427 DEBUG:boto:Port: 8001 2023-05-24T21:29:02.427 DEBUG:boto:Params: {} 2023-05-24T21:29:02.428 DEBUG:boto:Token: None 2023-05-24T21:29:02.428 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:29:02 GMT /qskrps-50/MULTIPART 2023-05-24T21:29:02.429 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:W9UWxdx6ix9NwPlklzb5++HtJns= 2023-05-24T21:29:02.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:29:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:W9UWxdx6ix9NwPlklzb5++HtJns=', 'Content-Length': '0'} 2023-05-24T21:29:02.431 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000004921abaede977f32-00646e819e-1625-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Wed, 24 May 2023 21:29:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:29:02.516 DEBUG:boto:path=/qskrps-50/MULTIPART 2023-05-24T21:29:02.517 DEBUG:boto:auth_path=/qskrps-50/MULTIPART 2023-05-24T21:29:02.517 DEBUG:boto:Method: GET 2023-05-24T21:29:02.518 DEBUG:boto:Path: /qskrps-50/MULTIPART 2023-05-24T21:29:02.518 DEBUG:boto:Data: 2023-05-24T21:29:02.518 DEBUG:boto:Headers: {} 2023-05-24T21:29:02.519 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:29:02.519 DEBUG:boto:Port: 8000 2023-05-24T21:29:02.519 DEBUG:boto:Params: {} 2023-05-24T21:29:02.520 DEBUG:boto:Token: None 2023-05-24T21:29:02.520 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:29:02 GMT /qskrps-50/MULTIPART 2023-05-24T21:29:02.521 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:W9UWxdx6ix9NwPlklzb5++HtJns= 2023-05-24T21:29:02.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:29:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:W9UWxdx6ix9NwPlklzb5++HtJns=', 'Content-Length': '0'} 2023-05-24T21:29:02.526 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:28:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000fa20bfc70f9e1739-00646e819e-1627-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Wed, 24 May 2023 21:29:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:29:02.621 DEBUG:boto:path=/qskrps-50/ 2023-05-24T21:29:02.622 DEBUG:boto:auth_path=/qskrps-50/ 2023-05-24T21:29:02.623 DEBUG:boto:path=/qskrps-50/?prefix=MULTIPART 2023-05-24T21:29:02.623 DEBUG:boto:auth_path=/qskrps-50/?prefix=MULTIPART 2023-05-24T21:29:02.623 DEBUG:boto:Method: GET 2023-05-24T21:29:02.624 DEBUG:boto:Path: /qskrps-50/?prefix=MULTIPART 2023-05-24T21:29:02.624 DEBUG:boto:Data: 2023-05-24T21:29:02.624 DEBUG:boto:Headers: {} 2023-05-24T21:29:02.625 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-05-24T21:29:02.625 DEBUG:boto:Port: 8001 2023-05-24T21:29:02.626 DEBUG:boto:Params: {} 2023-05-24T21:29:02.626 DEBUG:boto:Token: None 2023-05-24T21:29:02.626 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:29:02 GMT /qskrps-50/ 2023-05-24T21:29:02.627 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ndEWCPYJ90CMBvkBz6uziXOZTPo= 2023-05-24T21:29:02.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:29:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ndEWCPYJ90CMBvkBz6uziXOZTPo=', 'Content-Length': '0'} 2023-05-24T21:29:02.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c162c98373cb51b-00646e819e-1625-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:29:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:29:02.630 DEBUG:boto:b'qskrps-50MULTIPART1000falseMULTIPART2023-05-24T21:28:46.448Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:29:02.631 DEBUG:boto:path=/qskrps-50/ 2023-05-24T21:29:02.631 DEBUG:boto:auth_path=/qskrps-50/ 2023-05-24T21:29:02.632 DEBUG:boto:path=/qskrps-50/?prefix=MULTIPART 2023-05-24T21:29:02.632 DEBUG:boto:auth_path=/qskrps-50/?prefix=MULTIPART 2023-05-24T21:29:02.632 DEBUG:boto:Method: GET 2023-05-24T21:29:02.633 DEBUG:boto:Path: /qskrps-50/?prefix=MULTIPART 2023-05-24T21:29:02.633 DEBUG:boto:Data: 2023-05-24T21:29:02.634 DEBUG:boto:Headers: {} 2023-05-24T21:29:02.634 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:29:02.634 DEBUG:boto:Port: 8000 2023-05-24T21:29:02.635 DEBUG:boto:Params: {} 2023-05-24T21:29:02.635 DEBUG:boto:Token: None 2023-05-24T21:29:02.635 DEBUG:boto:StringToSign: GET Wed, 24 May 2023 21:29:02 GMT /qskrps-50/ 2023-05-24T21:29:02.636 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ndEWCPYJ90CMBvkBz6uziXOZTPo= 2023-05-24T21:29:02.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:29:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ndEWCPYJ90CMBvkBz6uziXOZTPo=', 'Content-Length': '0'} 2023-05-24T21:29:02.639 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c33b5f8fd136054-00646e819e-1627-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:29:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:29:02.639 DEBUG:boto:b'qskrps-50MULTIPART1000falseMULTIPART2023-05-24T21:28:46.448Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-24T21:29:02.640 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qskrps-50 zones={a2, a1} 2023-05-24T21:29:02.640 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multipart_object_sync ... ok 2023-05-24T21:29:02.641 INFO:rgw_multi.tests:create bucket zone=a1 name=qskrps-52 2023-05-24T21:29:02.642 DEBUG:boto:path=/qskrps-52/ 2023-05-24T21:29:02.642 DEBUG:boto:auth_path=/qskrps-52/ 2023-05-24T21:29:02.642 DEBUG:boto:Method: PUT 2023-05-24T21:29:02.643 DEBUG:boto:Path: /qskrps-52/ 2023-05-24T21:29:02.643 DEBUG:boto:Data: 2023-05-24T21:29:02.643 DEBUG:boto:Headers: {} 2023-05-24T21:29:02.644 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:29:02.644 DEBUG:boto:Port: 8000 2023-05-24T21:29:02.644 DEBUG:boto:Params: {} 2023-05-24T21:29:02.645 DEBUG:boto:Token: None 2023-05-24T21:29:02.645 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:29:02 GMT /qskrps-52/ 2023-05-24T21:29:02.645 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:S33yJGZOVB/Ai/BHOrSsR5D43Zo= 2023-05-24T21:29:02.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:29:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:S33yJGZOVB/Ai/BHOrSsR5D43Zo=', 'Content-Length': '0'} 2023-05-24T21:29:02.665 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ff344861062aa898-00646e819e-1627-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:29:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:29:02.666 DEBUG:boto:path=/qskrps-52/testobj-sse-c 2023-05-24T21:29:02.666 DEBUG:boto:auth_path=/qskrps-52/testobj-sse-c 2023-05-24T21:29:02.667 DEBUG:boto:Method: PUT 2023-05-24T21:29:02.667 DEBUG:boto:Path: /qskrps-52/testobj-sse-c 2023-05-24T21:29:02.667 DEBUG:boto:Data: 2023-05-24T21:29:02.668 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-05-24T21:29:02.668 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:29:02.668 DEBUG:boto:Port: 8000 2023-05-24T21:29:02.669 DEBUG:boto:Params: {} 2023-05-24T21:29:02.669 DEBUG:boto:Token: None 2023-05-24T21:29:02.670 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Wed, 24 May 2023 21:29:02 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /qskrps-52/testobj-sse-c 2023-05-24T21:29:02.670 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:N7YAD17l+Tu23Z3Kwa4hlhObGeI= 2023-05-24T21:29:02.670 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:29:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:N7YAD17l+Tu23Z3Kwa4hlhObGeI='} 2023-05-24T21:29:02.677 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('x-amz-request-id', 'tx0000096c7f1ae667a8e25-00646e819e-1627-a1'), ('Date', 'Wed, 24 May 2023 21:29:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:29:02.678 DEBUG:boto:path=/qskrps-52/testobj-sse-kms 2023-05-24T21:29:02.678 DEBUG:boto:auth_path=/qskrps-52/testobj-sse-kms 2023-05-24T21:29:02.678 DEBUG:boto:Method: PUT 2023-05-24T21:29:02.679 DEBUG:boto:Path: /qskrps-52/testobj-sse-kms 2023-05-24T21:29:02.679 DEBUG:boto:Data: 2023-05-24T21:29:02.679 DEBUG:boto:Headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-05-24T21:29:02.680 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:29:02.680 DEBUG:boto:Port: 8000 2023-05-24T21:29:02.681 DEBUG:boto:Params: {} 2023-05-24T21:29:02.681 DEBUG:boto:Token: None 2023-05-24T21:29:02.681 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Wed, 24 May 2023 21:29:02 GMT x-amz-server-side-encryption:aws:kms x-amz-server-side-encryption-aws-kms-key-id:testkey-1 /qskrps-52/testobj-sse-kms 2023-05-24T21:29:02.682 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:vE2nYt9i/CoIpvRdfKUx6PhXML8= 2023-05-24T21:29:02.682 DEBUG:boto:Final headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:29:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:vE2nYt9i/CoIpvRdfKUx6PhXML8='} 2023-05-24T21:29:02.687 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('x-amz-server-side-encryption-context', 'eyJhd3M6czM6YXJuIjoiYXJuOmF3czpzMzo6cmd3LW11bHRpc2l0ZS10ZXN0LXVzZXI6cXNrcnBzLTUyL3Rlc3RvYmotc3NlLWttcyJ9'), ('x-amz-request-id', 'tx00000372e5bf4ac75afb8-00646e819e-1627-a1'), ('Date', 'Wed, 24 May 2023 21:29:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-24T21:29:02.688 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:29:02.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:29:02.689 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:29:02.737 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:29:02.738 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:29:02.767 INFO:teuthology.orchestra.run.smithi071.stderr:No --period given, using current period=5c04936b-70ea-4415-b4eb-3aad3dc612e5 2023-05-24T21:29:02.779 INFO:teuthology.orchestra.run.smithi071.stdout:[ 2023-05-24T21:29:02.779 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:29:02.780 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963742.659011_575.1", 2023-05-24T21:29:02.780 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:29:02.659011Z" 2023-05-24T21:29:02.780 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:29:02.780 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:29:02.781 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:29:02.781 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "2023-05-24T21:06:32.333844Z" 2023-05-24T21:29:02.781 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:29:02.781 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:29:02.781 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:29:02.782 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:29:02.782 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:29:02.782 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:29:02.782 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:29:02.782 INFO:teuthology.orchestra.run.smithi071.stdout: "last_update": "0.000000" 2023-05-24T21:29:02.783 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:29:02.783 INFO:teuthology.orchestra.run.smithi071.stdout:] 2023-05-24T21:29:02.784 DEBUG:rgw_multi.tests:master meta markers={0: '1_1684963742.659011_575.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:29:02.784 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-24T21:29:02.784 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:29:02.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:29:02.785 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:29:02.837 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:29:02.838 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:29:02.870 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:29:02.864+0000 7fd7bd33d780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:29:02.870 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:29:02.865+0000 7fd7bd33d780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:29:02.901 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:29:02.901 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:29:02.902 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:29:02.902 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:29:02.902 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:29:02.902 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:29:02.902 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:29:02.902 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:29:02.903 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:29:02.903 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:29:02.903 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:29:02.903 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:29:02.903 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:29:02.903 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963742.651710_572.1", 2023-05-24T21:29:02.904 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:29:02.904 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:29:02.904 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:29:02.904 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:29:02.651710Z", 2023-05-24T21:29:02.904 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:29:02.904 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:29:02.905 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:29:02.905 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:29:02.905 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:29:02.905 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:29:02.905 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:29:02.905 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:29:02.906 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:29:02.906 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:29:02.906 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:29:02.906 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:29:02.906 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:29:02.906 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:29:02.906 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:29:02.907 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:29:02.907 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:29:02.907 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:29:02.907 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:29:02.907 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:29:02.907 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:29:02.908 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:29:02.908 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:29:02.908 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:29:02.908 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:29:02.908 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:29:02.908 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:29:02.908 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:29:02.909 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:29:02.909 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:29:02.909 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:29:02.909 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:29:02.909 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:29:02.910 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:29:02.910 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:29:02.910 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:29:02.910 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:29:02.910 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:29:02.910 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:29:02.911 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:29:02.911 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:29:02.911 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:29:02.911 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:29:02.911 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:29:02.912 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:29:02.912 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:29:02Z" 2023-05-24T21:29:02.912 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:29:02.913 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684963742.651710_572.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:29:02.651710Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:29:02Z" } 2023-05-24T21:29:02.913 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963742.651710_572.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:29:02.651710Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:29:02.914 DEBUG:rgw_multi.tests:log_status={0: '1_1684963742.659011_575.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:29:02.914 DEBUG:rgw_multi.tests:sync_status={0: '1_1684963742.651710_572.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:29:02.915 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1684963742.659011_575.1 target=1_1684963742.651710_572.1 2023-05-24T21:29:07.916 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:29:07.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:29:07.917 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:29:07.974 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:29:07.975 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:29:08.015 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:29:08.010+0000 7fb50ba5f780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:29:08.016 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:29:08.011+0000 7fb50ba5f780 20 RGW-SYNC:meta: read sync status 2023-05-24T21:29:08.032 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-05-24T21:29:08.033 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-05-24T21:29:08.033 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-05-24T21:29:08.033 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-05-24T21:29:08.033 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-05-24T21:29:08.033 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-24T21:29:08.034 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:29:08.034 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:29:08.034 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-05-24T21:29:08.034 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:29:08.034 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-05-24T21:29:08.034 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:29:08.034 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:29:08.035 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684963742.659011_575.1", 2023-05-24T21:29:08.035 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:29:08.035 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-05-24T21:29:08.035 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:29:08.035 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:29:02.659011Z", 2023-05-24T21:29:08.035 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:29:08.036 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:29:08.036 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:29:08.036 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:29:08.036 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-05-24T21:29:08.036 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:29:08.036 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:29:08.036 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1684962392.333844_91.1", 2023-05-24T21:29:08.037 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:29:08.037 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:29:08.037 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:29:08.037 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-24T21:29:08.037 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:29:08.037 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:29:08.037 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:29:08.038 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:29:08.038 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-05-24T21:29:08.038 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:29:08.038 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:29:08.038 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:29:08.038 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:29:08.038 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:29:08.039 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:29:08.039 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:29:08.039 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:29:08.039 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:29:08.039 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:29:08.039 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-05-24T21:29:08.040 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-05-24T21:29:08.040 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-05-24T21:29:08.040 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-05-24T21:29:08.040 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-05-24T21:29:08.040 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-05-24T21:29:08.040 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-05-24T21:29:08.040 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-05-24T21:29:08.041 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-05-24T21:29:08.041 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-05-24T21:29:08.041 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:29:08.041 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-05-24T21:29:08.041 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-05-24T21:29:08.041 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:29:08.041 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-05-24T21:29:08.042 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-05-24T21:29:08.042 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-05-24T21:29:08.042 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-05-24T21:29:08.042 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-05-24T21:29:08Z" 2023-05-24T21:29:08.042 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-05-24T21:29:08.043 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1684963742.659011_575.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-24T21:29:02.659011Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1684962392.333844_91.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-24T21:06:32.333844Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-24T21:29:08Z" } 2023-05-24T21:29:08.044 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963742.659011_575.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:29:02.659011Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-24T21:29:08.044 DEBUG:rgw_multi.tests:log_status={0: '1_1684963742.659011_575.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:29:08.044 DEBUG:rgw_multi.tests:sync_status={0: '1_1684963742.659011_575.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-24T21:29:08.045 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-24T21:29:08.045 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-52', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:29:08.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-52', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-24T21:29:08.046 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-52 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-24T21:29:08.103 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:29:08.103 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:29:08.149 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:29:08.143+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:29:13.149 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:29:13.144+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:29:13.675 INFO:tasks.rgw.c1.client.0.smithi071.stdout:tcmalloc: large alloc 9825697792 bytes == 0x55e5f074e000 @ 0x7f4a63069760 0x7f4a6308ac64 0x7f492584d166 0x7f492584be93 0x55e5e42eb399 0x55e5e45b9124 0x55e5e4599a56 0x55e5e45d6d11 0x55e5e45d75e4 0x55e5e4381402 0x55e5e4175a1c 0x55e5e4179134 0x55e5e40b49b6 0x55e5e40b5684 0x55e5e4f056cf 2023-05-24T21:29:13.682 INFO:tasks.rgw.c1.client.0.smithi071.stdout:tcmalloc: large alloc 15572418560 bytes == 0x55e83c146000 @ 0x7f4a63069760 0x7f4a6308ac64 0x7f492584d166 0x7f492584be93 0x55e5e42eb399 0x55e5e45b9124 0x55e5e4599a56 0x55e5e45d6d11 0x55e5e45d75e4 0x55e5e4381402 0x55e5e4175a1c 0x55e5e4179134 0x55e5e40b49b6 0x55e5e40b5684 0x55e5e4f056cf 2023-05-24T21:29:17.406 INFO:tasks.rgw.c1.client.0.smithi071.stdout:terminate called after throwing an instance of 'ceph::buffer::v15_2_0::end_of_buffer' 2023-05-24T21:29:17.406 INFO:tasks.rgw.c1.client.0.smithi071.stdout: what(): End of buffer [buffer:2] 2023-05-24T21:29:17.406 INFO:tasks.rgw.c1.client.0.smithi071.stdout:*** Caught signal (Aborted) ** 2023-05-24T21:29:17.406 INFO:tasks.rgw.c1.client.0.smithi071.stdout: in thread 7f497d388700 thread_name:radosgw 2023-05-24T21:29:17.407 INFO:tasks.rgw.c1.client.0.smithi071.stdout: ceph version 18.0.0-3443-g3395903b (3395903b291ea37c38b39a05e6d9aa4f40306c80) reef (dev) 2023-05-24T21:29:17.407 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f4a622c0cf0] 2023-05-24T21:29:17.407 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 2: gsignal() 2023-05-24T21:29:17.407 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 3: abort() 2023-05-24T21:29:17.407 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7f4a61a0609b] 2023-05-24T21:29:17.408 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 5: /lib64/libstdc++.so.6(+0x9653c) [0x7f4a61a0c53c] 2023-05-24T21:29:17.408 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 6: /lib64/libstdc++.so.6(+0x95559) [0x7f4a61a0b559] 2023-05-24T21:29:17.408 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 7: __gxx_personality_v0() 2023-05-24T21:29:17.408 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 8: /lib64/libgcc_s.so.1(+0x10b03) [0x7f4a613ecb03] 2023-05-24T21:29:17.408 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 9: _Unwind_Resume() 2023-05-24T21:29:17.408 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 10: radosgw(+0x50fa2e) [0x55e5e3f71a2e] 2023-05-24T21:29:17.409 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 11: radosgw(+0x6432cd) [0x55e5e40a52cd] 2023-05-24T21:29:17.409 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 12: /lib64/libstdc++.so.6(+0xc2b13) [0x7f4a61a38b13] 2023-05-24T21:29:17.409 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 13: /lib64/libpthread.so.0(+0x81ca) [0x7f4a622b61ca] 2023-05-24T21:29:17.409 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 14: clone() 2023-05-24T21:29:17.409 INFO:tasks.rgw.c1.client.0.smithi071.stdout:2023-05-24T21:29:17.401+0000 7f497d388700 -1 *** Caught signal (Aborted) ** 2023-05-24T21:29:17.409 INFO:tasks.rgw.c1.client.0.smithi071.stdout: in thread 7f497d388700 thread_name:radosgw 2023-05-24T21:29:17.409 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 2023-05-24T21:29:17.410 INFO:tasks.rgw.c1.client.0.smithi071.stdout: ceph version 18.0.0-3443-g3395903b (3395903b291ea37c38b39a05e6d9aa4f40306c80) reef (dev) 2023-05-24T21:29:17.410 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f4a622c0cf0] 2023-05-24T21:29:17.410 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 2: gsignal() 2023-05-24T21:29:17.410 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 3: abort() 2023-05-24T21:29:17.410 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7f4a61a0609b] 2023-05-24T21:29:17.410 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 5: /lib64/libstdc++.so.6(+0x9653c) [0x7f4a61a0c53c] 2023-05-24T21:29:17.410 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 6: /lib64/libstdc++.so.6(+0x95559) [0x7f4a61a0b559] 2023-05-24T21:29:17.411 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 7: __gxx_personality_v0() 2023-05-24T21:29:17.411 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 8: /lib64/libgcc_s.so.1(+0x10b03) [0x7f4a613ecb03] 2023-05-24T21:29:17.411 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 9: _Unwind_Resume() 2023-05-24T21:29:17.411 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 10: radosgw(+0x50fa2e) [0x55e5e3f71a2e] 2023-05-24T21:29:17.411 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 11: radosgw(+0x6432cd) [0x55e5e40a52cd] 2023-05-24T21:29:17.411 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 12: /lib64/libstdc++.so.6(+0xc2b13) [0x7f4a61a38b13] 2023-05-24T21:29:17.411 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 13: /lib64/libpthread.so.0(+0x81ca) [0x7f4a622b61ca] 2023-05-24T21:29:17.412 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 14: clone() 2023-05-24T21:29:17.412 INFO:tasks.rgw.c1.client.0.smithi071.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-05-24T21:29:17.412 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 2023-05-24T21:29:17.422 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 0> 2023-05-24T21:29:17.401+0000 7f497d388700 -1 *** Caught signal (Aborted) ** 2023-05-24T21:29:17.422 INFO:tasks.rgw.c1.client.0.smithi071.stdout: in thread 7f497d388700 thread_name:radosgw 2023-05-24T21:29:17.422 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 2023-05-24T21:29:17.422 INFO:tasks.rgw.c1.client.0.smithi071.stdout: ceph version 18.0.0-3443-g3395903b (3395903b291ea37c38b39a05e6d9aa4f40306c80) reef (dev) 2023-05-24T21:29:17.422 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f4a622c0cf0] 2023-05-24T21:29:17.422 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 2: gsignal() 2023-05-24T21:29:17.423 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 3: abort() 2023-05-24T21:29:17.423 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7f4a61a0609b] 2023-05-24T21:29:17.423 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 5: /lib64/libstdc++.so.6(+0x9653c) [0x7f4a61a0c53c] 2023-05-24T21:29:17.423 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 6: /lib64/libstdc++.so.6(+0x95559) [0x7f4a61a0b559] 2023-05-24T21:29:17.423 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 7: __gxx_personality_v0() 2023-05-24T21:29:17.423 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 8: /lib64/libgcc_s.so.1(+0x10b03) [0x7f4a613ecb03] 2023-05-24T21:29:17.423 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 9: _Unwind_Resume() 2023-05-24T21:29:17.424 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 10: radosgw(+0x50fa2e) [0x55e5e3f71a2e] 2023-05-24T21:29:17.424 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 11: radosgw(+0x6432cd) [0x55e5e40a52cd] 2023-05-24T21:29:17.424 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 12: /lib64/libstdc++.so.6(+0xc2b13) [0x7f4a61a38b13] 2023-05-24T21:29:17.424 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 13: /lib64/libpthread.so.0(+0x81ca) [0x7f4a622b61ca] 2023-05-24T21:29:17.424 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 14: clone() 2023-05-24T21:29:17.424 INFO:tasks.rgw.c1.client.0.smithi071.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-05-24T21:29:17.424 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 2023-05-24T21:29:17.438 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 0> 2023-05-24T21:29:17.401+0000 7f497d388700 -1 *** Caught signal (Aborted) ** 2023-05-24T21:29:17.438 INFO:tasks.rgw.c1.client.0.smithi071.stdout: in thread 7f497d388700 thread_name:radosgw 2023-05-24T21:29:17.438 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 2023-05-24T21:29:17.439 INFO:tasks.rgw.c1.client.0.smithi071.stdout: ceph version 18.0.0-3443-g3395903b (3395903b291ea37c38b39a05e6d9aa4f40306c80) reef (dev) 2023-05-24T21:29:17.439 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f4a622c0cf0] 2023-05-24T21:29:17.439 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 2: gsignal() 2023-05-24T21:29:17.439 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 3: abort() 2023-05-24T21:29:17.439 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7f4a61a0609b] 2023-05-24T21:29:17.439 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 5: /lib64/libstdc++.so.6(+0x9653c) [0x7f4a61a0c53c] 2023-05-24T21:29:17.439 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 6: /lib64/libstdc++.so.6(+0x95559) [0x7f4a61a0b559] 2023-05-24T21:29:17.440 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 7: __gxx_personality_v0() 2023-05-24T21:29:17.440 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 8: /lib64/libgcc_s.so.1(+0x10b03) [0x7f4a613ecb03] 2023-05-24T21:29:17.440 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 9: _Unwind_Resume() 2023-05-24T21:29:17.440 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 10: radosgw(+0x50fa2e) [0x55e5e3f71a2e] 2023-05-24T21:29:17.440 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 11: radosgw(+0x6432cd) [0x55e5e40a52cd] 2023-05-24T21:29:17.440 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 12: /lib64/libstdc++.so.6(+0xc2b13) [0x7f4a61a38b13] 2023-05-24T21:29:17.441 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 13: /lib64/libpthread.so.0(+0x81ca) [0x7f4a622b61ca] 2023-05-24T21:29:17.441 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 14: clone() 2023-05-24T21:29:17.441 INFO:tasks.rgw.c1.client.0.smithi071.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-05-24T21:29:17.441 INFO:tasks.rgw.c1.client.0.smithi071.stdout: 2023-05-24T21:29:18.150 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:29:18.144+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:29:18.198 INFO:tasks.rgw.c1.client.0.smithi071.stderr:daemon-helper: command crashed with signal 6 2023-05-24T21:29:23.150 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:29:23.144+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:29:23.966 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2023-05-24T21:29:28.151 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:29:28.144+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:29:29.570 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~6s 2023-05-24T21:29:33.151 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:29:33.144+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:29:35.174 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~11s 2023-05-24T21:29:38.151 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:29:38.145+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:29:40.778 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~17s 2023-05-24T21:29:43.152 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:29:43.146+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:29:46.382 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~22s 2023-05-24T21:29:48.152 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:29:48.147+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:29:51.985 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~28s 2023-05-24T21:29:53.153 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:29:53.147+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:29:57.590 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~34s 2023-05-24T21:29:58.154 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:29:58.148+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:30:03.157 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:30:03.149+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:30:03.194 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~39s 2023-05-24T21:30:08.155 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:30:08.149+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:30:08.798 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~45s 2023-05-24T21:30:13.156 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:30:13.150+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:30:14.401 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~50s 2023-05-24T21:30:18.156 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:30:18.150+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:30:20.004 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~56s 2023-05-24T21:30:23.157 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:30:23.150+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:30:25.610 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~62s 2023-05-24T21:30:28.157 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:30:28.151+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:30:31.213 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~67s 2023-05-24T21:30:33.158 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:30:33.152+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:30:36.817 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~73s 2023-05-24T21:30:38.159 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:30:38.153+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:30:42.421 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~78s 2023-05-24T21:30:43.159 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:30:43.154+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:30:46.613 INFO:tasks.rgw.c1.client.1.smithi071.stdout:2023-05-24T21:30:46.608+0000 7f1a4f21b700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-05-24T21:30:48.025 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~84s 2023-05-24T21:30:48.160 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:30:48.154+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:30:53.161 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:30:53.155+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:30:53.628 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~90s 2023-05-24T21:30:57.880 INFO:tasks.rgw.c2.client.1.smithi157.stdout:2023-05-24T21:30:57.874+0000 7f37d6f3f700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-05-24T21:30:58.161 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:30:58.156+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:30:59.232 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~95s 2023-05-24T21:31:03.162 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:31:03.156+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:31:04.836 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~101s 2023-05-24T21:31:08.162 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:31:08.157+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:31:10.439 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~106s 2023-05-24T21:31:13.163 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:31:13.156+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:31:16.043 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~112s 2023-05-24T21:31:18.164 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:31:18.157+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:31:21.647 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~118s 2023-05-24T21:31:23.164 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:31:23.158+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:31:27.250 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~123s 2023-05-24T21:31:28.165 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:31:28.158+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:31:32.855 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~129s 2023-05-24T21:31:33.166 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:31:33.160+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:31:38.166 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:31:38.161+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:31:38.459 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~134s 2023-05-24T21:31:43.167 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:31:43.161+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:31:44.063 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~140s 2023-05-24T21:31:48.168 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:31:48.162+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:31:49.668 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~146s 2023-05-24T21:31:53.168 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:31:53.162+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:31:55.272 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~151s 2023-05-24T21:31:58.169 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:31:58.163+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:32:00.876 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~157s 2023-05-24T21:32:03.169 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:32:03.164+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:32:06.481 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~163s 2023-05-24T21:32:08.170 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:32:08.163+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:32:12.086 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~168s 2023-05-24T21:32:13.171 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:32:13.164+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:32:17.690 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~174s 2023-05-24T21:32:18.171 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:32:18.165+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:32:23.172 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:32:23.166+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:32:23.294 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~179s 2023-05-24T21:32:28.172 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:32:28.167+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:32:28.898 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~185s 2023-05-24T21:32:33.173 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:32:33.167+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:32:34.501 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~191s 2023-05-24T21:32:38.173 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:32:38.168+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:32:40.105 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~196s 2023-05-24T21:32:43.174 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:32:43.169+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:32:45.708 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~202s 2023-05-24T21:32:48.175 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:32:48.169+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:32:51.313 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~207s 2023-05-24T21:32:53.175 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:32:53.170+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:32:56.916 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~213s 2023-05-24T21:32:58.176 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:32:58.170+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:33:02.520 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~219s 2023-05-24T21:33:03.176 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:33:03.170+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:33:08.125 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~224s 2023-05-24T21:33:08.177 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:33:08.171+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:33:13.178 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:33:13.171+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:33:13.729 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~230s 2023-05-24T21:33:18.178 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:33:18.172+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:33:19.333 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~235s 2023-05-24T21:33:23.179 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:33:23.173+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:33:24.937 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~241s 2023-05-24T21:33:28.180 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:33:28.174+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:33:30.542 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~247s 2023-05-24T21:33:33.180 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:33:33.175+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:33:36.145 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~252s 2023-05-24T21:33:38.181 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:33:38.175+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:33:41.749 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~258s 2023-05-24T21:33:43.182 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:33:43.176+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:33:47.353 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~263s 2023-05-24T21:33:48.182 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:33:48.177+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:33:52.956 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~269s 2023-05-24T21:33:53.183 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:33:53.177+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:33:58.184 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:33:58.177+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-24T21:33:58.560 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~275s 2023-05-24T21:34:03.184 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:34:03.178+0000 7fee02333780 -1 bucket checkpoint timed out waiting to reach incremental sync 2023-05-24T21:34:03.185 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:34:03.178+0000 7fee02333780 0 bucket sync checkpoint failed: (110) Connection timed out 2023-05-24T21:34:03.185 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:34:03.178+0000 7fee02333780 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-05-24T21:34:03.203 DEBUG:teuthology.orchestra.run:got remote process result: 110 2023-05-24T21:34:03.205 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_encrypted_object_sync ... FAIL 2023-05-24T21:34:03.206 INFO:rgw_multi.tests:create bucket zone=a1 name=qskrps-53 2023-05-24T21:34:03.206 DEBUG:boto:path=/qskrps-53/ 2023-05-24T21:34:03.207 DEBUG:boto:auth_path=/qskrps-53/ 2023-05-24T21:34:03.207 DEBUG:boto:Method: PUT 2023-05-24T21:34:03.208 DEBUG:boto:Path: /qskrps-53/ 2023-05-24T21:34:03.208 DEBUG:boto:Data: 2023-05-24T21:34:03.208 DEBUG:boto:Headers: {} 2023-05-24T21:34:03.209 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:34:03.209 DEBUG:boto:Port: 8000 2023-05-24T21:34:03.209 DEBUG:boto:Params: {} 2023-05-24T21:34:03.210 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T21:34:03.210 DEBUG:boto:Token: None 2023-05-24T21:34:03.211 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:34:03 GMT /qskrps-53/ 2023-05-24T21:34:03.211 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:vIo6xRa8IRKp4kRrcxO3plRPKkQ= 2023-05-24T21:34:03.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:34:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:vIo6xRa8IRKp4kRrcxO3plRPKkQ=', 'Content-Length': '0'} 2023-05-24T21:34:03.213 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T21:34:03.293 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T21:34:03.885 DEBUG:boto:Token: None 2023-05-24T21:34:03.885 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:34:03 GMT /qskrps-53/ 2023-05-24T21:34:03.886 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:vIo6xRa8IRKp4kRrcxO3plRPKkQ= 2023-05-24T21:34:03.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:34:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:vIo6xRa8IRKp4kRrcxO3plRPKkQ=', 'Content-Length': '0'} 2023-05-24T21:34:03.887 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T21:34:03.888 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T21:34:04.163 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~280s 2023-05-24T21:34:05.740 DEBUG:boto:Token: None 2023-05-24T21:34:05.740 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:34:03 GMT /qskrps-53/ 2023-05-24T21:34:05.741 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:vIo6xRa8IRKp4kRrcxO3plRPKkQ= 2023-05-24T21:34:05.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:34:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:vIo6xRa8IRKp4kRrcxO3plRPKkQ=', 'Content-Length': '0'} 2023-05-24T21:34:05.742 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T21:34:05.743 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T21:34:08.227 DEBUG:boto:Token: None 2023-05-24T21:34:08.228 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:34:03 GMT /qskrps-53/ 2023-05-24T21:34:08.229 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:vIo6xRa8IRKp4kRrcxO3plRPKkQ= 2023-05-24T21:34:08.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:34:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:vIo6xRa8IRKp4kRrcxO3plRPKkQ=', 'Content-Length': '0'} 2023-05-24T21:34:08.230 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T21:34:08.231 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T21:34:09.765 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~286s 2023-05-24T21:34:12.819 DEBUG:boto:Token: None 2023-05-24T21:34:12.819 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:34:03 GMT /qskrps-53/ 2023-05-24T21:34:12.820 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:vIo6xRa8IRKp4kRrcxO3plRPKkQ= 2023-05-24T21:34:12.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:34:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:vIo6xRa8IRKp4kRrcxO3plRPKkQ=', 'Content-Length': '0'} 2023-05-24T21:34:12.821 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T21:34:12.822 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T21:34:15.369 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~291s 2023-05-24T21:34:20.972 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~297s 2023-05-24T21:34:24.282 DEBUG:boto:Token: None 2023-05-24T21:34:24.283 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:34:03 GMT /qskrps-53/ 2023-05-24T21:34:24.283 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:vIo6xRa8IRKp4kRrcxO3plRPKkQ= 2023-05-24T21:34:24.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:34:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:vIo6xRa8IRKp4kRrcxO3plRPKkQ=', 'Content-Length': '0'} 2023-05-24T21:34:24.285 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T21:34:24.286 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T21:34:26.576 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~303s 2023-05-24T21:34:26.576 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2023-05-24T21:34:27.178 INFO:tasks.ceph.c1.osd.0:Sent signal 15 2023-05-24T21:34:27.179 INFO:tasks.ceph.c1.osd.1:Sent signal 15 2023-05-24T21:34:27.179 INFO:tasks.ceph.c1.osd.2:Sent signal 15 2023-05-24T21:34:27.180 INFO:tasks.ceph.c1.mon.a:Sent signal 15 2023-05-24T21:34:27.180 INFO:tasks.rgw.c1.client.1:Sent signal 15 2023-05-24T21:34:27.180 INFO:tasks.ceph.c1.mgr.x:Sent signal 15 2023-05-24T21:34:27.181 INFO:tasks.ceph.c1.osd.0.smithi071.stderr:2023-05-24T21:34:27.174+0000 7f2d1f1bb700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c1 -i 0 (PID: 104682) UID: 0 2023-05-24T21:34:27.181 INFO:tasks.ceph.c1.osd.0.smithi071.stderr:2023-05-24T21:34:27.174+0000 7f2d1f1bb700 -1 osd.0 68 *** Got signal Terminated *** 2023-05-24T21:34:27.181 INFO:tasks.ceph.c1.osd.0.smithi071.stderr:2023-05-24T21:34:27.174+0000 7f2d1f1bb700 -1 osd.0 68 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-05-24T21:34:27.182 INFO:tasks.ceph.c1.osd.1.smithi071.stderr:2023-05-24T21:34:27.174+0000 7fa0ae6a2700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c1 -i 1 (PID: 104681) UID: 0 2023-05-24T21:34:27.182 INFO:tasks.ceph.c1.osd.1.smithi071.stderr:2023-05-24T21:34:27.174+0000 7fa0ae6a2700 -1 osd.1 68 *** Got signal Terminated *** 2023-05-24T21:34:27.182 INFO:tasks.ceph.c1.osd.1.smithi071.stderr:2023-05-24T21:34:27.174+0000 7fa0ae6a2700 -1 osd.1 68 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-05-24T21:34:27.183 INFO:tasks.ceph.c1.osd.2.smithi071.stderr:2023-05-24T21:34:27.174+0000 7fb12b90a700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c1 -i 2 (PID: 104687) UID: 0 2023-05-24T21:34:27.183 INFO:tasks.ceph.c1.osd.2.smithi071.stderr:2023-05-24T21:34:27.174+0000 7fb12b90a700 -1 osd.2 68 *** Got signal Terminated *** 2023-05-24T21:34:27.183 INFO:tasks.ceph.c1.osd.2.smithi071.stderr:2023-05-24T21:34:27.174+0000 7fb12b90a700 -1 osd.2 68 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-05-24T21:34:27.183 INFO:tasks.ceph.c1.mon.a.smithi071.stderr:2023-05-24T21:34:27.175+0000 7f1be8674700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster c1 -i a (PID: 104273) UID: 0 2023-05-24T21:34:27.184 INFO:tasks.ceph.c1.mon.a.smithi071.stderr:2023-05-24T21:34:27.175+0000 7f1be8674700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-05-24T21:34:27.184 INFO:tasks.rgw.c1.client.1.smithi071.stdout:2023-05-24T21:34:27.175+0000 7f1a6ca56700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 113375) UID: 0 2023-05-24T21:34:27.184 INFO:tasks.rgw.c1.client.1.smithi071.stdout:2023-05-24T21:34:27.175+0000 7f1a70cbf7c0 -1 shutting down 2023-05-24T21:34:27.185 INFO:tasks.ceph.c1.mgr.x.smithi071.stderr:2023-05-24T21:34:27.175+0000 7fa4fc32e700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster c1 -i x (PID: 104276) UID: 0 2023-05-24T21:34:27.185 INFO:tasks.ceph.c1.mgr.x.smithi071.stderr:2023-05-24T21:34:27.175+0000 7fa4fc32e700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-05-24T21:34:30.685 DEBUG:boto:Token: None 2023-05-24T21:34:30.686 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:34:03 GMT /qskrps-53/ 2023-05-24T21:34:30.686 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:vIo6xRa8IRKp4kRrcxO3plRPKkQ= 2023-05-24T21:34:30.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:34:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:vIo6xRa8IRKp4kRrcxO3plRPKkQ=', 'Content-Length': '0'} 2023-05-24T21:34:30.688 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T21:34:30.688 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T21:35:05.716 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_index_log_trim ... ERROR 2023-05-24T21:35:05.717 INFO:rgw_multi.tests:create bucket zone=a1 name=qskrps-54 2023-05-24T21:35:05.717 DEBUG:boto:path=/qskrps-54/ 2023-05-24T21:35:05.718 DEBUG:boto:auth_path=/qskrps-54/ 2023-05-24T21:35:05.718 DEBUG:boto:Method: PUT 2023-05-24T21:35:05.718 DEBUG:boto:Path: /qskrps-54/ 2023-05-24T21:35:05.719 DEBUG:boto:Data: 2023-05-24T21:35:05.719 DEBUG:boto:Headers: {} 2023-05-24T21:35:05.719 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:35:05.720 DEBUG:boto:Port: 8000 2023-05-24T21:35:05.720 DEBUG:boto:Params: {} 2023-05-24T21:35:05.720 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T21:35:05.721 DEBUG:boto:Token: None 2023-05-24T21:35:05.721 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:35:05 GMT /qskrps-54/ 2023-05-24T21:35:05.722 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ZxBcg/djziioP4QWcmbU9TXsRBE= 2023-05-24T21:35:05.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:35:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ZxBcg/djziioP4QWcmbU9TXsRBE=', 'Content-Length': '0'} 2023-05-24T21:35:05.724 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T21:35:05.724 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T21:35:06.641 DEBUG:boto:Token: None 2023-05-24T21:35:06.642 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:35:05 GMT /qskrps-54/ 2023-05-24T21:35:06.643 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ZxBcg/djziioP4QWcmbU9TXsRBE= 2023-05-24T21:35:06.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:35:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ZxBcg/djziioP4QWcmbU9TXsRBE=', 'Content-Length': '0'} 2023-05-24T21:35:06.644 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T21:35:06.645 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T21:35:08.082 DEBUG:boto:Token: None 2023-05-24T21:35:08.082 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:35:05 GMT /qskrps-54/ 2023-05-24T21:35:08.083 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ZxBcg/djziioP4QWcmbU9TXsRBE= 2023-05-24T21:35:08.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:35:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ZxBcg/djziioP4QWcmbU9TXsRBE=', 'Content-Length': '0'} 2023-05-24T21:35:08.084 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T21:35:08.085 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T21:35:10.530 DEBUG:boto:Token: None 2023-05-24T21:35:10.531 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:35:05 GMT /qskrps-54/ 2023-05-24T21:35:10.531 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ZxBcg/djziioP4QWcmbU9TXsRBE= 2023-05-24T21:35:10.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:35:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ZxBcg/djziioP4QWcmbU9TXsRBE=', 'Content-Length': '0'} 2023-05-24T21:35:10.533 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T21:35:10.533 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T21:35:15.963 DEBUG:boto:Token: None 2023-05-24T21:35:15.964 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:35:05 GMT /qskrps-54/ 2023-05-24T21:35:15.964 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ZxBcg/djziioP4QWcmbU9TXsRBE= 2023-05-24T21:35:15.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:35:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ZxBcg/djziioP4QWcmbU9TXsRBE=', 'Content-Length': '0'} 2023-05-24T21:35:15.966 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T21:35:15.967 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T21:35:17.277 DEBUG:boto:Token: None 2023-05-24T21:35:17.278 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:35:05 GMT /qskrps-54/ 2023-05-24T21:35:17.278 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ZxBcg/djziioP4QWcmbU9TXsRBE= 2023-05-24T21:35:17.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:35:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ZxBcg/djziioP4QWcmbU9TXsRBE=', 'Content-Length': '0'} 2023-05-24T21:35:17.280 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T21:35:17.280 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T21:35:19.167 DEBUG:boto:Token: None 2023-05-24T21:35:19.168 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:35:05 GMT /qskrps-54/ 2023-05-24T21:35:19.168 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:ZxBcg/djziioP4QWcmbU9TXsRBE= 2023-05-24T21:35:19.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:35:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ZxBcg/djziioP4QWcmbU9TXsRBE=', 'Content-Length': '0'} 2023-05-24T21:35:19.170 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T21:35:19.171 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T21:35:53.340 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_index_log_trim ... ERROR 2023-05-24T21:35:53.341 DEBUG:boto:path=/qskrps-55/ 2023-05-24T21:35:53.342 DEBUG:boto:auth_path=/qskrps-55/ 2023-05-24T21:35:53.342 DEBUG:boto:Method: PUT 2023-05-24T21:35:53.343 DEBUG:boto:Path: /qskrps-55/ 2023-05-24T21:35:53.343 DEBUG:boto:Data: 2023-05-24T21:35:53.343 DEBUG:boto:Headers: {} 2023-05-24T21:35:53.344 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:35:53.344 DEBUG:boto:Port: 8000 2023-05-24T21:35:53.344 DEBUG:boto:Params: {} 2023-05-24T21:35:53.345 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T21:35:53.345 DEBUG:boto:Token: None 2023-05-24T21:35:53.345 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:35:53 GMT /qskrps-55/ 2023-05-24T21:35:53.346 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:hnePDUJz1zw2lpVHAIzITPuAL7I= 2023-05-24T21:35:53.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:35:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:hnePDUJz1zw2lpVHAIzITPuAL7I=', 'Content-Length': '0'} 2023-05-24T21:35:53.348 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T21:35:53.348 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T21:35:54.191 DEBUG:boto:Token: None 2023-05-24T21:35:54.191 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:35:53 GMT /qskrps-55/ 2023-05-24T21:35:54.192 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:hnePDUJz1zw2lpVHAIzITPuAL7I= 2023-05-24T21:35:54.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:35:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:hnePDUJz1zw2lpVHAIzITPuAL7I=', 'Content-Length': '0'} 2023-05-24T21:35:54.194 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T21:35:54.194 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T21:35:54.972 DEBUG:boto:Token: None 2023-05-24T21:35:54.972 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:35:53 GMT /qskrps-55/ 2023-05-24T21:35:54.973 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:hnePDUJz1zw2lpVHAIzITPuAL7I= 2023-05-24T21:35:54.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:35:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:hnePDUJz1zw2lpVHAIzITPuAL7I=', 'Content-Length': '0'} 2023-05-24T21:35:54.974 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T21:35:54.975 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T21:35:56.331 DEBUG:boto:Token: None 2023-05-24T21:35:56.332 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:35:53 GMT /qskrps-55/ 2023-05-24T21:35:56.332 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:hnePDUJz1zw2lpVHAIzITPuAL7I= 2023-05-24T21:35:56.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:35:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:hnePDUJz1zw2lpVHAIzITPuAL7I=', 'Content-Length': '0'} 2023-05-24T21:35:56.334 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T21:35:56.334 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T21:35:56.835 DEBUG:boto:Token: None 2023-05-24T21:35:56.836 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:35:53 GMT /qskrps-55/ 2023-05-24T21:35:56.836 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:hnePDUJz1zw2lpVHAIzITPuAL7I= 2023-05-24T21:35:56.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:35:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:hnePDUJz1zw2lpVHAIzITPuAL7I=', 'Content-Length': '0'} 2023-05-24T21:35:56.838 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T21:35:56.838 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T21:35:59.054 DEBUG:boto:Token: None 2023-05-24T21:35:59.054 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:35:53 GMT /qskrps-55/ 2023-05-24T21:35:59.055 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:hnePDUJz1zw2lpVHAIzITPuAL7I= 2023-05-24T21:35:59.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:35:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:hnePDUJz1zw2lpVHAIzITPuAL7I=', 'Content-Length': '0'} 2023-05-24T21:35:59.056 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T21:35:59.057 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T21:36:13.643 DEBUG:boto:Token: None 2023-05-24T21:36:13.643 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:35:53 GMT /qskrps-55/ 2023-05-24T21:36:13.644 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:hnePDUJz1zw2lpVHAIzITPuAL7I= 2023-05-24T21:36:13.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:35:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:hnePDUJz1zw2lpVHAIzITPuAL7I=', 'Content-Length': '0'} 2023-05-24T21:36:13.646 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T21:36:13.646 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T21:36:25.260 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_incremental ... ERROR 2023-05-24T21:36:25.261 DEBUG:boto:path=/qskrps-56/ 2023-05-24T21:36:25.262 DEBUG:boto:auth_path=/qskrps-56/ 2023-05-24T21:36:25.262 DEBUG:boto:Method: PUT 2023-05-24T21:36:25.262 DEBUG:boto:Path: /qskrps-56/ 2023-05-24T21:36:25.263 DEBUG:boto:Data: 2023-05-24T21:36:25.263 DEBUG:boto:Headers: {} 2023-05-24T21:36:25.263 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T21:36:25.264 DEBUG:boto:Port: 8000 2023-05-24T21:36:25.264 DEBUG:boto:Params: {} 2023-05-24T21:36:25.264 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T21:36:25.265 DEBUG:boto:Token: None 2023-05-24T21:36:25.265 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:36:25 GMT /qskrps-56/ 2023-05-24T21:36:25.266 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:WiWg5nhzoCRWqiQ23jKslJ0wk3k= 2023-05-24T21:36:25.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:36:25 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WiWg5nhzoCRWqiQ23jKslJ0wk3k=', 'Content-Length': '0'} 2023-05-24T21:36:25.267 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T21:36:25.268 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T21:36:25.797 DEBUG:boto:Token: None 2023-05-24T21:36:25.798 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:36:25 GMT /qskrps-56/ 2023-05-24T21:36:25.798 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:WiWg5nhzoCRWqiQ23jKslJ0wk3k= 2023-05-24T21:36:25.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:36:25 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WiWg5nhzoCRWqiQ23jKslJ0wk3k=', 'Content-Length': '0'} 2023-05-24T21:36:25.800 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T21:36:25.801 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T21:36:25.958 DEBUG:boto:Token: None 2023-05-24T21:36:25.958 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:36:25 GMT /qskrps-56/ 2023-05-24T21:36:25.959 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:WiWg5nhzoCRWqiQ23jKslJ0wk3k= 2023-05-24T21:36:25.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:36:25 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WiWg5nhzoCRWqiQ23jKslJ0wk3k=', 'Content-Length': '0'} 2023-05-24T21:36:25.960 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T21:36:25.961 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T21:36:27.723 DEBUG:boto:Token: None 2023-05-24T21:36:27.724 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:36:25 GMT /qskrps-56/ 2023-05-24T21:36:27.724 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:WiWg5nhzoCRWqiQ23jKslJ0wk3k= 2023-05-24T21:36:27.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:36:25 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WiWg5nhzoCRWqiQ23jKslJ0wk3k=', 'Content-Length': '0'} 2023-05-24T21:36:27.726 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T21:36:27.727 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T21:36:28.121 DEBUG:boto:Token: None 2023-05-24T21:36:28.122 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:36:25 GMT /qskrps-56/ 2023-05-24T21:36:28.122 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:WiWg5nhzoCRWqiQ23jKslJ0wk3k= 2023-05-24T21:36:28.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:36:25 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WiWg5nhzoCRWqiQ23jKslJ0wk3k=', 'Content-Length': '0'} 2023-05-24T21:36:28.124 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T21:36:28.124 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T21:36:37.032 DEBUG:boto:Token: None 2023-05-24T21:36:37.033 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:36:25 GMT /qskrps-56/ 2023-05-24T21:36:37.033 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:WiWg5nhzoCRWqiQ23jKslJ0wk3k= 2023-05-24T21:36:37.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:36:25 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WiWg5nhzoCRWqiQ23jKslJ0wk3k=', 'Content-Length': '0'} 2023-05-24T21:36:37.035 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T21:36:37.036 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T21:36:59.407 DEBUG:boto:Token: None 2023-05-24T21:36:59.408 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 21:36:25 GMT /qskrps-56/ 2023-05-24T21:36:59.408 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:WiWg5nhzoCRWqiQ23jKslJ0wk3k= 2023-05-24T21:36:59.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:36:25 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WiWg5nhzoCRWqiQ23jKslJ0wk3k=', 'Content-Length': '0'} 2023-05-24T21:36:59.410 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T21:36:59.410 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T21:37:20.672 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_full ... ERROR 2023-05-24T21:37:20.673 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:37:20.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T21:37:20.674 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T21:37:20.731 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T21:37:20.731 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T21:42:20.732 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:42:20.726+0000 7f812a6bf780 0 monclient(hunting): authenticate timed out after 300 2023-05-24T21:47:20.733 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:47:20.727+0000 7f812a6bf780 0 monclient(hunting): authenticate timed out after 300 2023-05-24T21:52:20.733 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:52:20.728+0000 7f812a6bf780 0 monclient(hunting): authenticate timed out after 300 2023-05-24T21:57:20.734 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T21:57:20.728+0000 7f812a6bf780 0 monclient(hunting): authenticate timed out after 300 2023-05-24T22:02:20.734 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T22:02:20.729+0000 7f812a6bf780 0 monclient(hunting): authenticate timed out after 300 2023-05-24T22:07:20.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T22:07:20.729+0000 7f812a6bf780 0 monclient(hunting): authenticate timed out after 300 2023-05-24T22:12:20.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T22:12:20.729+0000 7f812a6bf780 0 monclient(hunting): authenticate timed out after 300 2023-05-24T22:17:20.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T22:17:20.730+0000 7f812a6bf780 0 monclient(hunting): authenticate timed out after 300 2023-05-24T22:22:20.736 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T22:22:20.730+0000 7f812a6bf780 0 monclient(hunting): authenticate timed out after 300 2023-05-24T22:27:20.737 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T22:27:20.731+0000 7f812a6bf780 0 monclient(hunting): authenticate timed out after 300 2023-05-24T22:27:20.737 INFO:teuthology.orchestra.run.smithi071.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-05-24T22:27:20.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-24T22:27:20.740 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_creation_time ... FAIL 2023-05-24T22:27:20.741 DEBUG:boto:path=/qskrps-57/ 2023-05-24T22:27:20.742 DEBUG:boto:auth_path=/qskrps-57/ 2023-05-24T22:27:20.742 DEBUG:boto:Method: PUT 2023-05-24T22:27:20.743 DEBUG:boto:Path: /qskrps-57/ 2023-05-24T22:27:20.743 DEBUG:boto:Data: 2023-05-24T22:27:20.743 DEBUG:boto:Headers: {} 2023-05-24T22:27:20.744 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T22:27:20.744 DEBUG:boto:Port: 8000 2023-05-24T22:27:20.744 DEBUG:boto:Params: {} 2023-05-24T22:27:20.745 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:27:20.745 DEBUG:boto:Token: None 2023-05-24T22:27:20.746 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:27:20 GMT /qskrps-57/ 2023-05-24T22:27:20.746 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:QfljcrHW4xN2QSDQOfwOQOu+x8s= 2023-05-24T22:27:20.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:27:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:QfljcrHW4xN2QSDQOfwOQOu+x8s=', 'Content-Length': '0'} 2023-05-24T22:27:20.749 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:27:20.749 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:27:20.905 DEBUG:boto:Token: None 2023-05-24T22:27:20.905 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:27:20 GMT /qskrps-57/ 2023-05-24T22:27:20.906 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:QfljcrHW4xN2QSDQOfwOQOu+x8s= 2023-05-24T22:27:20.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:27:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:QfljcrHW4xN2QSDQOfwOQOu+x8s=', 'Content-Length': '0'} 2023-05-24T22:27:20.907 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:27:20.908 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:27:21.653 DEBUG:boto:Token: None 2023-05-24T22:27:21.654 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:27:20 GMT /qskrps-57/ 2023-05-24T22:27:21.654 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:QfljcrHW4xN2QSDQOfwOQOu+x8s= 2023-05-24T22:27:21.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:27:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:QfljcrHW4xN2QSDQOfwOQOu+x8s=', 'Content-Length': '0'} 2023-05-24T22:27:21.656 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:27:21.657 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:27:25.048 DEBUG:boto:Token: None 2023-05-24T22:27:25.049 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:27:20 GMT /qskrps-57/ 2023-05-24T22:27:25.049 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:QfljcrHW4xN2QSDQOfwOQOu+x8s= 2023-05-24T22:27:25.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:27:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:QfljcrHW4xN2QSDQOfwOQOu+x8s=', 'Content-Length': '0'} 2023-05-24T22:27:25.051 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:27:25.052 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:27:26.144 DEBUG:boto:Token: None 2023-05-24T22:27:26.144 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:27:20 GMT /qskrps-57/ 2023-05-24T22:27:26.145 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:QfljcrHW4xN2QSDQOfwOQOu+x8s= 2023-05-24T22:27:26.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:27:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:QfljcrHW4xN2QSDQOfwOQOu+x8s=', 'Content-Length': '0'} 2023-05-24T22:27:26.147 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:27:26.147 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:27:34.791 DEBUG:boto:Token: None 2023-05-24T22:27:34.792 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:27:20 GMT /qskrps-57/ 2023-05-24T22:27:34.792 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:QfljcrHW4xN2QSDQOfwOQOu+x8s= 2023-05-24T22:27:34.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:27:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:QfljcrHW4xN2QSDQOfwOQOu+x8s=', 'Content-Length': '0'} 2023-05-24T22:27:34.794 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:27:34.795 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:27:44.785 DEBUG:boto:Token: None 2023-05-24T22:27:44.785 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:27:20 GMT /qskrps-57/ 2023-05-24T22:27:44.786 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:QfljcrHW4xN2QSDQOfwOQOu+x8s= 2023-05-24T22:27:44.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:27:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:QfljcrHW4xN2QSDQOfwOQOu+x8s=', 'Content-Length': '0'} 2023-05-24T22:27:44.788 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:27:44.788 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:28:30.083 INFO:tasks.rgw_multisite_tests:Create several generations of objects, then run bucket sync ... ERROR 2023-05-24T22:28:30.084 DEBUG:boto:path=/qskrps-58/ 2023-05-24T22:28:30.084 DEBUG:boto:auth_path=/qskrps-58/ 2023-05-24T22:28:30.085 DEBUG:boto:Method: PUT 2023-05-24T22:28:30.085 DEBUG:boto:Path: /qskrps-58/ 2023-05-24T22:28:30.085 DEBUG:boto:Data: 2023-05-24T22:28:30.086 DEBUG:boto:Headers: {} 2023-05-24T22:28:30.086 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T22:28:30.086 DEBUG:boto:Port: 8000 2023-05-24T22:28:30.087 DEBUG:boto:Params: {} 2023-05-24T22:28:30.087 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:28:30.087 DEBUG:boto:Token: None 2023-05-24T22:28:30.088 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:28:30 GMT /qskrps-58/ 2023-05-24T22:28:30.088 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:AQDnqSLmNwk4FjpOhOd5x15rCIQ= 2023-05-24T22:28:30.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:28:30 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:AQDnqSLmNwk4FjpOhOd5x15rCIQ=', 'Content-Length': '0'} 2023-05-24T22:28:30.090 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:28:30.091 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:28:30.728 DEBUG:boto:Token: None 2023-05-24T22:28:30.728 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:28:30 GMT /qskrps-58/ 2023-05-24T22:28:30.729 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:AQDnqSLmNwk4FjpOhOd5x15rCIQ= 2023-05-24T22:28:30.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:28:30 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:AQDnqSLmNwk4FjpOhOd5x15rCIQ=', 'Content-Length': '0'} 2023-05-24T22:28:30.731 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:28:30.731 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:28:30.853 DEBUG:boto:Token: None 2023-05-24T22:28:30.853 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:28:30 GMT /qskrps-58/ 2023-05-24T22:28:30.854 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:AQDnqSLmNwk4FjpOhOd5x15rCIQ= 2023-05-24T22:28:30.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:28:30 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:AQDnqSLmNwk4FjpOhOd5x15rCIQ=', 'Content-Length': '0'} 2023-05-24T22:28:30.855 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:28:30.856 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:28:33.182 DEBUG:boto:Token: None 2023-05-24T22:28:33.182 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:28:30 GMT /qskrps-58/ 2023-05-24T22:28:33.183 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:AQDnqSLmNwk4FjpOhOd5x15rCIQ= 2023-05-24T22:28:33.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:28:30 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:AQDnqSLmNwk4FjpOhOd5x15rCIQ=', 'Content-Length': '0'} 2023-05-24T22:28:33.184 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:28:33.185 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:28:37.671 DEBUG:boto:Token: None 2023-05-24T22:28:37.671 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:28:30 GMT /qskrps-58/ 2023-05-24T22:28:37.672 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:AQDnqSLmNwk4FjpOhOd5x15rCIQ= 2023-05-24T22:28:37.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:28:30 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:AQDnqSLmNwk4FjpOhOd5x15rCIQ=', 'Content-Length': '0'} 2023-05-24T22:28:37.673 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:28:37.674 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:28:50.712 DEBUG:boto:Token: None 2023-05-24T22:28:50.713 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:28:30 GMT /qskrps-58/ 2023-05-24T22:28:50.713 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:AQDnqSLmNwk4FjpOhOd5x15rCIQ= 2023-05-24T22:28:50.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:28:30 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:AQDnqSLmNwk4FjpOhOd5x15rCIQ=', 'Content-Length': '0'} 2023-05-24T22:28:50.715 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:28:50.716 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:29:19.154 DEBUG:boto:Token: None 2023-05-24T22:29:19.155 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:28:30 GMT /qskrps-58/ 2023-05-24T22:29:19.155 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:AQDnqSLmNwk4FjpOhOd5x15rCIQ= 2023-05-24T22:29:19.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:28:30 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:AQDnqSLmNwk4FjpOhOd5x15rCIQ=', 'Content-Length': '0'} 2023-05-24T22:29:19.157 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:29:19.158 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:30:16.427 INFO:tasks.rgw_multisite_tests:Create several generations of objects, trash them, then run bucket sync init ... ERROR 2023-05-24T22:30:16.428 INFO:rgw_multi.tests:create role zone=a1 name=roles-qskrps-1 2023-05-24T22:30:16.429 DEBUG:boto:Method: POST 2023-05-24T22:30:16.429 DEBUG:boto:Path: / 2023-05-24T22:30:16.430 DEBUG:boto:Data: 2023-05-24T22:30:16.430 DEBUG:boto:Headers: {} 2023-05-24T22:30:16.430 DEBUG:boto:Host: smithi071.front.sepia.ceph.com 2023-05-24T22:30:16.431 DEBUG:boto:Port: 8000 2023-05-24T22:30:16.431 DEBUG:boto:Params: {'RoleName': 'roles-qskrps-1', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2023-05-24T22:30:16.431 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:30:16.432 DEBUG:boto:Token: None 2023-05-24T22:30:16.433 DEBUG:boto:CanonicalRequest: POST / host:smithi071.front.sepia.ceph.com:8000 x-amz-date:20230524T223016Z host;x-amz-date 6eda902334c0e592d2f03ea0d8914abd2dcb79ee8f570a7b5f29cce3899b188a 2023-05-24T22:30:16.434 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20230524T223016Z 20230524/front/smithi071/aws4_request d7d50c40a91b57044063d40217e986b4bf998e0934bd18825dd9e82e074ac762 2023-05-24T22:30:16.435 DEBUG:boto:Signature: d1a23842b2e02625ae4bc9f2c54bde824e17b09f672fd6426d5798f3021ed591 2023-05-24T22:30:16.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'X-Amz-Date': '20230524T223016Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=YAS2LRYUNTM46AEFZJ98/20230524/front/smithi071/aws4_request,SignedHeaders=host;x-amz-date,Signature=d1a23842b2e02625ae4bc9f2c54bde824e17b09f672fd6426d5798f3021ed591', 'Host': 'smithi071.front.sepia.ceph.com:8000'} 2023-05-24T22:30:16.437 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:30:16.438 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:30:17.341 DEBUG:boto:Token: None 2023-05-24T22:30:17.341 DEBUG:boto:CanonicalRequest: POST / host:smithi071.front.sepia.ceph.com:8000 x-amz-date:20230524T223017Z host;x-amz-date 6eda902334c0e592d2f03ea0d8914abd2dcb79ee8f570a7b5f29cce3899b188a 2023-05-24T22:30:17.342 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20230524T223017Z 20230524/front/smithi071/aws4_request 911c0b925409489cc856fe70045d7a7857132e546472aedadab27efd744ff065 2023-05-24T22:30:17.342 DEBUG:boto:Signature: 22812077c896ab902ae7d55ddfb52a3eaa5bc49db166f00775d15d9af4393ad4 2023-05-24T22:30:17.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'X-Amz-Date': '20230524T223017Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=YAS2LRYUNTM46AEFZJ98/20230524/front/smithi071/aws4_request,SignedHeaders=host;x-amz-date,Signature=22812077c896ab902ae7d55ddfb52a3eaa5bc49db166f00775d15d9af4393ad4', 'Host': 'smithi071.front.sepia.ceph.com:8000'} 2023-05-24T22:30:17.344 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:30:17.345 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:30:17.735 DEBUG:boto:Token: None 2023-05-24T22:30:17.736 DEBUG:boto:CanonicalRequest: POST / host:smithi071.front.sepia.ceph.com:8000 x-amz-date:20230524T223017Z host;x-amz-date 6eda902334c0e592d2f03ea0d8914abd2dcb79ee8f570a7b5f29cce3899b188a 2023-05-24T22:30:17.736 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20230524T223017Z 20230524/front/smithi071/aws4_request 911c0b925409489cc856fe70045d7a7857132e546472aedadab27efd744ff065 2023-05-24T22:30:17.737 DEBUG:boto:Signature: 22812077c896ab902ae7d55ddfb52a3eaa5bc49db166f00775d15d9af4393ad4 2023-05-24T22:30:17.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'X-Amz-Date': '20230524T223017Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=YAS2LRYUNTM46AEFZJ98/20230524/front/smithi071/aws4_request,SignedHeaders=host;x-amz-date,Signature=22812077c896ab902ae7d55ddfb52a3eaa5bc49db166f00775d15d9af4393ad4', 'Host': 'smithi071.front.sepia.ceph.com:8000'} 2023-05-24T22:30:17.739 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:30:17.739 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:30:19.886 DEBUG:boto:Token: None 2023-05-24T22:30:19.886 DEBUG:boto:CanonicalRequest: POST / host:smithi071.front.sepia.ceph.com:8000 x-amz-date:20230524T223019Z host;x-amz-date 6eda902334c0e592d2f03ea0d8914abd2dcb79ee8f570a7b5f29cce3899b188a 2023-05-24T22:30:19.887 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20230524T223019Z 20230524/front/smithi071/aws4_request 7562a772abf0e4c428fa28038942512b99cddec88b1c443e0933271b4df2dedd 2023-05-24T22:30:19.888 DEBUG:boto:Signature: 792dad4b2882b20b1cd6c6efd33bd14cc1734c888614ef8c21042dfb363c9b9c 2023-05-24T22:30:19.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'X-Amz-Date': '20230524T223019Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=YAS2LRYUNTM46AEFZJ98/20230524/front/smithi071/aws4_request,SignedHeaders=host;x-amz-date,Signature=792dad4b2882b20b1cd6c6efd33bd14cc1734c888614ef8c21042dfb363c9b9c', 'Host': 'smithi071.front.sepia.ceph.com:8000'} 2023-05-24T22:30:19.889 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:30:19.890 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:30:22.037 DEBUG:boto:Token: None 2023-05-24T22:30:22.037 DEBUG:boto:CanonicalRequest: POST / host:smithi071.front.sepia.ceph.com:8000 x-amz-date:20230524T223022Z host;x-amz-date 6eda902334c0e592d2f03ea0d8914abd2dcb79ee8f570a7b5f29cce3899b188a 2023-05-24T22:30:22.038 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20230524T223022Z 20230524/front/smithi071/aws4_request cb30a48d81f2ef83d23c4788a49f8b9b59deb8c74e28d44a0415ff98ece38131 2023-05-24T22:30:22.038 DEBUG:boto:Signature: bb32b65749141c6861299caff07da7cf5206458aa1ed9040bea97c802e3bf0f8 2023-05-24T22:30:22.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'X-Amz-Date': '20230524T223022Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=YAS2LRYUNTM46AEFZJ98/20230524/front/smithi071/aws4_request,SignedHeaders=host;x-amz-date,Signature=bb32b65749141c6861299caff07da7cf5206458aa1ed9040bea97c802e3bf0f8', 'Host': 'smithi071.front.sepia.ceph.com:8000'} 2023-05-24T22:30:22.040 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:30:22.041 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:30:27.516 DEBUG:boto:Token: None 2023-05-24T22:30:27.517 DEBUG:boto:CanonicalRequest: POST / host:smithi071.front.sepia.ceph.com:8000 x-amz-date:20230524T223027Z host;x-amz-date 6eda902334c0e592d2f03ea0d8914abd2dcb79ee8f570a7b5f29cce3899b188a 2023-05-24T22:30:27.517 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20230524T223027Z 20230524/front/smithi071/aws4_request e181d0d7dfbfd4802725e75d147c1c4a837b0a986a0fc4035a838645a6ce04a4 2023-05-24T22:30:27.518 DEBUG:boto:Signature: c4c8e65129a69b11fa4f586ea7eb8e3559375a6124e8872ad590a935354487eb 2023-05-24T22:30:27.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'X-Amz-Date': '20230524T223027Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=YAS2LRYUNTM46AEFZJ98/20230524/front/smithi071/aws4_request,SignedHeaders=host;x-amz-date,Signature=c4c8e65129a69b11fa4f586ea7eb8e3559375a6124e8872ad590a935354487eb', 'Host': 'smithi071.front.sepia.ceph.com:8000'} 2023-05-24T22:30:27.519 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:30:27.520 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:30:51.134 DEBUG:boto:Token: None 2023-05-24T22:30:51.135 DEBUG:boto:CanonicalRequest: POST / host:smithi071.front.sepia.ceph.com:8000 x-amz-date:20230524T223051Z host;x-amz-date 6eda902334c0e592d2f03ea0d8914abd2dcb79ee8f570a7b5f29cce3899b188a 2023-05-24T22:30:51.135 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20230524T223051Z 20230524/front/smithi071/aws4_request 95eda10695db0e8761b0126511b336b3d1b3b686e78e3c75d81e09e69db4e9a0 2023-05-24T22:30:51.135 DEBUG:boto:Signature: 5b6abcd66661200e32ffacaf843e2e76bdc3f65198f904eed72c3d1f5af5b34e 2023-05-24T22:30:51.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'X-Amz-Date': '20230524T223051Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=YAS2LRYUNTM46AEFZJ98/20230524/front/smithi071/aws4_request,SignedHeaders=host;x-amz-date,Signature=5b6abcd66661200e32ffacaf843e2e76bdc3f65198f904eed72c3d1f5af5b34e', 'Host': 'smithi071.front.sepia.ceph.com:8000'} 2023-05-24T22:30:51.137 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:30:51.138 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:31:14.534 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_role_sync ... ERROR 2023-05-24T22:31:14.535 DEBUG:boto:path=/qskrps-59/ 2023-05-24T22:31:14.535 DEBUG:boto:auth_path=/qskrps-59/ 2023-05-24T22:31:14.535 DEBUG:boto:Method: PUT 2023-05-24T22:31:14.536 DEBUG:boto:Path: /qskrps-59/ 2023-05-24T22:31:14.536 DEBUG:boto:Data: 2023-05-24T22:31:14.536 DEBUG:boto:Headers: {} 2023-05-24T22:31:14.537 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T22:31:14.537 DEBUG:boto:Port: 8000 2023-05-24T22:31:14.537 DEBUG:boto:Params: {} 2023-05-24T22:31:14.538 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:31:14.538 DEBUG:boto:Token: None 2023-05-24T22:31:14.539 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:31:14 GMT /qskrps-59/ 2023-05-24T22:31:14.539 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:BMGdGlkxu5a2MljWLy6cuIYfLtI= 2023-05-24T22:31:14.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:31:14 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:BMGdGlkxu5a2MljWLy6cuIYfLtI=', 'Content-Length': '0'} 2023-05-24T22:31:14.541 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:31:14.541 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:31:14.582 DEBUG:boto:Token: None 2023-05-24T22:31:14.583 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:31:14 GMT /qskrps-59/ 2023-05-24T22:31:14.583 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:BMGdGlkxu5a2MljWLy6cuIYfLtI= 2023-05-24T22:31:14.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:31:14 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:BMGdGlkxu5a2MljWLy6cuIYfLtI=', 'Content-Length': '0'} 2023-05-24T22:31:14.585 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:31:14.585 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:31:15.039 DEBUG:boto:Token: None 2023-05-24T22:31:15.040 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:31:14 GMT /qskrps-59/ 2023-05-24T22:31:15.040 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:BMGdGlkxu5a2MljWLy6cuIYfLtI= 2023-05-24T22:31:15.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:31:14 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:BMGdGlkxu5a2MljWLy6cuIYfLtI=', 'Content-Length': '0'} 2023-05-24T22:31:15.042 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:31:15.043 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:31:15.312 DEBUG:boto:Token: None 2023-05-24T22:31:15.313 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:31:14 GMT /qskrps-59/ 2023-05-24T22:31:15.313 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:BMGdGlkxu5a2MljWLy6cuIYfLtI= 2023-05-24T22:31:15.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:31:14 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:BMGdGlkxu5a2MljWLy6cuIYfLtI=', 'Content-Length': '0'} 2023-05-24T22:31:15.315 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:31:15.316 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:31:21.451 DEBUG:boto:Token: None 2023-05-24T22:31:21.451 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:31:14 GMT /qskrps-59/ 2023-05-24T22:31:21.452 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:BMGdGlkxu5a2MljWLy6cuIYfLtI= 2023-05-24T22:31:21.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:31:14 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:BMGdGlkxu5a2MljWLy6cuIYfLtI=', 'Content-Length': '0'} 2023-05-24T22:31:21.454 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:31:21.454 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:31:31.610 DEBUG:boto:Token: None 2023-05-24T22:31:31.610 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:31:14 GMT /qskrps-59/ 2023-05-24T22:31:31.611 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:BMGdGlkxu5a2MljWLy6cuIYfLtI= 2023-05-24T22:31:31.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:31:14 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:BMGdGlkxu5a2MljWLy6cuIYfLtI=', 'Content-Length': '0'} 2023-05-24T22:31:31.612 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:31:31.613 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:31:31.956 DEBUG:boto:Token: None 2023-05-24T22:31:31.957 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:31:14 GMT /qskrps-59/ 2023-05-24T22:31:31.957 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:BMGdGlkxu5a2MljWLy6cuIYfLtI= 2023-05-24T22:31:31.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:31:14 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:BMGdGlkxu5a2MljWLy6cuIYfLtI=', 'Content-Length': '0'} 2023-05-24T22:31:31.959 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:31:31.959 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:31:55.610 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_full_sync_after_data_sync_init ... ERROR 2023-05-24T22:31:55.612 DEBUG:boto:path=/qskrps-60/ 2023-05-24T22:31:55.612 DEBUG:boto:auth_path=/qskrps-60/ 2023-05-24T22:31:55.613 DEBUG:boto:Method: PUT 2023-05-24T22:31:55.613 DEBUG:boto:Path: /qskrps-60/ 2023-05-24T22:31:55.613 DEBUG:boto:Data: 2023-05-24T22:31:55.614 DEBUG:boto:Headers: {} 2023-05-24T22:31:55.614 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T22:31:55.614 DEBUG:boto:Port: 8000 2023-05-24T22:31:55.615 DEBUG:boto:Params: {} 2023-05-24T22:31:55.615 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:31:55.616 DEBUG:boto:Token: None 2023-05-24T22:31:55.616 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:31:55 GMT /qskrps-60/ 2023-05-24T22:31:55.616 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:PK7RnQU5YnNza4ZAb48nmLlQLHE= 2023-05-24T22:31:55.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:31:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PK7RnQU5YnNza4ZAb48nmLlQLHE=', 'Content-Length': '0'} 2023-05-24T22:31:55.619 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:31:55.619 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:31:55.982 DEBUG:boto:Token: None 2023-05-24T22:31:55.983 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:31:55 GMT /qskrps-60/ 2023-05-24T22:31:55.983 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:PK7RnQU5YnNza4ZAb48nmLlQLHE= 2023-05-24T22:31:55.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:31:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PK7RnQU5YnNza4ZAb48nmLlQLHE=', 'Content-Length': '0'} 2023-05-24T22:31:55.985 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:31:55.986 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:31:56.608 DEBUG:boto:Token: None 2023-05-24T22:31:56.609 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:31:55 GMT /qskrps-60/ 2023-05-24T22:31:56.609 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:PK7RnQU5YnNza4ZAb48nmLlQLHE= 2023-05-24T22:31:56.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:31:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PK7RnQU5YnNza4ZAb48nmLlQLHE=', 'Content-Length': '0'} 2023-05-24T22:31:56.611 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:31:56.612 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:31:56.936 DEBUG:boto:Token: None 2023-05-24T22:31:56.937 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:31:55 GMT /qskrps-60/ 2023-05-24T22:31:56.938 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:PK7RnQU5YnNza4ZAb48nmLlQLHE= 2023-05-24T22:31:56.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:31:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PK7RnQU5YnNza4ZAb48nmLlQLHE=', 'Content-Length': '0'} 2023-05-24T22:31:56.939 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:31:56.940 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:31:58.853 DEBUG:boto:Token: None 2023-05-24T22:31:58.854 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:31:55 GMT /qskrps-60/ 2023-05-24T22:31:58.854 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:PK7RnQU5YnNza4ZAb48nmLlQLHE= 2023-05-24T22:31:58.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:31:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PK7RnQU5YnNza4ZAb48nmLlQLHE=', 'Content-Length': '0'} 2023-05-24T22:31:58.856 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:31:58.857 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:32:10.940 DEBUG:boto:Token: None 2023-05-24T22:32:10.941 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:31:55 GMT /qskrps-60/ 2023-05-24T22:32:10.942 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:PK7RnQU5YnNza4ZAb48nmLlQLHE= 2023-05-24T22:32:10.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:31:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PK7RnQU5YnNza4ZAb48nmLlQLHE=', 'Content-Length': '0'} 2023-05-24T22:32:10.944 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:32:10.944 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:32:24.504 DEBUG:boto:Token: None 2023-05-24T22:32:24.505 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:31:55 GMT /qskrps-60/ 2023-05-24T22:32:24.505 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:PK7RnQU5YnNza4ZAb48nmLlQLHE= 2023-05-24T22:32:24.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:31:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PK7RnQU5YnNza4ZAb48nmLlQLHE=', 'Content-Length': '0'} 2023-05-24T22:32:24.507 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:32:24.508 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:33:14.268 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_full_sync_after_data_sync_init ... ERROR 2023-05-24T22:33:14.269 DEBUG:boto:path=/qskrps-61/ 2023-05-24T22:33:14.270 DEBUG:boto:auth_path=/qskrps-61/ 2023-05-24T22:33:14.270 DEBUG:boto:Method: PUT 2023-05-24T22:33:14.271 DEBUG:boto:Path: /qskrps-61/ 2023-05-24T22:33:14.271 DEBUG:boto:Data: 2023-05-24T22:33:14.271 DEBUG:boto:Headers: {} 2023-05-24T22:33:14.272 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T22:33:14.272 DEBUG:boto:Port: 8000 2023-05-24T22:33:14.272 DEBUG:boto:Params: {} 2023-05-24T22:33:14.273 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:33:14.273 DEBUG:boto:Token: None 2023-05-24T22:33:14.273 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:33:14 GMT /qskrps-61/ 2023-05-24T22:33:14.274 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:qEk59GYgwLFY3P5e/oa3vlt+F9A= 2023-05-24T22:33:14.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:33:14 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qEk59GYgwLFY3P5e/oa3vlt+F9A=', 'Content-Length': '0'} 2023-05-24T22:33:14.276 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:33:14.276 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:33:15.091 DEBUG:boto:Token: None 2023-05-24T22:33:15.092 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:33:14 GMT /qskrps-61/ 2023-05-24T22:33:15.093 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:qEk59GYgwLFY3P5e/oa3vlt+F9A= 2023-05-24T22:33:15.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:33:14 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qEk59GYgwLFY3P5e/oa3vlt+F9A=', 'Content-Length': '0'} 2023-05-24T22:33:15.094 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:33:15.095 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:33:15.478 DEBUG:boto:Token: None 2023-05-24T22:33:15.479 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:33:14 GMT /qskrps-61/ 2023-05-24T22:33:15.479 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:qEk59GYgwLFY3P5e/oa3vlt+F9A= 2023-05-24T22:33:15.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:33:14 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qEk59GYgwLFY3P5e/oa3vlt+F9A=', 'Content-Length': '0'} 2023-05-24T22:33:15.481 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:33:15.482 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:33:15.678 DEBUG:boto:Token: None 2023-05-24T22:33:15.678 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:33:14 GMT /qskrps-61/ 2023-05-24T22:33:15.679 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:qEk59GYgwLFY3P5e/oa3vlt+F9A= 2023-05-24T22:33:15.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:33:14 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qEk59GYgwLFY3P5e/oa3vlt+F9A=', 'Content-Length': '0'} 2023-05-24T22:33:15.680 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:33:15.681 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:33:19.691 DEBUG:boto:Token: None 2023-05-24T22:33:19.692 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:33:14 GMT /qskrps-61/ 2023-05-24T22:33:19.692 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:qEk59GYgwLFY3P5e/oa3vlt+F9A= 2023-05-24T22:33:19.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:33:14 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qEk59GYgwLFY3P5e/oa3vlt+F9A=', 'Content-Length': '0'} 2023-05-24T22:33:19.694 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:33:19.694 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:33:22.554 DEBUG:boto:Token: None 2023-05-24T22:33:22.555 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:33:14 GMT /qskrps-61/ 2023-05-24T22:33:22.555 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:qEk59GYgwLFY3P5e/oa3vlt+F9A= 2023-05-24T22:33:22.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:33:14 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qEk59GYgwLFY3P5e/oa3vlt+F9A=', 'Content-Length': '0'} 2023-05-24T22:33:22.557 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:33:22.557 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:33:39.361 DEBUG:boto:Token: None 2023-05-24T22:33:39.362 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:33:14 GMT /qskrps-61/ 2023-05-24T22:33:39.362 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:qEk59GYgwLFY3P5e/oa3vlt+F9A= 2023-05-24T22:33:39.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:33:14 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qEk59GYgwLFY3P5e/oa3vlt+F9A=', 'Content-Length': '0'} 2023-05-24T22:33:39.364 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:33:39.364 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:34:25.007 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_incremental_sync_after_data_sync_init ... ERROR 2023-05-24T22:34:25.008 DEBUG:boto:path=/qskrps-62/ 2023-05-24T22:34:25.009 DEBUG:boto:auth_path=/qskrps-62/ 2023-05-24T22:34:25.009 DEBUG:boto:Method: PUT 2023-05-24T22:34:25.010 DEBUG:boto:Path: /qskrps-62/ 2023-05-24T22:34:25.010 DEBUG:boto:Data: 2023-05-24T22:34:25.010 DEBUG:boto:Headers: {} 2023-05-24T22:34:25.011 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T22:34:25.011 DEBUG:boto:Port: 8000 2023-05-24T22:34:25.011 DEBUG:boto:Params: {} 2023-05-24T22:34:25.012 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:34:25.012 DEBUG:boto:Token: None 2023-05-24T22:34:25.012 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:34:25 GMT /qskrps-62/ 2023-05-24T22:34:25.013 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:TrGvGlCnXHV4QKtbiP9GiAWe/qk= 2023-05-24T22:34:25.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:34:25 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:TrGvGlCnXHV4QKtbiP9GiAWe/qk=', 'Content-Length': '0'} 2023-05-24T22:34:25.015 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:34:25.015 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:34:25.993 DEBUG:boto:Token: None 2023-05-24T22:34:25.993 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:34:25 GMT /qskrps-62/ 2023-05-24T22:34:25.994 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:TrGvGlCnXHV4QKtbiP9GiAWe/qk= 2023-05-24T22:34:25.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:34:25 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:TrGvGlCnXHV4QKtbiP9GiAWe/qk=', 'Content-Length': '0'} 2023-05-24T22:34:25.996 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:34:25.996 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:34:26.723 DEBUG:boto:Token: None 2023-05-24T22:34:26.724 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:34:25 GMT /qskrps-62/ 2023-05-24T22:34:26.724 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:TrGvGlCnXHV4QKtbiP9GiAWe/qk= 2023-05-24T22:34:26.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:34:25 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:TrGvGlCnXHV4QKtbiP9GiAWe/qk=', 'Content-Length': '0'} 2023-05-24T22:34:26.726 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:34:26.726 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:34:27.491 DEBUG:boto:Token: None 2023-05-24T22:34:27.492 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:34:25 GMT /qskrps-62/ 2023-05-24T22:34:27.492 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:TrGvGlCnXHV4QKtbiP9GiAWe/qk= 2023-05-24T22:34:27.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:34:25 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:TrGvGlCnXHV4QKtbiP9GiAWe/qk=', 'Content-Length': '0'} 2023-05-24T22:34:27.494 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:34:27.494 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:34:28.875 DEBUG:boto:Token: None 2023-05-24T22:34:28.876 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:34:25 GMT /qskrps-62/ 2023-05-24T22:34:28.876 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:TrGvGlCnXHV4QKtbiP9GiAWe/qk= 2023-05-24T22:34:28.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:34:25 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:TrGvGlCnXHV4QKtbiP9GiAWe/qk=', 'Content-Length': '0'} 2023-05-24T22:34:28.878 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:34:28.879 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:34:33.441 DEBUG:boto:Token: None 2023-05-24T22:34:33.442 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:34:25 GMT /qskrps-62/ 2023-05-24T22:34:33.442 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:TrGvGlCnXHV4QKtbiP9GiAWe/qk= 2023-05-24T22:34:33.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:34:25 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:TrGvGlCnXHV4QKtbiP9GiAWe/qk=', 'Content-Length': '0'} 2023-05-24T22:34:33.444 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:34:33.445 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:34:56.412 DEBUG:boto:Token: None 2023-05-24T22:34:56.413 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:34:25 GMT /qskrps-62/ 2023-05-24T22:34:56.413 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:TrGvGlCnXHV4QKtbiP9GiAWe/qk= 2023-05-24T22:34:56.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:34:25 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:TrGvGlCnXHV4QKtbiP9GiAWe/qk=', 'Content-Length': '0'} 2023-05-24T22:34:56.415 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:34:56.416 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:35:24.080 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init ... ERROR 2023-05-24T22:35:24.081 DEBUG:boto:path=/qskrps-63/ 2023-05-24T22:35:24.082 DEBUG:boto:auth_path=/qskrps-63/ 2023-05-24T22:35:24.082 DEBUG:boto:Method: PUT 2023-05-24T22:35:24.083 DEBUG:boto:Path: /qskrps-63/ 2023-05-24T22:35:24.083 DEBUG:boto:Data: 2023-05-24T22:35:24.083 DEBUG:boto:Headers: {} 2023-05-24T22:35:24.084 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-05-24T22:35:24.084 DEBUG:boto:Port: 8000 2023-05-24T22:35:24.084 DEBUG:boto:Params: {} 2023-05-24T22:35:24.085 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:35:24.085 DEBUG:boto:Token: None 2023-05-24T22:35:24.086 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:35:24 GMT /qskrps-63/ 2023-05-24T22:35:24.086 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:L92XP4kkn2SQxrKpTSx9CM9yhZM= 2023-05-24T22:35:24.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:35:24 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:L92XP4kkn2SQxrKpTSx9CM9yhZM=', 'Content-Length': '0'} 2023-05-24T22:35:24.088 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:35:24.089 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:35:25.026 DEBUG:boto:Token: None 2023-05-24T22:35:25.026 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:35:24 GMT /qskrps-63/ 2023-05-24T22:35:25.027 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:L92XP4kkn2SQxrKpTSx9CM9yhZM= 2023-05-24T22:35:25.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:35:24 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:L92XP4kkn2SQxrKpTSx9CM9yhZM=', 'Content-Length': '0'} 2023-05-24T22:35:25.028 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:35:25.029 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:35:26.322 DEBUG:boto:Token: None 2023-05-24T22:35:26.322 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:35:24 GMT /qskrps-63/ 2023-05-24T22:35:26.323 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:L92XP4kkn2SQxrKpTSx9CM9yhZM= 2023-05-24T22:35:26.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:35:24 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:L92XP4kkn2SQxrKpTSx9CM9yhZM=', 'Content-Length': '0'} 2023-05-24T22:35:26.324 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:35:26.325 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:35:26.898 DEBUG:boto:Token: None 2023-05-24T22:35:26.899 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:35:24 GMT /qskrps-63/ 2023-05-24T22:35:26.899 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:L92XP4kkn2SQxrKpTSx9CM9yhZM= 2023-05-24T22:35:26.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:35:24 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:L92XP4kkn2SQxrKpTSx9CM9yhZM=', 'Content-Length': '0'} 2023-05-24T22:35:26.901 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:35:26.902 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:35:32.163 DEBUG:boto:Token: None 2023-05-24T22:35:32.163 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:35:24 GMT /qskrps-63/ 2023-05-24T22:35:32.164 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:L92XP4kkn2SQxrKpTSx9CM9yhZM= 2023-05-24T22:35:32.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:35:24 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:L92XP4kkn2SQxrKpTSx9CM9yhZM=', 'Content-Length': '0'} 2023-05-24T22:35:32.165 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:35:32.166 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:35:46.451 DEBUG:boto:Token: None 2023-05-24T22:35:46.452 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:35:24 GMT /qskrps-63/ 2023-05-24T22:35:46.452 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:L92XP4kkn2SQxrKpTSx9CM9yhZM= 2023-05-24T22:35:46.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:35:24 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:L92XP4kkn2SQxrKpTSx9CM9yhZM=', 'Content-Length': '0'} 2023-05-24T22:35:46.454 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:35:46.455 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:35:52.998 DEBUG:boto:Token: None 2023-05-24T22:35:52.999 DEBUG:boto:StringToSign: PUT Wed, 24 May 2023 22:35:24 GMT /qskrps-63/ 2023-05-24T22:35:53.000 DEBUG:boto:Signature: AWS YAS2LRYUNTM46AEFZJ98:L92XP4kkn2SQxrKpTSx9CM9yhZM= 2023-05-24T22:35:53.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:35:24 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:L92XP4kkn2SQxrKpTSx9CM9yhZM=', 'Content-Length': '0'} 2023-05-24T22:35:53.002 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2023-05-24T22:35:53.003 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-24T22:36:19.167 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init ... ERROR 2023-05-24T22:36:19.168 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T22:36:19.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T22:36:19.169 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T22:36:19.228 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T22:36:19.229 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T22:41:19.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T22:41:19.225+0000 7f92ac2f4780 0 monclient(hunting): authenticate timed out after 300 2023-05-24T22:46:19.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T22:46:19.225+0000 7f92ac2f4780 0 monclient(hunting): authenticate timed out after 300 2023-05-24T22:51:19.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T22:51:19.226+0000 7f92ac2f4780 0 monclient(hunting): authenticate timed out after 300 2023-05-24T22:56:19.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T22:56:19.226+0000 7f92ac2f4780 0 monclient(hunting): authenticate timed out after 300 2023-05-24T23:01:19.232 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T23:01:19.227+0000 7f92ac2f4780 0 monclient(hunting): authenticate timed out after 300 2023-05-24T23:06:19.232 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T23:06:19.228+0000 7f92ac2f4780 0 monclient(hunting): authenticate timed out after 300 2023-05-24T23:11:19.233 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T23:11:19.227+0000 7f92ac2f4780 0 monclient(hunting): authenticate timed out after 300 2023-05-24T23:16:19.233 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T23:16:19.228+0000 7f92ac2f4780 0 monclient(hunting): authenticate timed out after 300 2023-05-24T23:21:19.234 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T23:21:19.229+0000 7f92ac2f4780 0 monclient(hunting): authenticate timed out after 300 2023-05-24T23:26:19.234 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T23:26:19.230+0000 7f92ac2f4780 0 monclient(hunting): authenticate timed out after 300 2023-05-24T23:26:19.235 INFO:teuthology.orchestra.run.smithi071.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-05-24T23:26:19.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-24T23:26:19.257 INFO:tasks.rgw_multisite_tests:test_sync_policy_config_zonegroup: ... FAIL 2023-05-24T23:26:19.258 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T23:26:19.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-24T23:26:19.259 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-24T23:26:19.331 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-24T23:26:19.332 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-24T23:31:19.330 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T23:31:19.326+0000 7f3853eaa780 0 monclient(hunting): authenticate timed out after 300 2023-05-24T23:36:19.331 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T23:36:19.326+0000 7f3853eaa780 0 monclient(hunting): authenticate timed out after 300 2023-05-24T23:41:19.331 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T23:41:19.327+0000 7f3853eaa780 0 monclient(hunting): authenticate timed out after 300 2023-05-24T23:46:19.333 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T23:46:19.327+0000 7f3853eaa780 0 monclient(hunting): authenticate timed out after 300 2023-05-24T23:51:19.332 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T23:51:19.328+0000 7f3853eaa780 0 monclient(hunting): authenticate timed out after 300 2023-05-24T23:56:19.333 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-24T23:56:19.328+0000 7f3853eaa780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T00:01:19.334 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T00:01:19.329+0000 7f3853eaa780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T00:06:19.336 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T00:06:19.329+0000 7f3853eaa780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T00:11:19.337 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T00:11:19.331+0000 7f3853eaa780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T00:16:19.339 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T00:16:19.331+0000 7f3853eaa780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T00:16:19.340 INFO:teuthology.orchestra.run.smithi071.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-05-25T00:16:19.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-25T00:16:19.343 INFO:tasks.rgw_multisite_tests:test_sync_flow_symmetrical_zonegroup_all: ... FAIL 2023-05-25T00:16:19.346 INFO:tasks.rgw_multisite_tests:test_sync_flow_symmetrical_zonegroup_select: ... SKIP: test_sync_flow_symmetrical_zonegroup_select skipped. Requires 3 or more zones in master zonegroup. 2023-05-25T00:16:19.347 INFO:tasks.rgw_multisite_tests:test_sync_flow_directional_zonegroup_select: ... SKIP: test_sync_flow_symmetrical_zonegroup_select skipped. Requires 3 or more zones in master zonegroup. 2023-05-25T00:16:19.349 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T00:16:19.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T00:16:19.350 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-25T00:16:19.408 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-25T00:16:19.408 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-25T00:21:19.410 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T00:21:19.403+0000 7f94180e8780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T00:26:19.412 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T00:26:19.403+0000 7f94180e8780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T00:31:19.412 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T00:31:19.404+0000 7f94180e8780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T00:36:19.415 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T00:36:19.404+0000 7f94180e8780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T00:41:19.414 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T00:41:19.405+0000 7f94180e8780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T00:46:19.415 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T00:46:19.405+0000 7f94180e8780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T00:51:19.415 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T00:51:19.406+0000 7f94180e8780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T00:56:19.416 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T00:56:19.406+0000 7f94180e8780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T01:01:19.417 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T01:01:19.406+0000 7f94180e8780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T01:06:19.417 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T01:06:19.406+0000 7f94180e8780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T01:06:19.418 INFO:teuthology.orchestra.run.smithi071.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-05-25T01:06:19.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-25T01:06:19.642 INFO:tasks.rgw_multisite_tests:test_sync_single_bucket: ... FAIL 2023-05-25T01:06:19.644 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T01:06:19.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T01:06:19.645 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-25T01:06:19.701 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-25T01:06:19.702 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-25T01:11:19.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T01:11:19.692+0000 7f015f792780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T01:16:19.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T01:16:19.693+0000 7f015f792780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T01:21:19.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T01:21:19.693+0000 7f015f792780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T01:26:19.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T01:26:19.694+0000 7f015f792780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T01:31:19.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T01:31:19.694+0000 7f015f792780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T01:36:19.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T01:36:19.695+0000 7f015f792780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T01:41:19.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T01:41:19.694+0000 7f015f792780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T01:46:19.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T01:46:19.696+0000 7f015f792780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T01:51:19.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T01:51:19.696+0000 7f015f792780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T01:56:19.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T01:56:19.697+0000 7f015f792780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T01:56:19.729 INFO:teuthology.orchestra.run.smithi071.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-05-25T01:56:19.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-25T01:56:20.058 INFO:tasks.rgw_multisite_tests:test_sync_different_buckets: ... FAIL 2023-05-25T01:56:20.059 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T01:56:20.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T01:56:20.060 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-25T01:56:20.117 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-25T01:56:20.117 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-25T02:01:20.119 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T02:01:20.115+0000 7fb9e9a76780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T02:06:20.120 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T02:06:20.116+0000 7fb9e9a76780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T02:11:20.121 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T02:11:20.116+0000 7fb9e9a76780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T02:16:20.122 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T02:16:20.117+0000 7fb9e9a76780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T02:21:20.122 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T02:21:20.117+0000 7fb9e9a76780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T02:26:20.125 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T02:26:20.119+0000 7fb9e9a76780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T02:31:20.123 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T02:31:20.119+0000 7fb9e9a76780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T02:36:20.123 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T02:36:20.120+0000 7fb9e9a76780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T02:41:20.124 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T02:41:20.120+0000 7fb9e9a76780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T02:46:20.124 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T02:46:20.120+0000 7fb9e9a76780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T02:46:20.151 INFO:teuthology.orchestra.run.smithi071.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-05-25T02:46:20.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-25T02:46:20.445 INFO:tasks.rgw_multisite_tests:test_sync_multiple_buckets_to_single: ... FAIL 2023-05-25T02:46:20.453 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T02:46:20.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T02:46:20.455 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-25T02:46:20.512 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-05-25T02:46:20.512 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-05-25T02:51:20.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T02:51:20.510+0000 7f04a241b780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T02:56:20.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T02:56:20.510+0000 7f04a241b780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:01:20.516 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T03:01:20.511+0000 7f04a241b780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:06:20.516 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T03:06:20.511+0000 7f04a241b780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:11:20.517 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T03:11:20.512+0000 7f04a241b780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:16:20.517 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T03:16:20.512+0000 7f04a241b780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:21:20.518 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T03:21:20.513+0000 7f04a241b780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:26:20.519 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T03:26:20.514+0000 7f04a241b780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:31:20.519 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T03:31:20.514+0000 7f04a241b780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:20.519 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T03:36:20.515+0000 7f04a241b780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:20.520 INFO:teuthology.orchestra.run.smithi071.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-05-25T03:36:20.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-25T03:36:20.523 INFO:tasks.rgw_multisite_tests:test_sync_single_bucket_to_multiple: ... FAIL 2023-05-25T03:36:20.524 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.524 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-05-25T03:36:20.524 INFO:tasks.rgw_multisite_tests:ERROR: rgw_multi.tests.test_bucket_index_log_trim 2023-05-25T03:36:20.525 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-05-25T03:36:20.525 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-05-25T03:36:20.526 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-05-25T03:36:20.526 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-05-25T03:36:20.527 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/../src/test/rgw/rgw_multi/tests.py", line 1310, in test_bucket_index_log_trim 2023-05-25T03:36:20.527 INFO:tasks.rgw_multisite_tests: cold_bucket = make_test_bucket() 2023-05-25T03:36:20.527 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/../src/test/rgw/rgw_multi/tests.py", line 1301, in make_test_bucket 2023-05-25T03:36:20.527 INFO:tasks.rgw_multisite_tests: bucket = zone.conn.create_bucket(name) 2023-05-25T03:36:20.528 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 618, in create_bucket 2023-05-25T03:36:20.528 INFO:tasks.rgw_multisite_tests: response = self.make_request('PUT', bucket_name, headers=headers, 2023-05-25T03:36:20.529 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 667, in make_request 2023-05-25T03:36:20.529 INFO:tasks.rgw_multisite_tests: return super(S3Connection, self).make_request( 2023-05-25T03:36:20.529 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1070, in make_request 2023-05-25T03:36:20.530 INFO:tasks.rgw_multisite_tests: return self._mexe(http_request, sender, override_num_retries, 2023-05-25T03:36:20.530 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1030, in _mexe 2023-05-25T03:36:20.530 INFO:tasks.rgw_multisite_tests: raise ex 2023-05-25T03:36:20.531 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 942, in _mexe 2023-05-25T03:36:20.531 INFO:tasks.rgw_multisite_tests: connection.request(request.method, request.path, 2023-05-25T03:36:20.531 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1234, in request 2023-05-25T03:36:20.531 INFO:tasks.rgw_multisite_tests: self._send_request(method, url, body, headers, encode_chunked) 2023-05-25T03:36:20.532 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1280, in _send_request 2023-05-25T03:36:20.532 INFO:tasks.rgw_multisite_tests: self.endheaders(body, encode_chunked=encode_chunked) 2023-05-25T03:36:20.532 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1229, in endheaders 2023-05-25T03:36:20.532 INFO:tasks.rgw_multisite_tests: self._send_output(message_body, encode_chunked=encode_chunked) 2023-05-25T03:36:20.533 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1008, in _send_output 2023-05-25T03:36:20.533 INFO:tasks.rgw_multisite_tests: self.send(msg) 2023-05-25T03:36:20.533 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 948, in send 2023-05-25T03:36:20.533 INFO:tasks.rgw_multisite_tests: self.connect() 2023-05-25T03:36:20.534 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 919, in connect 2023-05-25T03:36:20.534 INFO:tasks.rgw_multisite_tests: self.sock = self._create_connection( 2023-05-25T03:36:20.534 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/gevent/socket.py", line 101, in create_connection 2023-05-25T03:36:20.534 INFO:tasks.rgw_multisite_tests: sock.connect(sa) 2023-05-25T03:36:20.535 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 607, in connect 2023-05-25T03:36:20.535 INFO:tasks.rgw_multisite_tests: raise _SocketError(err, strerror(err)) 2023-05-25T03:36:20.535 INFO:tasks.rgw_multisite_tests:ConnectionRefusedError: [Errno 111] Connection refused 2023-05-25T03:36:20.535 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-05-25T03:36:20.536 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=qskrps-53 2023-05-25T03:36:20.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-53/ 2023-05-25T03:36:20.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-53/ 2023-05-25T03:36:20.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-25T03:36:20.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-53/ 2023-05-25T03:36:20.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:20.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:20.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:20.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:20.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:20.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.539 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.539 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.540 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.540 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:34:03 GMT 2023-05-25T03:36:20.540 INFO:tasks.rgw_multisite_tests:/qskrps-53/ 2023-05-25T03:36:20.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.541 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:vIo6xRa8IRKp4kRrcxO3plRPKkQ= 2023-05-25T03:36:20.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:34:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:vIo6xRa8IRKp4kRrcxO3plRPKkQ=', 'Content-Length': '0'} 2023-05-25T03:36:20.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.542 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.542 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.543 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.543 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:34:03 GMT 2023-05-25T03:36:20.543 INFO:tasks.rgw_multisite_tests:/qskrps-53/ 2023-05-25T03:36:20.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.544 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:vIo6xRa8IRKp4kRrcxO3plRPKkQ= 2023-05-25T03:36:20.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:34:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:vIo6xRa8IRKp4kRrcxO3plRPKkQ=', 'Content-Length': '0'} 2023-05-25T03:36:20.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.545 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~280s 2023-05-25T03:36:20.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.546 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.546 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.546 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.546 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:34:03 GMT 2023-05-25T03:36:20.547 INFO:tasks.rgw_multisite_tests:/qskrps-53/ 2023-05-25T03:36:20.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.547 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:vIo6xRa8IRKp4kRrcxO3plRPKkQ= 2023-05-25T03:36:20.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:34:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:vIo6xRa8IRKp4kRrcxO3plRPKkQ=', 'Content-Length': '0'} 2023-05-25T03:36:20.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.549 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.549 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.549 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.549 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:34:03 GMT 2023-05-25T03:36:20.550 INFO:tasks.rgw_multisite_tests:/qskrps-53/ 2023-05-25T03:36:20.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.550 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:vIo6xRa8IRKp4kRrcxO3plRPKkQ= 2023-05-25T03:36:20.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:34:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:vIo6xRa8IRKp4kRrcxO3plRPKkQ=', 'Content-Length': '0'} 2023-05-25T03:36:20.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.552 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~286s 2023-05-25T03:36:20.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.553 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.553 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.553 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.554 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:34:03 GMT 2023-05-25T03:36:20.554 INFO:tasks.rgw_multisite_tests:/qskrps-53/ 2023-05-25T03:36:20.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.554 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:vIo6xRa8IRKp4kRrcxO3plRPKkQ= 2023-05-25T03:36:20.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:34:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:vIo6xRa8IRKp4kRrcxO3plRPKkQ=', 'Content-Length': '0'} 2023-05-25T03:36:20.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.555 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~291s 2023-05-25T03:36:20.555 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~297s 2023-05-25T03:36:20.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.556 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.557 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.557 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.557 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:34:03 GMT 2023-05-25T03:36:20.557 INFO:tasks.rgw_multisite_tests:/qskrps-53/ 2023-05-25T03:36:20.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.558 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:vIo6xRa8IRKp4kRrcxO3plRPKkQ= 2023-05-25T03:36:20.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:34:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:vIo6xRa8IRKp4kRrcxO3plRPKkQ=', 'Content-Length': '0'} 2023-05-25T03:36:20.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.559 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~303s 2023-05-25T03:36:20.559 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: BARK! unmounting mounts and killing all daemons 2023-05-25T03:36:20.559 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.osd.0: INFO: Sent signal 15 2023-05-25T03:36:20.560 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.osd.1: INFO: Sent signal 15 2023-05-25T03:36:20.560 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.osd.2: INFO: Sent signal 15 2023-05-25T03:36:20.560 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.mon.a: INFO: Sent signal 15 2023-05-25T03:36:20.561 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Sent signal 15 2023-05-25T03:36:20.561 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.mgr.x: INFO: Sent signal 15 2023-05-25T03:36:20.561 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.osd.0.smithi071.stderr: INFO: 2023-05-24T21:34:27.174+0000 7f2d1f1bb700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c1 -i 0 (PID: 104682) UID: 0 2023-05-25T03:36:20.561 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.osd.0.smithi071.stderr: INFO: 2023-05-24T21:34:27.174+0000 7f2d1f1bb700 -1 osd.0 68 *** Got signal Terminated *** 2023-05-25T03:36:20.562 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.osd.0.smithi071.stderr: INFO: 2023-05-24T21:34:27.174+0000 7f2d1f1bb700 -1 osd.0 68 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-05-25T03:36:20.563 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.osd.1.smithi071.stderr: INFO: 2023-05-24T21:34:27.174+0000 7fa0ae6a2700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c1 -i 1 (PID: 104681) UID: 0 2023-05-25T03:36:20.563 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.osd.1.smithi071.stderr: INFO: 2023-05-24T21:34:27.174+0000 7fa0ae6a2700 -1 osd.1 68 *** Got signal Terminated *** 2023-05-25T03:36:20.564 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.osd.1.smithi071.stderr: INFO: 2023-05-24T21:34:27.174+0000 7fa0ae6a2700 -1 osd.1 68 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-05-25T03:36:20.565 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.osd.2.smithi071.stderr: INFO: 2023-05-24T21:34:27.174+0000 7fb12b90a700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c1 -i 2 (PID: 104687) UID: 0 2023-05-25T03:36:20.565 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.osd.2.smithi071.stderr: INFO: 2023-05-24T21:34:27.174+0000 7fb12b90a700 -1 osd.2 68 *** Got signal Terminated *** 2023-05-25T03:36:20.565 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.osd.2.smithi071.stderr: INFO: 2023-05-24T21:34:27.174+0000 7fb12b90a700 -1 osd.2 68 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-05-25T03:36:20.566 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.mon.a.smithi071.stderr: INFO: 2023-05-24T21:34:27.175+0000 7f1be8674700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster c1 -i a (PID: 104273) UID: 0 2023-05-25T03:36:20.566 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.mon.a.smithi071.stderr: INFO: 2023-05-24T21:34:27.175+0000 7f1be8674700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-05-25T03:36:20.566 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi071.stdout: INFO: 2023-05-24T21:34:27.175+0000 7f1a6ca56700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 113375) UID: 0 2023-05-25T03:36:20.567 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi071.stdout: INFO: 2023-05-24T21:34:27.175+0000 7f1a70cbf7c0 -1 shutting down 2023-05-25T03:36:20.567 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.mgr.x.smithi071.stderr: INFO: 2023-05-24T21:34:27.175+0000 7fa4fc32e700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster c1 -i x (PID: 104276) UID: 0 2023-05-25T03:36:20.567 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.mgr.x.smithi071.stderr: INFO: 2023-05-24T21:34:27.175+0000 7fa4fc32e700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-05-25T03:36:20.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.568 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.568 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.568 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.569 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:34:03 GMT 2023-05-25T03:36:20.569 INFO:tasks.rgw_multisite_tests:/qskrps-53/ 2023-05-25T03:36:20.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.569 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:vIo6xRa8IRKp4kRrcxO3plRPKkQ= 2023-05-25T03:36:20.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:34:03 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:vIo6xRa8IRKp4kRrcxO3plRPKkQ=', 'Content-Length': '0'} 2023-05-25T03:36:20.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.570 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-05-25T03:36:20.571 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.572 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-05-25T03:36:20.572 INFO:tasks.rgw_multisite_tests:ERROR: rgw_multi.tests.test_bucket_reshard_index_log_trim 2023-05-25T03:36:20.573 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-05-25T03:36:20.573 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-05-25T03:36:20.574 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-05-25T03:36:20.574 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-05-25T03:36:20.575 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/../src/test/rgw/rgw_multi/tests.py", line 1364, in test_bucket_reshard_index_log_trim 2023-05-25T03:36:20.575 INFO:tasks.rgw_multisite_tests: test_bucket = make_test_bucket() 2023-05-25T03:36:20.576 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/../src/test/rgw/rgw_multi/tests.py", line 1355, in make_test_bucket 2023-05-25T03:36:20.577 INFO:tasks.rgw_multisite_tests: bucket = zone.conn.create_bucket(name) 2023-05-25T03:36:20.577 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 618, in create_bucket 2023-05-25T03:36:20.577 INFO:tasks.rgw_multisite_tests: response = self.make_request('PUT', bucket_name, headers=headers, 2023-05-25T03:36:20.578 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 667, in make_request 2023-05-25T03:36:20.578 INFO:tasks.rgw_multisite_tests: return super(S3Connection, self).make_request( 2023-05-25T03:36:20.578 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1070, in make_request 2023-05-25T03:36:20.578 INFO:tasks.rgw_multisite_tests: return self._mexe(http_request, sender, override_num_retries, 2023-05-25T03:36:20.578 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1030, in _mexe 2023-05-25T03:36:20.579 INFO:tasks.rgw_multisite_tests: raise ex 2023-05-25T03:36:20.579 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 942, in _mexe 2023-05-25T03:36:20.579 INFO:tasks.rgw_multisite_tests: connection.request(request.method, request.path, 2023-05-25T03:36:20.580 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1234, in request 2023-05-25T03:36:20.580 INFO:tasks.rgw_multisite_tests: self._send_request(method, url, body, headers, encode_chunked) 2023-05-25T03:36:20.580 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1280, in _send_request 2023-05-25T03:36:20.580 INFO:tasks.rgw_multisite_tests: self.endheaders(body, encode_chunked=encode_chunked) 2023-05-25T03:36:20.581 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1229, in endheaders 2023-05-25T03:36:20.581 INFO:tasks.rgw_multisite_tests: self._send_output(message_body, encode_chunked=encode_chunked) 2023-05-25T03:36:20.581 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1008, in _send_output 2023-05-25T03:36:20.581 INFO:tasks.rgw_multisite_tests: self.send(msg) 2023-05-25T03:36:20.582 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 948, in send 2023-05-25T03:36:20.582 INFO:tasks.rgw_multisite_tests: self.connect() 2023-05-25T03:36:20.582 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 919, in connect 2023-05-25T03:36:20.582 INFO:tasks.rgw_multisite_tests: self.sock = self._create_connection( 2023-05-25T03:36:20.583 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/gevent/socket.py", line 101, in create_connection 2023-05-25T03:36:20.583 INFO:tasks.rgw_multisite_tests: sock.connect(sa) 2023-05-25T03:36:20.583 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 607, in connect 2023-05-25T03:36:20.583 INFO:tasks.rgw_multisite_tests: raise _SocketError(err, strerror(err)) 2023-05-25T03:36:20.584 INFO:tasks.rgw_multisite_tests:ConnectionRefusedError: [Errno 111] Connection refused 2023-05-25T03:36:20.584 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-05-25T03:36:20.584 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=qskrps-54 2023-05-25T03:36:20.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-54/ 2023-05-25T03:36:20.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-54/ 2023-05-25T03:36:20.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-25T03:36:20.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-54/ 2023-05-25T03:36:20.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:20.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:20.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:20.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:20.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:20.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.589 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.590 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.591 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.591 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:35:05 GMT 2023-05-25T03:36:20.592 INFO:tasks.rgw_multisite_tests:/qskrps-54/ 2023-05-25T03:36:20.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.593 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:ZxBcg/djziioP4QWcmbU9TXsRBE= 2023-05-25T03:36:20.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:35:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ZxBcg/djziioP4QWcmbU9TXsRBE=', 'Content-Length': '0'} 2023-05-25T03:36:20.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.597 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.597 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.597 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.598 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:35:05 GMT 2023-05-25T03:36:20.598 INFO:tasks.rgw_multisite_tests:/qskrps-54/ 2023-05-25T03:36:20.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.598 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:ZxBcg/djziioP4QWcmbU9TXsRBE= 2023-05-25T03:36:20.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:35:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ZxBcg/djziioP4QWcmbU9TXsRBE=', 'Content-Length': '0'} 2023-05-25T03:36:20.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.600 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.600 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.600 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.601 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:35:05 GMT 2023-05-25T03:36:20.601 INFO:tasks.rgw_multisite_tests:/qskrps-54/ 2023-05-25T03:36:20.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.601 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:ZxBcg/djziioP4QWcmbU9TXsRBE= 2023-05-25T03:36:20.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:35:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ZxBcg/djziioP4QWcmbU9TXsRBE=', 'Content-Length': '0'} 2023-05-25T03:36:20.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.603 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.603 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.603 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.604 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:35:05 GMT 2023-05-25T03:36:20.604 INFO:tasks.rgw_multisite_tests:/qskrps-54/ 2023-05-25T03:36:20.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.604 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:ZxBcg/djziioP4QWcmbU9TXsRBE= 2023-05-25T03:36:20.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:35:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ZxBcg/djziioP4QWcmbU9TXsRBE=', 'Content-Length': '0'} 2023-05-25T03:36:20.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.606 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.606 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.606 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.606 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:35:05 GMT 2023-05-25T03:36:20.607 INFO:tasks.rgw_multisite_tests:/qskrps-54/ 2023-05-25T03:36:20.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.607 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:ZxBcg/djziioP4QWcmbU9TXsRBE= 2023-05-25T03:36:20.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:35:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ZxBcg/djziioP4QWcmbU9TXsRBE=', 'Content-Length': '0'} 2023-05-25T03:36:20.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.609 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.609 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.609 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.609 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:35:05 GMT 2023-05-25T03:36:20.610 INFO:tasks.rgw_multisite_tests:/qskrps-54/ 2023-05-25T03:36:20.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.610 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:ZxBcg/djziioP4QWcmbU9TXsRBE= 2023-05-25T03:36:20.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:35:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ZxBcg/djziioP4QWcmbU9TXsRBE=', 'Content-Length': '0'} 2023-05-25T03:36:20.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.611 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.612 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.612 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.612 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:35:05 GMT 2023-05-25T03:36:20.613 INFO:tasks.rgw_multisite_tests:/qskrps-54/ 2023-05-25T03:36:20.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.613 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:ZxBcg/djziioP4QWcmbU9TXsRBE= 2023-05-25T03:36:20.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:35:05 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ZxBcg/djziioP4QWcmbU9TXsRBE=', 'Content-Length': '0'} 2023-05-25T03:36:20.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.614 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-05-25T03:36:20.614 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.615 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-05-25T03:36:20.615 INFO:tasks.rgw_multisite_tests:ERROR: rgw_multi.tests.test_bucket_reshard_incremental 2023-05-25T03:36:20.615 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-05-25T03:36:20.615 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-05-25T03:36:20.616 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-05-25T03:36:20.616 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-05-25T03:36:20.617 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/../src/test/rgw/rgw_multi/tests.py", line 1439, in test_bucket_reshard_incremental 2023-05-25T03:36:20.617 INFO:tasks.rgw_multisite_tests: bucket = zone.create_bucket(gen_bucket_name()) 2023-05-25T03:36:20.618 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/tasks/rgw_multi/zone_rados.py", line 55, in create_bucket 2023-05-25T03:36:20.618 INFO:tasks.rgw_multisite_tests: return self.conn.create_bucket(name) 2023-05-25T03:36:20.618 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 618, in create_bucket 2023-05-25T03:36:20.618 INFO:tasks.rgw_multisite_tests: response = self.make_request('PUT', bucket_name, headers=headers, 2023-05-25T03:36:20.619 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 667, in make_request 2023-05-25T03:36:20.619 INFO:tasks.rgw_multisite_tests: return super(S3Connection, self).make_request( 2023-05-25T03:36:20.619 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1070, in make_request 2023-05-25T03:36:20.619 INFO:tasks.rgw_multisite_tests: return self._mexe(http_request, sender, override_num_retries, 2023-05-25T03:36:20.620 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1030, in _mexe 2023-05-25T03:36:20.620 INFO:tasks.rgw_multisite_tests: raise ex 2023-05-25T03:36:20.620 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 942, in _mexe 2023-05-25T03:36:20.620 INFO:tasks.rgw_multisite_tests: connection.request(request.method, request.path, 2023-05-25T03:36:20.621 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1234, in request 2023-05-25T03:36:20.621 INFO:tasks.rgw_multisite_tests: self._send_request(method, url, body, headers, encode_chunked) 2023-05-25T03:36:20.621 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1280, in _send_request 2023-05-25T03:36:20.621 INFO:tasks.rgw_multisite_tests: self.endheaders(body, encode_chunked=encode_chunked) 2023-05-25T03:36:20.622 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1229, in endheaders 2023-05-25T03:36:20.622 INFO:tasks.rgw_multisite_tests: self._send_output(message_body, encode_chunked=encode_chunked) 2023-05-25T03:36:20.622 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1008, in _send_output 2023-05-25T03:36:20.622 INFO:tasks.rgw_multisite_tests: self.send(msg) 2023-05-25T03:36:20.622 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 948, in send 2023-05-25T03:36:20.623 INFO:tasks.rgw_multisite_tests: self.connect() 2023-05-25T03:36:20.623 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 919, in connect 2023-05-25T03:36:20.623 INFO:tasks.rgw_multisite_tests: self.sock = self._create_connection( 2023-05-25T03:36:20.623 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/gevent/socket.py", line 101, in create_connection 2023-05-25T03:36:20.624 INFO:tasks.rgw_multisite_tests: sock.connect(sa) 2023-05-25T03:36:20.624 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 607, in connect 2023-05-25T03:36:20.624 INFO:tasks.rgw_multisite_tests: raise _SocketError(err, strerror(err)) 2023-05-25T03:36:20.624 INFO:tasks.rgw_multisite_tests:ConnectionRefusedError: [Errno 111] Connection refused 2023-05-25T03:36:20.625 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-05-25T03:36:20.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-55/ 2023-05-25T03:36:20.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-55/ 2023-05-25T03:36:20.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-25T03:36:20.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-55/ 2023-05-25T03:36:20.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:20.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:20.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:20.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:20.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:20.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.628 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.628 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.628 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.629 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:35:53 GMT 2023-05-25T03:36:20.629 INFO:tasks.rgw_multisite_tests:/qskrps-55/ 2023-05-25T03:36:20.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.629 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:hnePDUJz1zw2lpVHAIzITPuAL7I= 2023-05-25T03:36:20.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:35:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:hnePDUJz1zw2lpVHAIzITPuAL7I=', 'Content-Length': '0'} 2023-05-25T03:36:20.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.631 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.631 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.631 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.631 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:35:53 GMT 2023-05-25T03:36:20.632 INFO:tasks.rgw_multisite_tests:/qskrps-55/ 2023-05-25T03:36:20.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.632 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:hnePDUJz1zw2lpVHAIzITPuAL7I= 2023-05-25T03:36:20.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:35:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:hnePDUJz1zw2lpVHAIzITPuAL7I=', 'Content-Length': '0'} 2023-05-25T03:36:20.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.634 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.634 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.634 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.634 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:35:53 GMT 2023-05-25T03:36:20.634 INFO:tasks.rgw_multisite_tests:/qskrps-55/ 2023-05-25T03:36:20.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.635 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:hnePDUJz1zw2lpVHAIzITPuAL7I= 2023-05-25T03:36:20.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:35:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:hnePDUJz1zw2lpVHAIzITPuAL7I=', 'Content-Length': '0'} 2023-05-25T03:36:20.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.636 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.637 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.637 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.637 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:35:53 GMT 2023-05-25T03:36:20.637 INFO:tasks.rgw_multisite_tests:/qskrps-55/ 2023-05-25T03:36:20.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.638 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:hnePDUJz1zw2lpVHAIzITPuAL7I= 2023-05-25T03:36:20.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:35:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:hnePDUJz1zw2lpVHAIzITPuAL7I=', 'Content-Length': '0'} 2023-05-25T03:36:20.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.639 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.639 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.640 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.640 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:35:53 GMT 2023-05-25T03:36:20.640 INFO:tasks.rgw_multisite_tests:/qskrps-55/ 2023-05-25T03:36:20.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.641 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:hnePDUJz1zw2lpVHAIzITPuAL7I= 2023-05-25T03:36:20.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:35:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:hnePDUJz1zw2lpVHAIzITPuAL7I=', 'Content-Length': '0'} 2023-05-25T03:36:20.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.642 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.642 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.643 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.643 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:35:53 GMT 2023-05-25T03:36:20.643 INFO:tasks.rgw_multisite_tests:/qskrps-55/ 2023-05-25T03:36:20.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.644 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:hnePDUJz1zw2lpVHAIzITPuAL7I= 2023-05-25T03:36:20.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:35:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:hnePDUJz1zw2lpVHAIzITPuAL7I=', 'Content-Length': '0'} 2023-05-25T03:36:20.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.645 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.646 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.646 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.646 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:35:53 GMT 2023-05-25T03:36:20.646 INFO:tasks.rgw_multisite_tests:/qskrps-55/ 2023-05-25T03:36:20.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.647 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:hnePDUJz1zw2lpVHAIzITPuAL7I= 2023-05-25T03:36:20.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:35:53 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:hnePDUJz1zw2lpVHAIzITPuAL7I=', 'Content-Length': '0'} 2023-05-25T03:36:20.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.648 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-05-25T03:36:20.648 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.648 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-05-25T03:36:20.648 INFO:tasks.rgw_multisite_tests:ERROR: rgw_multi.tests.test_bucket_reshard_full 2023-05-25T03:36:20.649 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-05-25T03:36:20.649 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-05-25T03:36:20.649 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-05-25T03:36:20.649 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-05-25T03:36:20.650 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/../src/test/rgw/rgw_multi/tests.py", line 1469, in test_bucket_reshard_full 2023-05-25T03:36:20.650 INFO:tasks.rgw_multisite_tests: bucket = zone.create_bucket(gen_bucket_name()) 2023-05-25T03:36:20.650 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/tasks/rgw_multi/zone_rados.py", line 55, in create_bucket 2023-05-25T03:36:20.650 INFO:tasks.rgw_multisite_tests: return self.conn.create_bucket(name) 2023-05-25T03:36:20.651 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 618, in create_bucket 2023-05-25T03:36:20.651 INFO:tasks.rgw_multisite_tests: response = self.make_request('PUT', bucket_name, headers=headers, 2023-05-25T03:36:20.651 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 667, in make_request 2023-05-25T03:36:20.651 INFO:tasks.rgw_multisite_tests: return super(S3Connection, self).make_request( 2023-05-25T03:36:20.652 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1070, in make_request 2023-05-25T03:36:20.652 INFO:tasks.rgw_multisite_tests: return self._mexe(http_request, sender, override_num_retries, 2023-05-25T03:36:20.652 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1030, in _mexe 2023-05-25T03:36:20.653 INFO:tasks.rgw_multisite_tests: raise ex 2023-05-25T03:36:20.653 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 942, in _mexe 2023-05-25T03:36:20.653 INFO:tasks.rgw_multisite_tests: connection.request(request.method, request.path, 2023-05-25T03:36:20.653 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1234, in request 2023-05-25T03:36:20.654 INFO:tasks.rgw_multisite_tests: self._send_request(method, url, body, headers, encode_chunked) 2023-05-25T03:36:20.654 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1280, in _send_request 2023-05-25T03:36:20.654 INFO:tasks.rgw_multisite_tests: self.endheaders(body, encode_chunked=encode_chunked) 2023-05-25T03:36:20.654 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1229, in endheaders 2023-05-25T03:36:20.655 INFO:tasks.rgw_multisite_tests: self._send_output(message_body, encode_chunked=encode_chunked) 2023-05-25T03:36:20.655 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1008, in _send_output 2023-05-25T03:36:20.655 INFO:tasks.rgw_multisite_tests: self.send(msg) 2023-05-25T03:36:20.655 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 948, in send 2023-05-25T03:36:20.656 INFO:tasks.rgw_multisite_tests: self.connect() 2023-05-25T03:36:20.656 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 919, in connect 2023-05-25T03:36:20.656 INFO:tasks.rgw_multisite_tests: self.sock = self._create_connection( 2023-05-25T03:36:20.656 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/gevent/socket.py", line 101, in create_connection 2023-05-25T03:36:20.657 INFO:tasks.rgw_multisite_tests: sock.connect(sa) 2023-05-25T03:36:20.657 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 607, in connect 2023-05-25T03:36:20.657 INFO:tasks.rgw_multisite_tests: raise _SocketError(err, strerror(err)) 2023-05-25T03:36:20.657 INFO:tasks.rgw_multisite_tests:ConnectionRefusedError: [Errno 111] Connection refused 2023-05-25T03:36:20.657 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-05-25T03:36:20.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-56/ 2023-05-25T03:36:20.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-56/ 2023-05-25T03:36:20.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-25T03:36:20.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-56/ 2023-05-25T03:36:20.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:20.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:20.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:20.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:20.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:20.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.661 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.661 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.661 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.661 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:36:25 GMT 2023-05-25T03:36:20.661 INFO:tasks.rgw_multisite_tests:/qskrps-56/ 2023-05-25T03:36:20.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.662 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:WiWg5nhzoCRWqiQ23jKslJ0wk3k= 2023-05-25T03:36:20.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:36:25 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WiWg5nhzoCRWqiQ23jKslJ0wk3k=', 'Content-Length': '0'} 2023-05-25T03:36:20.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.663 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.664 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.664 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.664 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:36:25 GMT 2023-05-25T03:36:20.664 INFO:tasks.rgw_multisite_tests:/qskrps-56/ 2023-05-25T03:36:20.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.665 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:WiWg5nhzoCRWqiQ23jKslJ0wk3k= 2023-05-25T03:36:20.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:36:25 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WiWg5nhzoCRWqiQ23jKslJ0wk3k=', 'Content-Length': '0'} 2023-05-25T03:36:20.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.666 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.666 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.667 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.667 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:36:25 GMT 2023-05-25T03:36:20.667 INFO:tasks.rgw_multisite_tests:/qskrps-56/ 2023-05-25T03:36:20.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.668 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:WiWg5nhzoCRWqiQ23jKslJ0wk3k= 2023-05-25T03:36:20.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:36:25 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WiWg5nhzoCRWqiQ23jKslJ0wk3k=', 'Content-Length': '0'} 2023-05-25T03:36:20.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.669 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.669 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.669 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.670 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:36:25 GMT 2023-05-25T03:36:20.670 INFO:tasks.rgw_multisite_tests:/qskrps-56/ 2023-05-25T03:36:20.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.670 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:WiWg5nhzoCRWqiQ23jKslJ0wk3k= 2023-05-25T03:36:20.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:36:25 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WiWg5nhzoCRWqiQ23jKslJ0wk3k=', 'Content-Length': '0'} 2023-05-25T03:36:20.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.672 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.672 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.672 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.673 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:36:25 GMT 2023-05-25T03:36:20.673 INFO:tasks.rgw_multisite_tests:/qskrps-56/ 2023-05-25T03:36:20.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.673 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:WiWg5nhzoCRWqiQ23jKslJ0wk3k= 2023-05-25T03:36:20.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:36:25 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WiWg5nhzoCRWqiQ23jKslJ0wk3k=', 'Content-Length': '0'} 2023-05-25T03:36:20.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.675 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.675 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.675 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.675 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:36:25 GMT 2023-05-25T03:36:20.676 INFO:tasks.rgw_multisite_tests:/qskrps-56/ 2023-05-25T03:36:20.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.676 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:WiWg5nhzoCRWqiQ23jKslJ0wk3k= 2023-05-25T03:36:20.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:36:25 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WiWg5nhzoCRWqiQ23jKslJ0wk3k=', 'Content-Length': '0'} 2023-05-25T03:36:20.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.678 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.678 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.678 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.678 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:36:25 GMT 2023-05-25T03:36:20.679 INFO:tasks.rgw_multisite_tests:/qskrps-56/ 2023-05-25T03:36:20.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.679 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:WiWg5nhzoCRWqiQ23jKslJ0wk3k= 2023-05-25T03:36:20.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:36:25 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:WiWg5nhzoCRWqiQ23jKslJ0wk3k=', 'Content-Length': '0'} 2023-05-25T03:36:20.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.680 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-05-25T03:36:20.681 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.681 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-05-25T03:36:20.681 INFO:tasks.rgw_multisite_tests:ERROR: Create several generations of objects, then run bucket sync 2023-05-25T03:36:20.681 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-05-25T03:36:20.682 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-05-25T03:36:20.682 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-05-25T03:36:20.682 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-05-25T03:36:20.682 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/../src/test/rgw/rgw_multi/tests.py", line 1587, in test_bucket_sync_run_basic_incremental 2023-05-25T03:36:20.683 INFO:tasks.rgw_multisite_tests: bucket = primary.create_bucket(gen_bucket_name()) 2023-05-25T03:36:20.683 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/tasks/rgw_multi/zone_rados.py", line 55, in create_bucket 2023-05-25T03:36:20.683 INFO:tasks.rgw_multisite_tests: return self.conn.create_bucket(name) 2023-05-25T03:36:20.683 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 618, in create_bucket 2023-05-25T03:36:20.684 INFO:tasks.rgw_multisite_tests: response = self.make_request('PUT', bucket_name, headers=headers, 2023-05-25T03:36:20.684 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 667, in make_request 2023-05-25T03:36:20.684 INFO:tasks.rgw_multisite_tests: return super(S3Connection, self).make_request( 2023-05-25T03:36:20.684 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1070, in make_request 2023-05-25T03:36:20.685 INFO:tasks.rgw_multisite_tests: return self._mexe(http_request, sender, override_num_retries, 2023-05-25T03:36:20.685 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1030, in _mexe 2023-05-25T03:36:20.685 INFO:tasks.rgw_multisite_tests: raise ex 2023-05-25T03:36:20.685 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 942, in _mexe 2023-05-25T03:36:20.686 INFO:tasks.rgw_multisite_tests: connection.request(request.method, request.path, 2023-05-25T03:36:20.686 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1234, in request 2023-05-25T03:36:20.686 INFO:tasks.rgw_multisite_tests: self._send_request(method, url, body, headers, encode_chunked) 2023-05-25T03:36:20.686 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1280, in _send_request 2023-05-25T03:36:20.687 INFO:tasks.rgw_multisite_tests: self.endheaders(body, encode_chunked=encode_chunked) 2023-05-25T03:36:20.687 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1229, in endheaders 2023-05-25T03:36:20.687 INFO:tasks.rgw_multisite_tests: self._send_output(message_body, encode_chunked=encode_chunked) 2023-05-25T03:36:20.687 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1008, in _send_output 2023-05-25T03:36:20.687 INFO:tasks.rgw_multisite_tests: self.send(msg) 2023-05-25T03:36:20.688 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 948, in send 2023-05-25T03:36:20.688 INFO:tasks.rgw_multisite_tests: self.connect() 2023-05-25T03:36:20.688 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 919, in connect 2023-05-25T03:36:20.688 INFO:tasks.rgw_multisite_tests: self.sock = self._create_connection( 2023-05-25T03:36:20.689 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/gevent/socket.py", line 101, in create_connection 2023-05-25T03:36:20.689 INFO:tasks.rgw_multisite_tests: sock.connect(sa) 2023-05-25T03:36:20.689 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 607, in connect 2023-05-25T03:36:20.690 INFO:tasks.rgw_multisite_tests: raise _SocketError(err, strerror(err)) 2023-05-25T03:36:20.690 INFO:tasks.rgw_multisite_tests:ConnectionRefusedError: [Errno 111] Connection refused 2023-05-25T03:36:20.690 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-05-25T03:36:20.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-57/ 2023-05-25T03:36:20.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-57/ 2023-05-25T03:36:20.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-25T03:36:20.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-57/ 2023-05-25T03:36:20.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:20.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:20.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:20.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:20.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:20.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.693 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.693 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.694 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.694 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:27:20 GMT 2023-05-25T03:36:20.695 INFO:tasks.rgw_multisite_tests:/qskrps-57/ 2023-05-25T03:36:20.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.695 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:QfljcrHW4xN2QSDQOfwOQOu+x8s= 2023-05-25T03:36:20.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:27:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:QfljcrHW4xN2QSDQOfwOQOu+x8s=', 'Content-Length': '0'} 2023-05-25T03:36:20.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.697 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.697 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.697 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.697 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:27:20 GMT 2023-05-25T03:36:20.698 INFO:tasks.rgw_multisite_tests:/qskrps-57/ 2023-05-25T03:36:20.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.698 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:QfljcrHW4xN2QSDQOfwOQOu+x8s= 2023-05-25T03:36:20.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:27:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:QfljcrHW4xN2QSDQOfwOQOu+x8s=', 'Content-Length': '0'} 2023-05-25T03:36:20.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.699 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.700 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.700 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.700 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:27:20 GMT 2023-05-25T03:36:20.700 INFO:tasks.rgw_multisite_tests:/qskrps-57/ 2023-05-25T03:36:20.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.701 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:QfljcrHW4xN2QSDQOfwOQOu+x8s= 2023-05-25T03:36:20.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:27:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:QfljcrHW4xN2QSDQOfwOQOu+x8s=', 'Content-Length': '0'} 2023-05-25T03:36:20.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.702 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.702 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.703 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.703 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:27:20 GMT 2023-05-25T03:36:20.703 INFO:tasks.rgw_multisite_tests:/qskrps-57/ 2023-05-25T03:36:20.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.704 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:QfljcrHW4xN2QSDQOfwOQOu+x8s= 2023-05-25T03:36:20.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:27:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:QfljcrHW4xN2QSDQOfwOQOu+x8s=', 'Content-Length': '0'} 2023-05-25T03:36:20.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.705 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.705 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.705 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.706 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:27:20 GMT 2023-05-25T03:36:20.706 INFO:tasks.rgw_multisite_tests:/qskrps-57/ 2023-05-25T03:36:20.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.706 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:QfljcrHW4xN2QSDQOfwOQOu+x8s= 2023-05-25T03:36:20.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:27:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:QfljcrHW4xN2QSDQOfwOQOu+x8s=', 'Content-Length': '0'} 2023-05-25T03:36:20.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.708 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.708 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.708 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.709 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:27:20 GMT 2023-05-25T03:36:20.709 INFO:tasks.rgw_multisite_tests:/qskrps-57/ 2023-05-25T03:36:20.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.709 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:QfljcrHW4xN2QSDQOfwOQOu+x8s= 2023-05-25T03:36:20.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:27:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:QfljcrHW4xN2QSDQOfwOQOu+x8s=', 'Content-Length': '0'} 2023-05-25T03:36:20.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.711 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.711 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.711 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.711 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:27:20 GMT 2023-05-25T03:36:20.712 INFO:tasks.rgw_multisite_tests:/qskrps-57/ 2023-05-25T03:36:20.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.712 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:QfljcrHW4xN2QSDQOfwOQOu+x8s= 2023-05-25T03:36:20.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:27:20 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:QfljcrHW4xN2QSDQOfwOQOu+x8s=', 'Content-Length': '0'} 2023-05-25T03:36:20.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.713 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-05-25T03:36:20.713 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.714 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-05-25T03:36:20.714 INFO:tasks.rgw_multisite_tests:ERROR: Create several generations of objects, trash them, then run bucket sync init 2023-05-25T03:36:20.714 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-05-25T03:36:20.714 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-05-25T03:36:20.715 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-05-25T03:36:20.715 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-05-25T03:36:20.715 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/../src/test/rgw/rgw_multi/tests.py", line 1650, in test_zap_init_bucket_sync_run 2023-05-25T03:36:20.715 INFO:tasks.rgw_multisite_tests: bucket = primary.create_bucket(gen_bucket_name()) 2023-05-25T03:36:20.716 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/tasks/rgw_multi/zone_rados.py", line 55, in create_bucket 2023-05-25T03:36:20.716 INFO:tasks.rgw_multisite_tests: return self.conn.create_bucket(name) 2023-05-25T03:36:20.716 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 618, in create_bucket 2023-05-25T03:36:20.716 INFO:tasks.rgw_multisite_tests: response = self.make_request('PUT', bucket_name, headers=headers, 2023-05-25T03:36:20.717 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 667, in make_request 2023-05-25T03:36:20.717 INFO:tasks.rgw_multisite_tests: return super(S3Connection, self).make_request( 2023-05-25T03:36:20.717 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1070, in make_request 2023-05-25T03:36:20.717 INFO:tasks.rgw_multisite_tests: return self._mexe(http_request, sender, override_num_retries, 2023-05-25T03:36:20.717 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1030, in _mexe 2023-05-25T03:36:20.718 INFO:tasks.rgw_multisite_tests: raise ex 2023-05-25T03:36:20.718 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 942, in _mexe 2023-05-25T03:36:20.718 INFO:tasks.rgw_multisite_tests: connection.request(request.method, request.path, 2023-05-25T03:36:20.718 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1234, in request 2023-05-25T03:36:20.719 INFO:tasks.rgw_multisite_tests: self._send_request(method, url, body, headers, encode_chunked) 2023-05-25T03:36:20.719 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1280, in _send_request 2023-05-25T03:36:20.719 INFO:tasks.rgw_multisite_tests: self.endheaders(body, encode_chunked=encode_chunked) 2023-05-25T03:36:20.719 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1229, in endheaders 2023-05-25T03:36:20.720 INFO:tasks.rgw_multisite_tests: self._send_output(message_body, encode_chunked=encode_chunked) 2023-05-25T03:36:20.720 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1008, in _send_output 2023-05-25T03:36:20.720 INFO:tasks.rgw_multisite_tests: self.send(msg) 2023-05-25T03:36:20.720 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 948, in send 2023-05-25T03:36:20.721 INFO:tasks.rgw_multisite_tests: self.connect() 2023-05-25T03:36:20.721 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 919, in connect 2023-05-25T03:36:20.721 INFO:tasks.rgw_multisite_tests: self.sock = self._create_connection( 2023-05-25T03:36:20.721 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/gevent/socket.py", line 101, in create_connection 2023-05-25T03:36:20.721 INFO:tasks.rgw_multisite_tests: sock.connect(sa) 2023-05-25T03:36:20.722 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 607, in connect 2023-05-25T03:36:20.722 INFO:tasks.rgw_multisite_tests: raise _SocketError(err, strerror(err)) 2023-05-25T03:36:20.722 INFO:tasks.rgw_multisite_tests:ConnectionRefusedError: [Errno 111] Connection refused 2023-05-25T03:36:20.722 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-05-25T03:36:20.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-58/ 2023-05-25T03:36:20.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-58/ 2023-05-25T03:36:20.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-25T03:36:20.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-58/ 2023-05-25T03:36:20.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:20.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:20.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:20.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:20.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:20.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.725 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.726 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.726 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.726 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:28:30 GMT 2023-05-25T03:36:20.726 INFO:tasks.rgw_multisite_tests:/qskrps-58/ 2023-05-25T03:36:20.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.727 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:AQDnqSLmNwk4FjpOhOd5x15rCIQ= 2023-05-25T03:36:20.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:28:30 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:AQDnqSLmNwk4FjpOhOd5x15rCIQ=', 'Content-Length': '0'} 2023-05-25T03:36:20.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.728 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.729 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.729 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.729 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:28:30 GMT 2023-05-25T03:36:20.729 INFO:tasks.rgw_multisite_tests:/qskrps-58/ 2023-05-25T03:36:20.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.730 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:AQDnqSLmNwk4FjpOhOd5x15rCIQ= 2023-05-25T03:36:20.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:28:30 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:AQDnqSLmNwk4FjpOhOd5x15rCIQ=', 'Content-Length': '0'} 2023-05-25T03:36:20.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.731 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.731 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.732 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.732 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:28:30 GMT 2023-05-25T03:36:20.732 INFO:tasks.rgw_multisite_tests:/qskrps-58/ 2023-05-25T03:36:20.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.733 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:AQDnqSLmNwk4FjpOhOd5x15rCIQ= 2023-05-25T03:36:20.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:28:30 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:AQDnqSLmNwk4FjpOhOd5x15rCIQ=', 'Content-Length': '0'} 2023-05-25T03:36:20.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.734 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.734 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.734 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.735 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:28:30 GMT 2023-05-25T03:36:20.735 INFO:tasks.rgw_multisite_tests:/qskrps-58/ 2023-05-25T03:36:20.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.735 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:AQDnqSLmNwk4FjpOhOd5x15rCIQ= 2023-05-25T03:36:20.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:28:30 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:AQDnqSLmNwk4FjpOhOd5x15rCIQ=', 'Content-Length': '0'} 2023-05-25T03:36:20.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.737 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.737 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.737 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.737 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:28:30 GMT 2023-05-25T03:36:20.738 INFO:tasks.rgw_multisite_tests:/qskrps-58/ 2023-05-25T03:36:20.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.738 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:AQDnqSLmNwk4FjpOhOd5x15rCIQ= 2023-05-25T03:36:20.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:28:30 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:AQDnqSLmNwk4FjpOhOd5x15rCIQ=', 'Content-Length': '0'} 2023-05-25T03:36:20.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.740 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.740 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.740 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.740 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:28:30 GMT 2023-05-25T03:36:20.741 INFO:tasks.rgw_multisite_tests:/qskrps-58/ 2023-05-25T03:36:20.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.741 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:AQDnqSLmNwk4FjpOhOd5x15rCIQ= 2023-05-25T03:36:20.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:28:30 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:AQDnqSLmNwk4FjpOhOd5x15rCIQ=', 'Content-Length': '0'} 2023-05-25T03:36:20.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.742 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.743 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.743 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.743 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:28:30 GMT 2023-05-25T03:36:20.743 INFO:tasks.rgw_multisite_tests:/qskrps-58/ 2023-05-25T03:36:20.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.744 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:AQDnqSLmNwk4FjpOhOd5x15rCIQ= 2023-05-25T03:36:20.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:28:30 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:AQDnqSLmNwk4FjpOhOd5x15rCIQ=', 'Content-Length': '0'} 2023-05-25T03:36:20.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.745 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-05-25T03:36:20.745 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.746 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-05-25T03:36:20.746 INFO:tasks.rgw_multisite_tests:ERROR: rgw_multi.tests.test_role_sync 2023-05-25T03:36:20.746 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-05-25T03:36:20.747 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-05-25T03:36:20.747 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-05-25T03:36:20.747 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-05-25T03:36:20.747 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/../src/test/rgw/rgw_multi/tests.py", line 1697, in test_role_sync 2023-05-25T03:36:20.748 INFO:tasks.rgw_multisite_tests: roles, zone_role = create_role_per_zone(zonegroup_conns) 2023-05-25T03:36:20.748 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/../src/test/rgw/rgw_multi/tests.py", line 511, in create_role_per_zone 2023-05-25T03:36:20.748 INFO:tasks.rgw_multisite_tests: role = zone.create_role("", role_name, policy_document, "") 2023-05-25T03:36:20.748 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/tasks/rgw_multi/zone_rados.py", line 130, in create_role 2023-05-25T03:36:20.748 INFO:tasks.rgw_multisite_tests: return self.iam_conn.create_role(rolename, policy_document, path) 2023-05-25T03:36:20.749 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/iam/connection.py", line 1155, in create_role 2023-05-25T03:36:20.749 INFO:tasks.rgw_multisite_tests: return self.get_response('CreateRole', params) 2023-05-25T03:36:20.749 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/iam/connection.py", line 85, in get_response 2023-05-25T03:36:20.749 INFO:tasks.rgw_multisite_tests: response = self.make_request(action, params, path, verb) 2023-05-25T03:36:20.750 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1116, in make_request 2023-05-25T03:36:20.750 INFO:tasks.rgw_multisite_tests: return self._mexe(http_request) 2023-05-25T03:36:20.750 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1030, in _mexe 2023-05-25T03:36:20.750 INFO:tasks.rgw_multisite_tests: raise ex 2023-05-25T03:36:20.751 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 942, in _mexe 2023-05-25T03:36:20.751 INFO:tasks.rgw_multisite_tests: connection.request(request.method, request.path, 2023-05-25T03:36:20.751 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1234, in request 2023-05-25T03:36:20.751 INFO:tasks.rgw_multisite_tests: self._send_request(method, url, body, headers, encode_chunked) 2023-05-25T03:36:20.751 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1280, in _send_request 2023-05-25T03:36:20.752 INFO:tasks.rgw_multisite_tests: self.endheaders(body, encode_chunked=encode_chunked) 2023-05-25T03:36:20.752 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1229, in endheaders 2023-05-25T03:36:20.752 INFO:tasks.rgw_multisite_tests: self._send_output(message_body, encode_chunked=encode_chunked) 2023-05-25T03:36:20.752 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1008, in _send_output 2023-05-25T03:36:20.753 INFO:tasks.rgw_multisite_tests: self.send(msg) 2023-05-25T03:36:20.753 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 948, in send 2023-05-25T03:36:20.753 INFO:tasks.rgw_multisite_tests: self.connect() 2023-05-25T03:36:20.754 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 919, in connect 2023-05-25T03:36:20.754 INFO:tasks.rgw_multisite_tests: self.sock = self._create_connection( 2023-05-25T03:36:20.754 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/gevent/socket.py", line 101, in create_connection 2023-05-25T03:36:20.754 INFO:tasks.rgw_multisite_tests: sock.connect(sa) 2023-05-25T03:36:20.755 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 607, in connect 2023-05-25T03:36:20.755 INFO:tasks.rgw_multisite_tests: raise _SocketError(err, strerror(err)) 2023-05-25T03:36:20.755 INFO:tasks.rgw_multisite_tests:ConnectionRefusedError: [Errno 111] Connection refused 2023-05-25T03:36:20.756 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-05-25T03:36:20.756 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create role zone=a1 name=roles-qskrps-1 2023-05-25T03:36:20.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: POST 2023-05-25T03:36:20.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: / 2023-05-25T03:36:20.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:20.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:20.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2023-05-25T03:36:20.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:20.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {'RoleName': 'roles-qskrps-1', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2023-05-25T03:36:20.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2023-05-25T03:36:20.762 INFO:tasks.rgw_multisite_tests:POST 2023-05-25T03:36:20.763 INFO:tasks.rgw_multisite_tests:/ 2023-05-25T03:36:20.763 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.763 INFO:tasks.rgw_multisite_tests:host:smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:20.763 INFO:tasks.rgw_multisite_tests:x-amz-date:20230524T223016Z 2023-05-25T03:36:20.764 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.764 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2023-05-25T03:36:20.764 INFO:tasks.rgw_multisite_tests:6eda902334c0e592d2f03ea0d8914abd2dcb79ee8f570a7b5f29cce3899b188a 2023-05-25T03:36:20.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.765 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2023-05-25T03:36:20.765 INFO:tasks.rgw_multisite_tests:20230524T223016Z 2023-05-25T03:36:20.765 INFO:tasks.rgw_multisite_tests:20230524/front/smithi071/aws4_request 2023-05-25T03:36:20.765 INFO:tasks.rgw_multisite_tests:d7d50c40a91b57044063d40217e986b4bf998e0934bd18825dd9e82e074ac762 2023-05-25T03:36:20.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.766 INFO:tasks.rgw_multisite_tests:d1a23842b2e02625ae4bc9f2c54bde824e17b09f672fd6426d5798f3021ed591 2023-05-25T03:36:20.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'X-Amz-Date': '20230524T223016Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=YAS2LRYUNTM46AEFZJ98/20230524/front/smithi071/aws4_request,SignedHeaders=host;x-amz-date,Signature=d1a23842b2e02625ae4bc9f2c54bde824e17b09f672fd6426d5798f3021ed591', 'Host': 'smithi071.front.sepia.ceph.com:8000'} 2023-05-25T03:36:20.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2023-05-25T03:36:20.767 INFO:tasks.rgw_multisite_tests:POST 2023-05-25T03:36:20.768 INFO:tasks.rgw_multisite_tests:/ 2023-05-25T03:36:20.768 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.768 INFO:tasks.rgw_multisite_tests:host:smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:20.768 INFO:tasks.rgw_multisite_tests:x-amz-date:20230524T223017Z 2023-05-25T03:36:20.769 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.769 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2023-05-25T03:36:20.769 INFO:tasks.rgw_multisite_tests:6eda902334c0e592d2f03ea0d8914abd2dcb79ee8f570a7b5f29cce3899b188a 2023-05-25T03:36:20.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.770 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2023-05-25T03:36:20.770 INFO:tasks.rgw_multisite_tests:20230524T223017Z 2023-05-25T03:36:20.770 INFO:tasks.rgw_multisite_tests:20230524/front/smithi071/aws4_request 2023-05-25T03:36:20.771 INFO:tasks.rgw_multisite_tests:911c0b925409489cc856fe70045d7a7857132e546472aedadab27efd744ff065 2023-05-25T03:36:20.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.771 INFO:tasks.rgw_multisite_tests:22812077c896ab902ae7d55ddfb52a3eaa5bc49db166f00775d15d9af4393ad4 2023-05-25T03:36:20.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'X-Amz-Date': '20230524T223017Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=YAS2LRYUNTM46AEFZJ98/20230524/front/smithi071/aws4_request,SignedHeaders=host;x-amz-date,Signature=22812077c896ab902ae7d55ddfb52a3eaa5bc49db166f00775d15d9af4393ad4', 'Host': 'smithi071.front.sepia.ceph.com:8000'} 2023-05-25T03:36:20.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2023-05-25T03:36:20.773 INFO:tasks.rgw_multisite_tests:POST 2023-05-25T03:36:20.773 INFO:tasks.rgw_multisite_tests:/ 2023-05-25T03:36:20.773 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.773 INFO:tasks.rgw_multisite_tests:host:smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:20.774 INFO:tasks.rgw_multisite_tests:x-amz-date:20230524T223017Z 2023-05-25T03:36:20.774 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.774 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2023-05-25T03:36:20.774 INFO:tasks.rgw_multisite_tests:6eda902334c0e592d2f03ea0d8914abd2dcb79ee8f570a7b5f29cce3899b188a 2023-05-25T03:36:20.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.775 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2023-05-25T03:36:20.775 INFO:tasks.rgw_multisite_tests:20230524T223017Z 2023-05-25T03:36:20.775 INFO:tasks.rgw_multisite_tests:20230524/front/smithi071/aws4_request 2023-05-25T03:36:20.776 INFO:tasks.rgw_multisite_tests:911c0b925409489cc856fe70045d7a7857132e546472aedadab27efd744ff065 2023-05-25T03:36:20.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.776 INFO:tasks.rgw_multisite_tests:22812077c896ab902ae7d55ddfb52a3eaa5bc49db166f00775d15d9af4393ad4 2023-05-25T03:36:20.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'X-Amz-Date': '20230524T223017Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=YAS2LRYUNTM46AEFZJ98/20230524/front/smithi071/aws4_request,SignedHeaders=host;x-amz-date,Signature=22812077c896ab902ae7d55ddfb52a3eaa5bc49db166f00775d15d9af4393ad4', 'Host': 'smithi071.front.sepia.ceph.com:8000'} 2023-05-25T03:36:20.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2023-05-25T03:36:20.778 INFO:tasks.rgw_multisite_tests:POST 2023-05-25T03:36:20.778 INFO:tasks.rgw_multisite_tests:/ 2023-05-25T03:36:20.778 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.778 INFO:tasks.rgw_multisite_tests:host:smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:20.779 INFO:tasks.rgw_multisite_tests:x-amz-date:20230524T223019Z 2023-05-25T03:36:20.779 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.779 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2023-05-25T03:36:20.779 INFO:tasks.rgw_multisite_tests:6eda902334c0e592d2f03ea0d8914abd2dcb79ee8f570a7b5f29cce3899b188a 2023-05-25T03:36:20.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.780 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2023-05-25T03:36:20.780 INFO:tasks.rgw_multisite_tests:20230524T223019Z 2023-05-25T03:36:20.781 INFO:tasks.rgw_multisite_tests:20230524/front/smithi071/aws4_request 2023-05-25T03:36:20.781 INFO:tasks.rgw_multisite_tests:7562a772abf0e4c428fa28038942512b99cddec88b1c443e0933271b4df2dedd 2023-05-25T03:36:20.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.781 INFO:tasks.rgw_multisite_tests:792dad4b2882b20b1cd6c6efd33bd14cc1734c888614ef8c21042dfb363c9b9c 2023-05-25T03:36:20.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'X-Amz-Date': '20230524T223019Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=YAS2LRYUNTM46AEFZJ98/20230524/front/smithi071/aws4_request,SignedHeaders=host;x-amz-date,Signature=792dad4b2882b20b1cd6c6efd33bd14cc1734c888614ef8c21042dfb363c9b9c', 'Host': 'smithi071.front.sepia.ceph.com:8000'} 2023-05-25T03:36:20.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2023-05-25T03:36:20.783 INFO:tasks.rgw_multisite_tests:POST 2023-05-25T03:36:20.783 INFO:tasks.rgw_multisite_tests:/ 2023-05-25T03:36:20.783 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.783 INFO:tasks.rgw_multisite_tests:host:smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:20.784 INFO:tasks.rgw_multisite_tests:x-amz-date:20230524T223022Z 2023-05-25T03:36:20.784 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.784 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2023-05-25T03:36:20.784 INFO:tasks.rgw_multisite_tests:6eda902334c0e592d2f03ea0d8914abd2dcb79ee8f570a7b5f29cce3899b188a 2023-05-25T03:36:20.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.785 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2023-05-25T03:36:20.785 INFO:tasks.rgw_multisite_tests:20230524T223022Z 2023-05-25T03:36:20.785 INFO:tasks.rgw_multisite_tests:20230524/front/smithi071/aws4_request 2023-05-25T03:36:20.786 INFO:tasks.rgw_multisite_tests:cb30a48d81f2ef83d23c4788a49f8b9b59deb8c74e28d44a0415ff98ece38131 2023-05-25T03:36:20.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.786 INFO:tasks.rgw_multisite_tests:bb32b65749141c6861299caff07da7cf5206458aa1ed9040bea97c802e3bf0f8 2023-05-25T03:36:20.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'X-Amz-Date': '20230524T223022Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=YAS2LRYUNTM46AEFZJ98/20230524/front/smithi071/aws4_request,SignedHeaders=host;x-amz-date,Signature=bb32b65749141c6861299caff07da7cf5206458aa1ed9040bea97c802e3bf0f8', 'Host': 'smithi071.front.sepia.ceph.com:8000'} 2023-05-25T03:36:20.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2023-05-25T03:36:20.787 INFO:tasks.rgw_multisite_tests:POST 2023-05-25T03:36:20.788 INFO:tasks.rgw_multisite_tests:/ 2023-05-25T03:36:20.788 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.788 INFO:tasks.rgw_multisite_tests:host:smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:20.788 INFO:tasks.rgw_multisite_tests:x-amz-date:20230524T223027Z 2023-05-25T03:36:20.789 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.789 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2023-05-25T03:36:20.789 INFO:tasks.rgw_multisite_tests:6eda902334c0e592d2f03ea0d8914abd2dcb79ee8f570a7b5f29cce3899b188a 2023-05-25T03:36:20.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.790 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2023-05-25T03:36:20.790 INFO:tasks.rgw_multisite_tests:20230524T223027Z 2023-05-25T03:36:20.790 INFO:tasks.rgw_multisite_tests:20230524/front/smithi071/aws4_request 2023-05-25T03:36:20.790 INFO:tasks.rgw_multisite_tests:e181d0d7dfbfd4802725e75d147c1c4a837b0a986a0fc4035a838645a6ce04a4 2023-05-25T03:36:20.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.791 INFO:tasks.rgw_multisite_tests:c4c8e65129a69b11fa4f586ea7eb8e3559375a6124e8872ad590a935354487eb 2023-05-25T03:36:20.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'X-Amz-Date': '20230524T223027Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=YAS2LRYUNTM46AEFZJ98/20230524/front/smithi071/aws4_request,SignedHeaders=host;x-amz-date,Signature=c4c8e65129a69b11fa4f586ea7eb8e3559375a6124e8872ad590a935354487eb', 'Host': 'smithi071.front.sepia.ceph.com:8000'} 2023-05-25T03:36:20.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2023-05-25T03:36:20.792 INFO:tasks.rgw_multisite_tests:POST 2023-05-25T03:36:20.793 INFO:tasks.rgw_multisite_tests:/ 2023-05-25T03:36:20.793 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.793 INFO:tasks.rgw_multisite_tests:host:smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:20.793 INFO:tasks.rgw_multisite_tests:x-amz-date:20230524T223051Z 2023-05-25T03:36:20.794 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.794 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2023-05-25T03:36:20.794 INFO:tasks.rgw_multisite_tests:6eda902334c0e592d2f03ea0d8914abd2dcb79ee8f570a7b5f29cce3899b188a 2023-05-25T03:36:20.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.795 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2023-05-25T03:36:20.795 INFO:tasks.rgw_multisite_tests:20230524T223051Z 2023-05-25T03:36:20.795 INFO:tasks.rgw_multisite_tests:20230524/front/smithi071/aws4_request 2023-05-25T03:36:20.795 INFO:tasks.rgw_multisite_tests:95eda10695db0e8761b0126511b336b3d1b3b686e78e3c75d81e09e69db4e9a0 2023-05-25T03:36:20.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.796 INFO:tasks.rgw_multisite_tests:5b6abcd66661200e32ffacaf843e2e76bdc3f65198f904eed72c3d1f5af5b34e 2023-05-25T03:36:20.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'X-Amz-Date': '20230524T223051Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=YAS2LRYUNTM46AEFZJ98/20230524/front/smithi071/aws4_request,SignedHeaders=host;x-amz-date,Signature=5b6abcd66661200e32ffacaf843e2e76bdc3f65198f904eed72c3d1f5af5b34e', 'Host': 'smithi071.front.sepia.ceph.com:8000'} 2023-05-25T03:36:20.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.797 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-05-25T03:36:20.797 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.797 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-05-25T03:36:20.798 INFO:tasks.rgw_multisite_tests:ERROR: rgw_multi.tests.test_bucket_full_sync_after_data_sync_init 2023-05-25T03:36:20.798 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-05-25T03:36:20.798 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-05-25T03:36:20.798 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-05-25T03:36:20.799 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-05-25T03:36:20.799 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/../src/test/rgw/rgw_multi/tests.py", line 1715, in test_bucket_full_sync_after_data_sync_init 2023-05-25T03:36:20.799 INFO:tasks.rgw_multisite_tests: bucket = primary.create_bucket(gen_bucket_name()) 2023-05-25T03:36:20.799 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/tasks/rgw_multi/zone_rados.py", line 55, in create_bucket 2023-05-25T03:36:20.800 INFO:tasks.rgw_multisite_tests: return self.conn.create_bucket(name) 2023-05-25T03:36:20.800 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 618, in create_bucket 2023-05-25T03:36:20.800 INFO:tasks.rgw_multisite_tests: response = self.make_request('PUT', bucket_name, headers=headers, 2023-05-25T03:36:20.800 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 667, in make_request 2023-05-25T03:36:20.801 INFO:tasks.rgw_multisite_tests: return super(S3Connection, self).make_request( 2023-05-25T03:36:20.801 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1070, in make_request 2023-05-25T03:36:20.801 INFO:tasks.rgw_multisite_tests: return self._mexe(http_request, sender, override_num_retries, 2023-05-25T03:36:20.801 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1030, in _mexe 2023-05-25T03:36:20.802 INFO:tasks.rgw_multisite_tests: raise ex 2023-05-25T03:36:20.802 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 942, in _mexe 2023-05-25T03:36:20.802 INFO:tasks.rgw_multisite_tests: connection.request(request.method, request.path, 2023-05-25T03:36:20.802 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1234, in request 2023-05-25T03:36:20.802 INFO:tasks.rgw_multisite_tests: self._send_request(method, url, body, headers, encode_chunked) 2023-05-25T03:36:20.803 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1280, in _send_request 2023-05-25T03:36:20.803 INFO:tasks.rgw_multisite_tests: self.endheaders(body, encode_chunked=encode_chunked) 2023-05-25T03:36:20.803 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1229, in endheaders 2023-05-25T03:36:20.803 INFO:tasks.rgw_multisite_tests: self._send_output(message_body, encode_chunked=encode_chunked) 2023-05-25T03:36:20.804 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1008, in _send_output 2023-05-25T03:36:20.804 INFO:tasks.rgw_multisite_tests: self.send(msg) 2023-05-25T03:36:20.804 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 948, in send 2023-05-25T03:36:20.804 INFO:tasks.rgw_multisite_tests: self.connect() 2023-05-25T03:36:20.805 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 919, in connect 2023-05-25T03:36:20.805 INFO:tasks.rgw_multisite_tests: self.sock = self._create_connection( 2023-05-25T03:36:20.805 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/gevent/socket.py", line 101, in create_connection 2023-05-25T03:36:20.805 INFO:tasks.rgw_multisite_tests: sock.connect(sa) 2023-05-25T03:36:20.806 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 607, in connect 2023-05-25T03:36:20.806 INFO:tasks.rgw_multisite_tests: raise _SocketError(err, strerror(err)) 2023-05-25T03:36:20.806 INFO:tasks.rgw_multisite_tests:ConnectionRefusedError: [Errno 111] Connection refused 2023-05-25T03:36:20.806 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-05-25T03:36:20.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-59/ 2023-05-25T03:36:20.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-59/ 2023-05-25T03:36:20.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-25T03:36:20.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-59/ 2023-05-25T03:36:20.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:20.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:20.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:20.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:20.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:20.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.809 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.810 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.810 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.810 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:31:14 GMT 2023-05-25T03:36:20.810 INFO:tasks.rgw_multisite_tests:/qskrps-59/ 2023-05-25T03:36:20.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.811 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:BMGdGlkxu5a2MljWLy6cuIYfLtI= 2023-05-25T03:36:20.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:31:14 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:BMGdGlkxu5a2MljWLy6cuIYfLtI=', 'Content-Length': '0'} 2023-05-25T03:36:20.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.812 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.813 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.813 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.813 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:31:14 GMT 2023-05-25T03:36:20.813 INFO:tasks.rgw_multisite_tests:/qskrps-59/ 2023-05-25T03:36:20.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.814 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:BMGdGlkxu5a2MljWLy6cuIYfLtI= 2023-05-25T03:36:20.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:31:14 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:BMGdGlkxu5a2MljWLy6cuIYfLtI=', 'Content-Length': '0'} 2023-05-25T03:36:20.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.815 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.815 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.816 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.816 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:31:14 GMT 2023-05-25T03:36:20.816 INFO:tasks.rgw_multisite_tests:/qskrps-59/ 2023-05-25T03:36:20.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.817 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:BMGdGlkxu5a2MljWLy6cuIYfLtI= 2023-05-25T03:36:20.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:31:14 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:BMGdGlkxu5a2MljWLy6cuIYfLtI=', 'Content-Length': '0'} 2023-05-25T03:36:20.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.818 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.818 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.818 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.819 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:31:14 GMT 2023-05-25T03:36:20.819 INFO:tasks.rgw_multisite_tests:/qskrps-59/ 2023-05-25T03:36:20.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.820 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:BMGdGlkxu5a2MljWLy6cuIYfLtI= 2023-05-25T03:36:20.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:31:14 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:BMGdGlkxu5a2MljWLy6cuIYfLtI=', 'Content-Length': '0'} 2023-05-25T03:36:20.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.821 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.821 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.822 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.822 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:31:14 GMT 2023-05-25T03:36:20.822 INFO:tasks.rgw_multisite_tests:/qskrps-59/ 2023-05-25T03:36:20.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.823 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:BMGdGlkxu5a2MljWLy6cuIYfLtI= 2023-05-25T03:36:20.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:31:14 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:BMGdGlkxu5a2MljWLy6cuIYfLtI=', 'Content-Length': '0'} 2023-05-25T03:36:20.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.825 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.825 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.825 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.825 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:31:14 GMT 2023-05-25T03:36:20.826 INFO:tasks.rgw_multisite_tests:/qskrps-59/ 2023-05-25T03:36:20.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.826 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:BMGdGlkxu5a2MljWLy6cuIYfLtI= 2023-05-25T03:36:20.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:31:14 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:BMGdGlkxu5a2MljWLy6cuIYfLtI=', 'Content-Length': '0'} 2023-05-25T03:36:20.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.827 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.828 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.828 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.828 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:31:14 GMT 2023-05-25T03:36:20.829 INFO:tasks.rgw_multisite_tests:/qskrps-59/ 2023-05-25T03:36:20.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.829 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:BMGdGlkxu5a2MljWLy6cuIYfLtI= 2023-05-25T03:36:20.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:31:14 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:BMGdGlkxu5a2MljWLy6cuIYfLtI=', 'Content-Length': '0'} 2023-05-25T03:36:20.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.830 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-05-25T03:36:20.830 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.830 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-05-25T03:36:20.831 INFO:tasks.rgw_multisite_tests:ERROR: rgw_multi.tests.test_resharded_bucket_full_sync_after_data_sync_init 2023-05-25T03:36:20.831 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-05-25T03:36:20.831 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-05-25T03:36:20.832 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-05-25T03:36:20.832 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-05-25T03:36:20.832 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/../src/test/rgw/rgw_multi/tests.py", line 1747, in test_resharded_bucket_full_sync_after_data_sync_init 2023-05-25T03:36:20.832 INFO:tasks.rgw_multisite_tests: bucket = primary.create_bucket(gen_bucket_name()) 2023-05-25T03:36:20.833 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/tasks/rgw_multi/zone_rados.py", line 55, in create_bucket 2023-05-25T03:36:20.833 INFO:tasks.rgw_multisite_tests: return self.conn.create_bucket(name) 2023-05-25T03:36:20.833 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 618, in create_bucket 2023-05-25T03:36:20.833 INFO:tasks.rgw_multisite_tests: response = self.make_request('PUT', bucket_name, headers=headers, 2023-05-25T03:36:20.833 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 667, in make_request 2023-05-25T03:36:20.834 INFO:tasks.rgw_multisite_tests: return super(S3Connection, self).make_request( 2023-05-25T03:36:20.834 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1070, in make_request 2023-05-25T03:36:20.834 INFO:tasks.rgw_multisite_tests: return self._mexe(http_request, sender, override_num_retries, 2023-05-25T03:36:20.834 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1030, in _mexe 2023-05-25T03:36:20.835 INFO:tasks.rgw_multisite_tests: raise ex 2023-05-25T03:36:20.835 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 942, in _mexe 2023-05-25T03:36:20.835 INFO:tasks.rgw_multisite_tests: connection.request(request.method, request.path, 2023-05-25T03:36:20.835 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1234, in request 2023-05-25T03:36:20.836 INFO:tasks.rgw_multisite_tests: self._send_request(method, url, body, headers, encode_chunked) 2023-05-25T03:36:20.836 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1280, in _send_request 2023-05-25T03:36:20.836 INFO:tasks.rgw_multisite_tests: self.endheaders(body, encode_chunked=encode_chunked) 2023-05-25T03:36:20.836 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1229, in endheaders 2023-05-25T03:36:20.837 INFO:tasks.rgw_multisite_tests: self._send_output(message_body, encode_chunked=encode_chunked) 2023-05-25T03:36:20.837 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1008, in _send_output 2023-05-25T03:36:20.837 INFO:tasks.rgw_multisite_tests: self.send(msg) 2023-05-25T03:36:20.837 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 948, in send 2023-05-25T03:36:20.838 INFO:tasks.rgw_multisite_tests: self.connect() 2023-05-25T03:36:20.838 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 919, in connect 2023-05-25T03:36:20.838 INFO:tasks.rgw_multisite_tests: self.sock = self._create_connection( 2023-05-25T03:36:20.838 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/gevent/socket.py", line 101, in create_connection 2023-05-25T03:36:20.839 INFO:tasks.rgw_multisite_tests: sock.connect(sa) 2023-05-25T03:36:20.839 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 607, in connect 2023-05-25T03:36:20.839 INFO:tasks.rgw_multisite_tests: raise _SocketError(err, strerror(err)) 2023-05-25T03:36:20.839 INFO:tasks.rgw_multisite_tests:ConnectionRefusedError: [Errno 111] Connection refused 2023-05-25T03:36:20.840 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-05-25T03:36:20.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-60/ 2023-05-25T03:36:20.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-60/ 2023-05-25T03:36:20.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-25T03:36:20.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-60/ 2023-05-25T03:36:20.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:20.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:20.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:20.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:20.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:20.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.843 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.843 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.843 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.843 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:31:55 GMT 2023-05-25T03:36:20.844 INFO:tasks.rgw_multisite_tests:/qskrps-60/ 2023-05-25T03:36:20.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.844 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:PK7RnQU5YnNza4ZAb48nmLlQLHE= 2023-05-25T03:36:20.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:31:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PK7RnQU5YnNza4ZAb48nmLlQLHE=', 'Content-Length': '0'} 2023-05-25T03:36:20.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.847 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.848 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.848 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.849 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:31:55 GMT 2023-05-25T03:36:20.850 INFO:tasks.rgw_multisite_tests:/qskrps-60/ 2023-05-25T03:36:20.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.851 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:PK7RnQU5YnNza4ZAb48nmLlQLHE= 2023-05-25T03:36:20.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:31:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PK7RnQU5YnNza4ZAb48nmLlQLHE=', 'Content-Length': '0'} 2023-05-25T03:36:20.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.854 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.855 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.855 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.855 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:31:55 GMT 2023-05-25T03:36:20.855 INFO:tasks.rgw_multisite_tests:/qskrps-60/ 2023-05-25T03:36:20.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.856 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:PK7RnQU5YnNza4ZAb48nmLlQLHE= 2023-05-25T03:36:20.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:31:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PK7RnQU5YnNza4ZAb48nmLlQLHE=', 'Content-Length': '0'} 2023-05-25T03:36:20.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.857 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.858 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.858 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.858 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:31:55 GMT 2023-05-25T03:36:20.858 INFO:tasks.rgw_multisite_tests:/qskrps-60/ 2023-05-25T03:36:20.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.859 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:PK7RnQU5YnNza4ZAb48nmLlQLHE= 2023-05-25T03:36:20.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:31:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PK7RnQU5YnNza4ZAb48nmLlQLHE=', 'Content-Length': '0'} 2023-05-25T03:36:20.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.860 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.860 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.861 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.861 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:31:55 GMT 2023-05-25T03:36:20.861 INFO:tasks.rgw_multisite_tests:/qskrps-60/ 2023-05-25T03:36:20.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.862 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:PK7RnQU5YnNza4ZAb48nmLlQLHE= 2023-05-25T03:36:20.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:31:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PK7RnQU5YnNza4ZAb48nmLlQLHE=', 'Content-Length': '0'} 2023-05-25T03:36:20.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.863 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.863 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.864 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.864 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:31:55 GMT 2023-05-25T03:36:20.864 INFO:tasks.rgw_multisite_tests:/qskrps-60/ 2023-05-25T03:36:20.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.865 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:PK7RnQU5YnNza4ZAb48nmLlQLHE= 2023-05-25T03:36:20.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:31:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PK7RnQU5YnNza4ZAb48nmLlQLHE=', 'Content-Length': '0'} 2023-05-25T03:36:20.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.866 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.867 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.867 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.867 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:31:55 GMT 2023-05-25T03:36:20.867 INFO:tasks.rgw_multisite_tests:/qskrps-60/ 2023-05-25T03:36:20.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.868 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:PK7RnQU5YnNza4ZAb48nmLlQLHE= 2023-05-25T03:36:20.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:31:55 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:PK7RnQU5YnNza4ZAb48nmLlQLHE=', 'Content-Length': '0'} 2023-05-25T03:36:20.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.869 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-05-25T03:36:20.869 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.870 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-05-25T03:36:20.870 INFO:tasks.rgw_multisite_tests:ERROR: rgw_multi.tests.test_bucket_incremental_sync_after_data_sync_init 2023-05-25T03:36:20.870 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-05-25T03:36:20.870 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-05-25T03:36:20.871 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-05-25T03:36:20.871 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-05-25T03:36:20.871 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/../src/test/rgw/rgw_multi/tests.py", line 1786, in test_bucket_incremental_sync_after_data_sync_init 2023-05-25T03:36:20.871 INFO:tasks.rgw_multisite_tests: bucket = primary.create_bucket(gen_bucket_name()) 2023-05-25T03:36:20.872 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/tasks/rgw_multi/zone_rados.py", line 55, in create_bucket 2023-05-25T03:36:20.872 INFO:tasks.rgw_multisite_tests: return self.conn.create_bucket(name) 2023-05-25T03:36:20.872 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 618, in create_bucket 2023-05-25T03:36:20.873 INFO:tasks.rgw_multisite_tests: response = self.make_request('PUT', bucket_name, headers=headers, 2023-05-25T03:36:20.873 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 667, in make_request 2023-05-25T03:36:20.874 INFO:tasks.rgw_multisite_tests: return super(S3Connection, self).make_request( 2023-05-25T03:36:20.874 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1070, in make_request 2023-05-25T03:36:20.874 INFO:tasks.rgw_multisite_tests: return self._mexe(http_request, sender, override_num_retries, 2023-05-25T03:36:20.874 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1030, in _mexe 2023-05-25T03:36:20.875 INFO:tasks.rgw_multisite_tests: raise ex 2023-05-25T03:36:20.875 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 942, in _mexe 2023-05-25T03:36:20.875 INFO:tasks.rgw_multisite_tests: connection.request(request.method, request.path, 2023-05-25T03:36:20.875 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1234, in request 2023-05-25T03:36:20.876 INFO:tasks.rgw_multisite_tests: self._send_request(method, url, body, headers, encode_chunked) 2023-05-25T03:36:20.876 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1280, in _send_request 2023-05-25T03:36:20.876 INFO:tasks.rgw_multisite_tests: self.endheaders(body, encode_chunked=encode_chunked) 2023-05-25T03:36:20.876 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1229, in endheaders 2023-05-25T03:36:20.877 INFO:tasks.rgw_multisite_tests: self._send_output(message_body, encode_chunked=encode_chunked) 2023-05-25T03:36:20.877 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1008, in _send_output 2023-05-25T03:36:20.877 INFO:tasks.rgw_multisite_tests: self.send(msg) 2023-05-25T03:36:20.877 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 948, in send 2023-05-25T03:36:20.878 INFO:tasks.rgw_multisite_tests: self.connect() 2023-05-25T03:36:20.878 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 919, in connect 2023-05-25T03:36:20.878 INFO:tasks.rgw_multisite_tests: self.sock = self._create_connection( 2023-05-25T03:36:20.878 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/gevent/socket.py", line 101, in create_connection 2023-05-25T03:36:20.878 INFO:tasks.rgw_multisite_tests: sock.connect(sa) 2023-05-25T03:36:20.879 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 607, in connect 2023-05-25T03:36:20.879 INFO:tasks.rgw_multisite_tests: raise _SocketError(err, strerror(err)) 2023-05-25T03:36:20.879 INFO:tasks.rgw_multisite_tests:ConnectionRefusedError: [Errno 111] Connection refused 2023-05-25T03:36:20.879 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-05-25T03:36:20.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-61/ 2023-05-25T03:36:20.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-61/ 2023-05-25T03:36:20.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-25T03:36:20.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-61/ 2023-05-25T03:36:20.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:20.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:20.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:20.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:20.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:20.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.885 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.886 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.886 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.887 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:33:14 GMT 2023-05-25T03:36:20.888 INFO:tasks.rgw_multisite_tests:/qskrps-61/ 2023-05-25T03:36:20.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.889 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:qEk59GYgwLFY3P5e/oa3vlt+F9A= 2023-05-25T03:36:20.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:33:14 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qEk59GYgwLFY3P5e/oa3vlt+F9A=', 'Content-Length': '0'} 2023-05-25T03:36:20.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.891 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.891 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.892 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.892 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:33:14 GMT 2023-05-25T03:36:20.892 INFO:tasks.rgw_multisite_tests:/qskrps-61/ 2023-05-25T03:36:20.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.893 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:qEk59GYgwLFY3P5e/oa3vlt+F9A= 2023-05-25T03:36:20.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:33:14 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qEk59GYgwLFY3P5e/oa3vlt+F9A=', 'Content-Length': '0'} 2023-05-25T03:36:20.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.894 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.894 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.895 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.895 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:33:14 GMT 2023-05-25T03:36:20.895 INFO:tasks.rgw_multisite_tests:/qskrps-61/ 2023-05-25T03:36:20.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.896 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:qEk59GYgwLFY3P5e/oa3vlt+F9A= 2023-05-25T03:36:20.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:33:14 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qEk59GYgwLFY3P5e/oa3vlt+F9A=', 'Content-Length': '0'} 2023-05-25T03:36:20.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.897 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.898 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.898 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.898 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:33:14 GMT 2023-05-25T03:36:20.899 INFO:tasks.rgw_multisite_tests:/qskrps-61/ 2023-05-25T03:36:20.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.899 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:qEk59GYgwLFY3P5e/oa3vlt+F9A= 2023-05-25T03:36:20.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:33:14 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qEk59GYgwLFY3P5e/oa3vlt+F9A=', 'Content-Length': '0'} 2023-05-25T03:36:20.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.901 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.901 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.902 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.902 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:33:14 GMT 2023-05-25T03:36:20.902 INFO:tasks.rgw_multisite_tests:/qskrps-61/ 2023-05-25T03:36:20.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.903 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:qEk59GYgwLFY3P5e/oa3vlt+F9A= 2023-05-25T03:36:20.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:33:14 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qEk59GYgwLFY3P5e/oa3vlt+F9A=', 'Content-Length': '0'} 2023-05-25T03:36:20.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.905 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.905 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.905 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.906 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:33:14 GMT 2023-05-25T03:36:20.906 INFO:tasks.rgw_multisite_tests:/qskrps-61/ 2023-05-25T03:36:20.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.906 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:qEk59GYgwLFY3P5e/oa3vlt+F9A= 2023-05-25T03:36:20.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:33:14 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qEk59GYgwLFY3P5e/oa3vlt+F9A=', 'Content-Length': '0'} 2023-05-25T03:36:20.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.908 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.908 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.909 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.909 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:33:14 GMT 2023-05-25T03:36:20.909 INFO:tasks.rgw_multisite_tests:/qskrps-61/ 2023-05-25T03:36:20.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.910 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:qEk59GYgwLFY3P5e/oa3vlt+F9A= 2023-05-25T03:36:20.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:33:14 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qEk59GYgwLFY3P5e/oa3vlt+F9A=', 'Content-Length': '0'} 2023-05-25T03:36:20.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.911 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-05-25T03:36:20.911 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.911 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-05-25T03:36:20.912 INFO:tasks.rgw_multisite_tests:ERROR: rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2023-05-25T03:36:20.912 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-05-25T03:36:20.912 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-05-25T03:36:20.913 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-05-25T03:36:20.913 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-05-25T03:36:20.913 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/../src/test/rgw/rgw_multi/tests.py", line 1824, in test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2023-05-25T03:36:20.914 INFO:tasks.rgw_multisite_tests: bucket = primary.create_bucket(gen_bucket_name()) 2023-05-25T03:36:20.914 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/tasks/rgw_multi/zone_rados.py", line 55, in create_bucket 2023-05-25T03:36:20.914 INFO:tasks.rgw_multisite_tests: return self.conn.create_bucket(name) 2023-05-25T03:36:20.915 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 618, in create_bucket 2023-05-25T03:36:20.915 INFO:tasks.rgw_multisite_tests: response = self.make_request('PUT', bucket_name, headers=headers, 2023-05-25T03:36:20.915 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 667, in make_request 2023-05-25T03:36:20.915 INFO:tasks.rgw_multisite_tests: return super(S3Connection, self).make_request( 2023-05-25T03:36:20.916 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1070, in make_request 2023-05-25T03:36:20.916 INFO:tasks.rgw_multisite_tests: return self._mexe(http_request, sender, override_num_retries, 2023-05-25T03:36:20.916 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1030, in _mexe 2023-05-25T03:36:20.917 INFO:tasks.rgw_multisite_tests: raise ex 2023-05-25T03:36:20.917 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 942, in _mexe 2023-05-25T03:36:20.917 INFO:tasks.rgw_multisite_tests: connection.request(request.method, request.path, 2023-05-25T03:36:20.917 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1234, in request 2023-05-25T03:36:20.918 INFO:tasks.rgw_multisite_tests: self._send_request(method, url, body, headers, encode_chunked) 2023-05-25T03:36:20.918 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1280, in _send_request 2023-05-25T03:36:20.918 INFO:tasks.rgw_multisite_tests: self.endheaders(body, encode_chunked=encode_chunked) 2023-05-25T03:36:20.918 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1229, in endheaders 2023-05-25T03:36:20.919 INFO:tasks.rgw_multisite_tests: self._send_output(message_body, encode_chunked=encode_chunked) 2023-05-25T03:36:20.919 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1008, in _send_output 2023-05-25T03:36:20.919 INFO:tasks.rgw_multisite_tests: self.send(msg) 2023-05-25T03:36:20.919 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 948, in send 2023-05-25T03:36:20.919 INFO:tasks.rgw_multisite_tests: self.connect() 2023-05-25T03:36:20.920 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 919, in connect 2023-05-25T03:36:20.920 INFO:tasks.rgw_multisite_tests: self.sock = self._create_connection( 2023-05-25T03:36:20.920 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/gevent/socket.py", line 101, in create_connection 2023-05-25T03:36:20.921 INFO:tasks.rgw_multisite_tests: sock.connect(sa) 2023-05-25T03:36:20.921 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 607, in connect 2023-05-25T03:36:20.921 INFO:tasks.rgw_multisite_tests: raise _SocketError(err, strerror(err)) 2023-05-25T03:36:20.921 INFO:tasks.rgw_multisite_tests:ConnectionRefusedError: [Errno 111] Connection refused 2023-05-25T03:36:20.922 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-05-25T03:36:20.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-62/ 2023-05-25T03:36:20.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-62/ 2023-05-25T03:36:20.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-25T03:36:20.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-62/ 2023-05-25T03:36:20.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:20.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:20.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:20.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:20.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:20.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.928 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.929 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.929 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.929 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:34:25 GMT 2023-05-25T03:36:20.929 INFO:tasks.rgw_multisite_tests:/qskrps-62/ 2023-05-25T03:36:20.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.930 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:TrGvGlCnXHV4QKtbiP9GiAWe/qk= 2023-05-25T03:36:20.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:34:25 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:TrGvGlCnXHV4QKtbiP9GiAWe/qk=', 'Content-Length': '0'} 2023-05-25T03:36:20.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.931 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.932 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.932 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.932 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:34:25 GMT 2023-05-25T03:36:20.932 INFO:tasks.rgw_multisite_tests:/qskrps-62/ 2023-05-25T03:36:20.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.933 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:TrGvGlCnXHV4QKtbiP9GiAWe/qk= 2023-05-25T03:36:20.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:34:25 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:TrGvGlCnXHV4QKtbiP9GiAWe/qk=', 'Content-Length': '0'} 2023-05-25T03:36:20.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.934 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.935 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.935 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.935 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:34:25 GMT 2023-05-25T03:36:20.936 INFO:tasks.rgw_multisite_tests:/qskrps-62/ 2023-05-25T03:36:20.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.936 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:TrGvGlCnXHV4QKtbiP9GiAWe/qk= 2023-05-25T03:36:20.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:34:25 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:TrGvGlCnXHV4QKtbiP9GiAWe/qk=', 'Content-Length': '0'} 2023-05-25T03:36:20.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.938 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.938 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.938 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.939 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:34:25 GMT 2023-05-25T03:36:20.939 INFO:tasks.rgw_multisite_tests:/qskrps-62/ 2023-05-25T03:36:20.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.939 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:TrGvGlCnXHV4QKtbiP9GiAWe/qk= 2023-05-25T03:36:20.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:34:25 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:TrGvGlCnXHV4QKtbiP9GiAWe/qk=', 'Content-Length': '0'} 2023-05-25T03:36:20.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.941 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.941 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.941 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.942 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:34:25 GMT 2023-05-25T03:36:20.942 INFO:tasks.rgw_multisite_tests:/qskrps-62/ 2023-05-25T03:36:20.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.942 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:TrGvGlCnXHV4QKtbiP9GiAWe/qk= 2023-05-25T03:36:20.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:34:25 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:TrGvGlCnXHV4QKtbiP9GiAWe/qk=', 'Content-Length': '0'} 2023-05-25T03:36:20.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.944 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.944 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.945 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.945 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:34:25 GMT 2023-05-25T03:36:20.945 INFO:tasks.rgw_multisite_tests:/qskrps-62/ 2023-05-25T03:36:20.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.946 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:TrGvGlCnXHV4QKtbiP9GiAWe/qk= 2023-05-25T03:36:20.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:34:25 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:TrGvGlCnXHV4QKtbiP9GiAWe/qk=', 'Content-Length': '0'} 2023-05-25T03:36:20.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.948 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.948 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.949 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.949 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:34:25 GMT 2023-05-25T03:36:20.949 INFO:tasks.rgw_multisite_tests:/qskrps-62/ 2023-05-25T03:36:20.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.950 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:TrGvGlCnXHV4QKtbiP9GiAWe/qk= 2023-05-25T03:36:20.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:34:25 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:TrGvGlCnXHV4QKtbiP9GiAWe/qk=', 'Content-Length': '0'} 2023-05-25T03:36:20.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.951 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-05-25T03:36:20.951 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.952 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-05-25T03:36:20.952 INFO:tasks.rgw_multisite_tests:ERROR: rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init 2023-05-25T03:36:20.952 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-05-25T03:36:20.953 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-05-25T03:36:20.953 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-05-25T03:36:20.953 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-05-25T03:36:20.953 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/../src/test/rgw/rgw_multi/tests.py", line 1872, in test_resharded_bucket_incremental_sync_oldest_after_data_sync_init 2023-05-25T03:36:20.954 INFO:tasks.rgw_multisite_tests: bucket = primary.create_bucket(gen_bucket_name()) 2023-05-25T03:36:20.954 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/tasks/rgw_multi/zone_rados.py", line 55, in create_bucket 2023-05-25T03:36:20.955 INFO:tasks.rgw_multisite_tests: return self.conn.create_bucket(name) 2023-05-25T03:36:20.955 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 618, in create_bucket 2023-05-25T03:36:20.956 INFO:tasks.rgw_multisite_tests: response = self.make_request('PUT', bucket_name, headers=headers, 2023-05-25T03:36:20.956 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 667, in make_request 2023-05-25T03:36:20.956 INFO:tasks.rgw_multisite_tests: return super(S3Connection, self).make_request( 2023-05-25T03:36:20.956 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1070, in make_request 2023-05-25T03:36:20.957 INFO:tasks.rgw_multisite_tests: return self._mexe(http_request, sender, override_num_retries, 2023-05-25T03:36:20.957 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1030, in _mexe 2023-05-25T03:36:20.957 INFO:tasks.rgw_multisite_tests: raise ex 2023-05-25T03:36:20.957 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 942, in _mexe 2023-05-25T03:36:20.958 INFO:tasks.rgw_multisite_tests: connection.request(request.method, request.path, 2023-05-25T03:36:20.958 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1234, in request 2023-05-25T03:36:20.958 INFO:tasks.rgw_multisite_tests: self._send_request(method, url, body, headers, encode_chunked) 2023-05-25T03:36:20.959 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1280, in _send_request 2023-05-25T03:36:20.959 INFO:tasks.rgw_multisite_tests: self.endheaders(body, encode_chunked=encode_chunked) 2023-05-25T03:36:20.959 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1229, in endheaders 2023-05-25T03:36:20.959 INFO:tasks.rgw_multisite_tests: self._send_output(message_body, encode_chunked=encode_chunked) 2023-05-25T03:36:20.960 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1008, in _send_output 2023-05-25T03:36:20.960 INFO:tasks.rgw_multisite_tests: self.send(msg) 2023-05-25T03:36:20.961 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 948, in send 2023-05-25T03:36:20.961 INFO:tasks.rgw_multisite_tests: self.connect() 2023-05-25T03:36:20.961 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 919, in connect 2023-05-25T03:36:20.962 INFO:tasks.rgw_multisite_tests: self.sock = self._create_connection( 2023-05-25T03:36:20.962 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/gevent/socket.py", line 101, in create_connection 2023-05-25T03:36:20.962 INFO:tasks.rgw_multisite_tests: sock.connect(sa) 2023-05-25T03:36:20.962 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 607, in connect 2023-05-25T03:36:20.962 INFO:tasks.rgw_multisite_tests: raise _SocketError(err, strerror(err)) 2023-05-25T03:36:20.963 INFO:tasks.rgw_multisite_tests:ConnectionRefusedError: [Errno 111] Connection refused 2023-05-25T03:36:20.963 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-05-25T03:36:20.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-63/ 2023-05-25T03:36:20.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-63/ 2023-05-25T03:36:20.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-25T03:36:20.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-63/ 2023-05-25T03:36:20.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:20.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:20.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:20.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:20.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:20.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.967 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.967 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.967 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.967 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:35:24 GMT 2023-05-25T03:36:20.968 INFO:tasks.rgw_multisite_tests:/qskrps-63/ 2023-05-25T03:36:20.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.968 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:L92XP4kkn2SQxrKpTSx9CM9yhZM= 2023-05-25T03:36:20.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:35:24 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:L92XP4kkn2SQxrKpTSx9CM9yhZM=', 'Content-Length': '0'} 2023-05-25T03:36:20.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.970 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.970 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.970 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.971 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:35:24 GMT 2023-05-25T03:36:20.971 INFO:tasks.rgw_multisite_tests:/qskrps-63/ 2023-05-25T03:36:20.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.971 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:L92XP4kkn2SQxrKpTSx9CM9yhZM= 2023-05-25T03:36:20.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:35:24 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:L92XP4kkn2SQxrKpTSx9CM9yhZM=', 'Content-Length': '0'} 2023-05-25T03:36:20.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.973 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.974 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.974 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.975 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:35:24 GMT 2023-05-25T03:36:20.975 INFO:tasks.rgw_multisite_tests:/qskrps-63/ 2023-05-25T03:36:20.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.976 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:L92XP4kkn2SQxrKpTSx9CM9yhZM= 2023-05-25T03:36:20.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:35:24 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:L92XP4kkn2SQxrKpTSx9CM9yhZM=', 'Content-Length': '0'} 2023-05-25T03:36:20.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.979 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.979 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.980 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.980 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:35:24 GMT 2023-05-25T03:36:20.981 INFO:tasks.rgw_multisite_tests:/qskrps-63/ 2023-05-25T03:36:20.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.982 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:L92XP4kkn2SQxrKpTSx9CM9yhZM= 2023-05-25T03:36:20.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:35:24 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:L92XP4kkn2SQxrKpTSx9CM9yhZM=', 'Content-Length': '0'} 2023-05-25T03:36:20.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.985 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.985 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.986 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.986 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:35:24 GMT 2023-05-25T03:36:20.986 INFO:tasks.rgw_multisite_tests:/qskrps-63/ 2023-05-25T03:36:20.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.987 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:L92XP4kkn2SQxrKpTSx9CM9yhZM= 2023-05-25T03:36:20.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:35:24 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:L92XP4kkn2SQxrKpTSx9CM9yhZM=', 'Content-Length': '0'} 2023-05-25T03:36:20.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.990 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.991 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.991 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.992 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:35:24 GMT 2023-05-25T03:36:20.992 INFO:tasks.rgw_multisite_tests:/qskrps-63/ 2023-05-25T03:36:20.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.993 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:L92XP4kkn2SQxrKpTSx9CM9yhZM= 2023-05-25T03:36:20.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:35:24 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:L92XP4kkn2SQxrKpTSx9CM9yhZM=', 'Content-Length': '0'} 2023-05-25T03:36:20.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:20.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:20.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:20.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:20.996 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:20.996 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.997 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:20.997 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 22:35:24 GMT 2023-05-25T03:36:20.998 INFO:tasks.rgw_multisite_tests:/qskrps-63/ 2023-05-25T03:36:20.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:20.999 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:L92XP4kkn2SQxrKpTSx9CM9yhZM= 2023-05-25T03:36:20.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 22:35:24 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:L92XP4kkn2SQxrKpTSx9CM9yhZM=', 'Content-Length': '0'} 2023-05-25T03:36:21.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2023-05-25T03:36:21.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-25T03:36:21.001 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-05-25T03:36:21.001 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.002 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-05-25T03:36:21.002 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_version_suspended_incremental_sync 2023-05-25T03:36:21.003 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-05-25T03:36:21.004 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-05-25T03:36:21.004 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-05-25T03:36:21.004 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-05-25T03:36:21.004 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/../src/test/rgw/rgw_multi/tests.py", line 791, in test_version_suspended_incremental_sync 2023-05-25T03:36:21.005 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2023-05-25T03:36:21.005 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/../src/test/rgw/rgw_multi/tests.py", line 398, in zonegroup_bucket_checkpoint 2023-05-25T03:36:21.005 INFO:tasks.rgw_multisite_tests: target_conn.check_bucket_eq(source_conn, bucket_name) 2023-05-25T03:36:21.005 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/tasks/rgw_multi/zone_rados.py", line 93, in check_bucket_eq 2023-05-25T03:36:21.006 INFO:tasks.rgw_multisite_tests: check_object_eq(k1_head, k2_head, False) 2023-05-25T03:36:21.006 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/tasks/rgw_multi/zone_rados.py", line 19, in check_object_eq 2023-05-25T03:36:21.006 INFO:tasks.rgw_multisite_tests: eq(k1.last_modified, k2.last_modified) 2023-05-25T03:36:21.006 INFO:tasks.rgw_multisite_tests:AssertionError: 'Wed, 24 May 2023 21:11:41 GMT' != 'Wed, 24 May 2023 21:12:52 GMT' 2023-05-25T03:36:21.007 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-05-25T03:36:21.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/ 2023-05-25T03:36:21.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/ 2023-05-25T03:36:21.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-25T03:36:21.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/ 2023-05-25T03:36:21.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.009 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:21.010 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.010 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.010 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:11:40 GMT 2023-05-25T03:36:21.010 INFO:tasks.rgw_multisite_tests:/qskrps-14/ 2023-05-25T03:36:21.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.011 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:B3uJ+rZHydWmG5yhitJbcB3rJPs= 2023-05-25T03:36:21.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:40 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:B3uJ+rZHydWmG5yhitJbcB3rJPs=', 'Content-Length': '0'} 2023-05-25T03:36:21.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000003a1cad87f9178037-00646e7d8c-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.012 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=qskrps-14 2023-05-25T03:36:21.012 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.012 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-25T03:36:21.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-25T03:36:21.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-25T03:36:21.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: No --period given, using current period=5c04936b-70ea-4415-b4eb-3aad3dc612e5 2023-05-25T03:36:21.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: [ 2023-05-25T03:36:21.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "1_1684962700.813147_143.1", 2023-05-25T03:36:21.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "last_update": "2023-05-24T21:11:40.813147Z" 2023-05-25T03:36:21.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "1_1684962392.333844_91.1", 2023-05-25T03:36:21.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "last_update": "2023-05-24T21:06:32.333844Z" 2023-05-25T03:36:21.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "", 2023-05-25T03:36:21.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "last_update": "0.000000" 2023-05-25T03:36:21.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "", 2023-05-25T03:36:21.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "last_update": "0.000000" 2023-05-25T03:36:21.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: ] 2023-05-25T03:36:21.018 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1684962700.813147_143.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-25T03:36:21.018 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-05-25T03:36:21.018 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.018 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-25T03:36:21.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-25T03:36:21.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-25T03:36:21.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:11:41.015+0000 7f0d0664e780 20 RGW-SYNC:meta: read sync status 2023-05-25T03:36:21.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:11:41.016+0000 7f0d0664e780 20 RGW-SYNC:meta: read sync status 2023-05-25T03:36:21.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "sync_status": { 2023-05-25T03:36:21.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "info": { 2023-05-25T03:36:21.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "status": "sync", 2023-05-25T03:36:21.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "num_shards": 4, 2023-05-25T03:36:21.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-25T03:36:21.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "realm_epoch": 2 2023-05-25T03:36:21.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "markers": [ 2023-05-25T03:36:21.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 0, 2023-05-25T03:36:21.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "state": 1, 2023-05-25T03:36:21.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "1_1684962700.813147_143.1", 2023-05-25T03:36:21.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 1, 2023-05-25T03:36:21.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "2023-05-24T21:11:40.813147Z", 2023-05-25T03:36:21.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "realm_epoch": 2 2023-05-25T03:36:21.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 1, 2023-05-25T03:36:21.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "state": 1, 2023-05-25T03:36:21.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "1_1684962392.333844_91.1", 2023-05-25T03:36:21.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-25T03:36:21.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "realm_epoch": 2 2023-05-25T03:36:21.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 2, 2023-05-25T03:36:21.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "state": 1, 2023-05-25T03:36:21.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "", 2023-05-25T03:36:21.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "0.000000", 2023-05-25T03:36:21.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "realm_epoch": 2 2023-05-25T03:36:21.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 3, 2023-05-25T03:36:21.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "state": 1, 2023-05-25T03:36:21.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "", 2023-05-25T03:36:21.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "0.000000", 2023-05-25T03:36:21.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "realm_epoch": 2 2023-05-25T03:36:21.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: ] 2023-05-25T03:36:21.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "full_sync": { 2023-05-25T03:36:21.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total": 1, 2023-05-25T03:36:21.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "complete": 1 2023-05-25T03:36:21.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "current_time": "2023-05-24T21:11:41Z" 2023-05-25T03:36:21.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.035 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-25T03:36:21.035 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-25T03:36:21.036 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-25T03:36:21.036 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-25T03:36:21.036 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-25T03:36:21.037 INFO:tasks.rgw_multisite_tests: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-25T03:36:21.037 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-25T03:36:21.037 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.038 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-25T03:36:21.038 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.039 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-25T03:36:21.039 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.039 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-25T03:36:21.040 INFO:tasks.rgw_multisite_tests: "marker": "1_1684962700.813147_143.1", 2023-05-25T03:36:21.040 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.040 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-25T03:36:21.041 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.041 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-24T21:11:40.813147Z", 2023-05-25T03:36:21.042 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-25T03:36:21.042 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.042 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.043 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.043 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-25T03:36:21.043 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.044 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-25T03:36:21.044 INFO:tasks.rgw_multisite_tests: "marker": "1_1684962392.333844_91.1", 2023-05-25T03:36:21.045 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.045 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.045 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.046 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-25T03:36:21.046 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-25T03:36:21.046 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.047 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.047 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.048 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-25T03:36:21.048 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.048 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-25T03:36:21.049 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-25T03:36:21.049 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.050 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.050 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.050 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-25T03:36:21.050 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-25T03:36:21.051 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.051 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.051 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.051 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-25T03:36:21.052 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.052 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-25T03:36:21.052 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-25T03:36:21.052 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.053 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.053 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.053 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-25T03:36:21.053 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-25T03:36:21.053 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.054 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.054 INFO:tasks.rgw_multisite_tests: ] 2023-05-25T03:36:21.054 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.054 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-25T03:36:21.055 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-25T03:36:21.055 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-25T03:36:21.055 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.055 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-24T21:11:41Z" 2023-05-25T03:36:21.056 INFO:tasks.rgw_multisite_tests:} 2023-05-25T03:36:21.056 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.056 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962700.813147_143.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:11:40.813147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-25T03:36:21.056 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1684962700.813147_143.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-25T03:36:21.057 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1684962700.813147_143.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-25T03:36:21.057 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-05-25T03:36:21.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/ 2023-05-25T03:36:21.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/ 2023-05-25T03:36:21.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-25T03:36:21.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/ 2023-05-25T03:36:21.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.060 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-25T03:36:21.060 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.060 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.060 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:11:41 GMT 2023-05-25T03:36:21.061 INFO:tasks.rgw_multisite_tests:/qskrps-14/ 2023-05-25T03:36:21.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.061 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:5Mv1WH25Sxq05ugzijQ9YeppWFI= 2023-05-25T03:36:21.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:11:41 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5Mv1WH25Sxq05ugzijQ9YeppWFI=', 'Content-Length': '0'} 2023-05-25T03:36:21.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f87f5650fe1fcf6-00646e7d8d-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj 2023-05-25T03:36:21.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj 2023-05-25T03:36:21.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-25T03:36:21.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/obj 2023-05-25T03:36:21.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-05-25T03:36:21.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.064 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:21.065 INFO:tasks.rgw_multisite_tests:1B2M2Y8AsgTpgAmY7PhCfg== 2023-05-25T03:36:21.065 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-25T03:36:21.065 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:11:41 GMT 2023-05-25T03:36:21.065 INFO:tasks.rgw_multisite_tests:/qskrps-14/obj 2023-05-25T03:36:21.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.066 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:6440LndfcooM5MH7mE7QEYqinK4= 2023-05-25T03:36:21.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:11:41 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:6440LndfcooM5MH7mE7QEYqinK4='} 2023-05-25T03:36:21.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b25652dab95cc86c-00646e7d8d-110d-a1'), ('Date', 'Wed, 24 May 2023 21:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.066 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created initial version id=None 2023-05-25T03:36:21.067 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=qskrps-14 2023-05-25T03:36:21.067 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-25T03:36:21.067 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-25T03:36:21.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-14 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-25T03:36:21.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-25T03:36:21.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-25T03:36:21.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:11:41.166+0000 7f9bb2287780 1 waiting to reach incremental sync.. 2023-05-25T03:36:21.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:11:46.167+0000 7f9bb2287780 1 waiting to reach incremental sync.. 2023-05-25T03:36:21.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:11:51.168+0000 7f9bb2287780 1 waiting to reach incremental sync.. 2023-05-25T03:36:21.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:11:56.170+0000 7f9bb2287780 1 bucket sync caught up with source: 2023-05-25T03:36:21.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: local status: [, , , , , , , , 00000000001.6.6, , ] 2023-05-25T03:36:21.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: remote markers: [, , , , , , , , 00000000001.6.6, , ] 2023-05-25T03:36:21.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:11:56.170+0000 7f9bb2287780 0 bucket checkpoint complete 2023-05-25T03:36:21.070 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=qskrps-14 2023-05-25T03:36:21.070 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-25T03:36:21.070 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-25T03:36:21.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-25T03:36:21.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-25T03:36:21.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-25T03:36:21.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:11:56.308+0000 7faebd007780 1 waiting to reach incremental sync.. 2023-05-25T03:36:21.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:12:01.308+0000 7faebd007780 1 waiting to reach incremental sync.. 2023-05-25T03:36:21.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:12:06.309+0000 7faebd007780 1 waiting to reach incremental sync.. 2023-05-25T03:36:21.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:12:11.311+0000 7faebd007780 1 bucket sync caught up with source: 2023-05-25T03:36:21.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: local status: [, , , , , , , , 00000000001.14.6, , ] 2023-05-25T03:36:21.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: remote markers: [, , , , , , , , 00000000001.14.6, , ] 2023-05-25T03:36:21.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:12:11.311+0000 7faebd007780 0 bucket checkpoint complete 2023-05-25T03:36:21.073 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=qskrps-14 zones={a2, a1} 2023-05-25T03:36:21.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/ 2023-05-25T03:36:21.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/ 2023-05-25T03:36:21.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-25T03:36:21.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/ 2023-05-25T03:36:21.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8001 2023-05-25T03:36:21.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-25T03:36:21.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.076 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-25T03:36:21.076 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.076 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.076 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:11 GMT 2023-05-25T03:36:21.077 INFO:tasks.rgw_multisite_tests:/qskrps-14/ 2023-05-25T03:36:21.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.077 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:MzwBoA7no3UPTBSmsVhvqJO7UXI= 2023-05-25T03:36:21.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:MzwBoA7no3UPTBSmsVhvqJO7UXI=', 'Content-Length': '0'} 2023-05-25T03:36:21.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cd975f8963a85d9e-00646e7dab-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/ 2023-05-25T03:36:21.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/ 2023-05-25T03:36:21.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-25T03:36:21.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/ 2023-05-25T03:36:21.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.080 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-25T03:36:21.081 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.081 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.081 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:11 GMT 2023-05-25T03:36:21.081 INFO:tasks.rgw_multisite_tests:/qskrps-14/ 2023-05-25T03:36:21.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.082 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:MzwBoA7no3UPTBSmsVhvqJO7UXI= 2023-05-25T03:36:21.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:MzwBoA7no3UPTBSmsVhvqJO7UXI=', 'Content-Length': '0'} 2023-05-25T03:36:21.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003de92953b58cb115-00646e7dab-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.083 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2023-05-25T03:36:21.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/ 2023-05-25T03:36:21.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/ 2023-05-25T03:36:21.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/?versions&max-keys=999 2023-05-25T03:36:21.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/?versions&max-keys=999 2023-05-25T03:36:21.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-25T03:36:21.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/?versions&max-keys=999 2023-05-25T03:36:21.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8001 2023-05-25T03:36:21.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-25T03:36:21.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.086 INFO:tasks.rgw_multisite_tests:GET 2023-05-25T03:36:21.086 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.086 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.087 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:11 GMT 2023-05-25T03:36:21.087 INFO:tasks.rgw_multisite_tests:/qskrps-14/?versions 2023-05-25T03:36:21.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.087 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:rWahvQXcFcgDsm20fzKP07CRdZs= 2023-05-25T03:36:21.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:rWahvQXcFcgDsm20fzKP07CRdZs=', 'Content-Length': '0'} 2023-05-25T03:36:21.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb34778a565a30ee-00646e7dab-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'qskrps-14999falseobjnulltrue2023-05-24T21:11:41.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-25T03:36:21.088 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj 2023-05-25T03:36:21.088 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2023-05-25T03:36:21.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/ 2023-05-25T03:36:21.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/ 2023-05-25T03:36:21.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/?versions&max-keys=999 2023-05-25T03:36:21.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/?versions&max-keys=999 2023-05-25T03:36:21.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-25T03:36:21.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/?versions&max-keys=999 2023-05-25T03:36:21.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.092 INFO:tasks.rgw_multisite_tests:GET 2023-05-25T03:36:21.092 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.092 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.092 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:11 GMT 2023-05-25T03:36:21.093 INFO:tasks.rgw_multisite_tests:/qskrps-14/?versions 2023-05-25T03:36:21.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.093 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:rWahvQXcFcgDsm20fzKP07CRdZs= 2023-05-25T03:36:21.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:rWahvQXcFcgDsm20fzKP07CRdZs=', 'Content-Length': '0'} 2023-05-25T03:36:21.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a4d57d7ce506012-00646e7dab-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'qskrps-14999falseobjnulltrue2023-05-24T21:11:41.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-25T03:36:21.094 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj 2023-05-25T03:36:21.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/ 2023-05-25T03:36:21.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/ 2023-05-25T03:36:21.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/?versions&max-keys=999 2023-05-25T03:36:21.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/?versions&max-keys=999 2023-05-25T03:36:21.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-25T03:36:21.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/?versions&max-keys=999 2023-05-25T03:36:21.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8001 2023-05-25T03:36:21.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-25T03:36:21.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.097 INFO:tasks.rgw_multisite_tests:GET 2023-05-25T03:36:21.097 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.098 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.098 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:11 GMT 2023-05-25T03:36:21.098 INFO:tasks.rgw_multisite_tests:/qskrps-14/?versions 2023-05-25T03:36:21.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.099 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:rWahvQXcFcgDsm20fzKP07CRdZs= 2023-05-25T03:36:21.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:rWahvQXcFcgDsm20fzKP07CRdZs=', 'Content-Length': '0'} 2023-05-25T03:36:21.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfe391492b2411e3-00646e7dab-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'qskrps-14999falseobjnulltrue2023-05-24T21:11:41.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-25T03:36:21.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/ 2023-05-25T03:36:21.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/ 2023-05-25T03:36:21.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/?versions&max-keys=999 2023-05-25T03:36:21.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/?versions&max-keys=999 2023-05-25T03:36:21.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-25T03:36:21.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/?versions&max-keys=999 2023-05-25T03:36:21.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.103 INFO:tasks.rgw_multisite_tests:GET 2023-05-25T03:36:21.103 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.103 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.103 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:11 GMT 2023-05-25T03:36:21.104 INFO:tasks.rgw_multisite_tests:/qskrps-14/?versions 2023-05-25T03:36:21.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.104 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:rWahvQXcFcgDsm20fzKP07CRdZs= 2023-05-25T03:36:21.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:rWahvQXcFcgDsm20fzKP07CRdZs=', 'Content-Length': '0'} 2023-05-25T03:36:21.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022293414f59031c2-00646e7dab-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'qskrps-14999falseobjnulltrue2023-05-24T21:11:41.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-25T03:36:21.105 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj 2023-05-25T03:36:21.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj 2023-05-25T03:36:21.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj 2023-05-25T03:36:21.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-25T03:36:21.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/obj?versionId=null 2023-05-25T03:36:21.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8001 2023-05-25T03:36:21.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-25T03:36:21.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.108 INFO:tasks.rgw_multisite_tests:GET 2023-05-25T03:36:21.109 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.109 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.109 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:11 GMT 2023-05-25T03:36:21.109 INFO:tasks.rgw_multisite_tests:/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.110 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:qe78chEEAKZ29B8x9kmpdOcJ1zw= 2023-05-25T03:36:21.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qe78chEEAKZ29B8x9kmpdOcJ1zw=', 'Content-Length': '0'} 2023-05-25T03:36:21.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:11:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b16d896956fd782e-00646e7dab-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj 2023-05-25T03:36:21.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj 2023-05-25T03:36:21.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-25T03:36:21.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/obj?versionId=null 2023-05-25T03:36:21.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.114 INFO:tasks.rgw_multisite_tests:GET 2023-05-25T03:36:21.114 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.114 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.114 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:11 GMT 2023-05-25T03:36:21.115 INFO:tasks.rgw_multisite_tests:/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.115 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:qe78chEEAKZ29B8x9kmpdOcJ1zw= 2023-05-25T03:36:21.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qe78chEEAKZ29B8x9kmpdOcJ1zw=', 'Content-Length': '0'} 2023-05-25T03:36:21.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:11:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000016712e6ae7788499-00646e7dab-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj 2023-05-25T03:36:21.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj 2023-05-25T03:36:21.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-25T03:36:21.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/obj?versionId=null 2023-05-25T03:36:21.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8001 2023-05-25T03:36:21.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-25T03:36:21.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.119 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-25T03:36:21.119 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.119 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.119 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:11 GMT 2023-05-25T03:36:21.120 INFO:tasks.rgw_multisite_tests:/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.120 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:pJ4J/0fPJ9kmeb1IxabNPyGN3Cg= 2023-05-25T03:36:21.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:pJ4J/0fPJ9kmeb1IxabNPyGN3Cg=', 'Content-Length': '0'} 2023-05-25T03:36:21.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:11:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004662871a439eaa4e-00646e7dab-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj 2023-05-25T03:36:21.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj 2023-05-25T03:36:21.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-25T03:36:21.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/obj?versionId=null 2023-05-25T03:36:21.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.124 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-25T03:36:21.124 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.124 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.125 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:11 GMT 2023-05-25T03:36:21.125 INFO:tasks.rgw_multisite_tests:/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.125 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:pJ4J/0fPJ9kmeb1IxabNPyGN3Cg= 2023-05-25T03:36:21.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:pJ4J/0fPJ9kmeb1IxabNPyGN3Cg=', 'Content-Length': '0'} 2023-05-25T03:36:21.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:11:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b9241e9a801cb341-00646e7dab-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.126 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj 2023-05-25T03:36:21.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj 2023-05-25T03:36:21.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj 2023-05-25T03:36:21.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-25T03:36:21.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/obj?versionId=null 2023-05-25T03:36:21.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8001 2023-05-25T03:36:21.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-25T03:36:21.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.129 INFO:tasks.rgw_multisite_tests:GET 2023-05-25T03:36:21.130 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.130 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.130 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:11 GMT 2023-05-25T03:36:21.130 INFO:tasks.rgw_multisite_tests:/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.131 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:qe78chEEAKZ29B8x9kmpdOcJ1zw= 2023-05-25T03:36:21.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qe78chEEAKZ29B8x9kmpdOcJ1zw=', 'Content-Length': '0'} 2023-05-25T03:36:21.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:11:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004ed791d419f0fabf-00646e7dab-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj 2023-05-25T03:36:21.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj 2023-05-25T03:36:21.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-25T03:36:21.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/obj?versionId=null 2023-05-25T03:36:21.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.135 INFO:tasks.rgw_multisite_tests:GET 2023-05-25T03:36:21.135 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.135 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.135 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:11 GMT 2023-05-25T03:36:21.136 INFO:tasks.rgw_multisite_tests:/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.136 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:qe78chEEAKZ29B8x9kmpdOcJ1zw= 2023-05-25T03:36:21.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:qe78chEEAKZ29B8x9kmpdOcJ1zw=', 'Content-Length': '0'} 2023-05-25T03:36:21.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:11:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006fa04035c52fec5a-00646e7dab-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/ 2023-05-25T03:36:21.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/ 2023-05-25T03:36:21.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/?prefix=obj 2023-05-25T03:36:21.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/?prefix=obj 2023-05-25T03:36:21.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-25T03:36:21.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/?prefix=obj 2023-05-25T03:36:21.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8001 2023-05-25T03:36:21.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-25T03:36:21.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.140 INFO:tasks.rgw_multisite_tests:GET 2023-05-25T03:36:21.140 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.140 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.140 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:11 GMT 2023-05-25T03:36:21.141 INFO:tasks.rgw_multisite_tests:/qskrps-14/ 2023-05-25T03:36:21.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.141 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:hMEQm9ksDbEDTkkWUFU5Xqblnyg= 2023-05-25T03:36:21.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:hMEQm9ksDbEDTkkWUFU5Xqblnyg=', 'Content-Length': '0'} 2023-05-25T03:36:21.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a13f9fe12b370ca0-00646e7dab-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'qskrps-14obj1000falseobj2023-05-24T21:11:41.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-25T03:36:21.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/ 2023-05-25T03:36:21.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/ 2023-05-25T03:36:21.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/?prefix=obj 2023-05-25T03:36:21.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/?prefix=obj 2023-05-25T03:36:21.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-25T03:36:21.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/?prefix=obj 2023-05-25T03:36:21.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.145 INFO:tasks.rgw_multisite_tests:GET 2023-05-25T03:36:21.145 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.146 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.146 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:11 GMT 2023-05-25T03:36:21.146 INFO:tasks.rgw_multisite_tests:/qskrps-14/ 2023-05-25T03:36:21.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.147 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:hMEQm9ksDbEDTkkWUFU5Xqblnyg= 2023-05-25T03:36:21.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:hMEQm9ksDbEDTkkWUFU5Xqblnyg=', 'Content-Length': '0'} 2023-05-25T03:36:21.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a512af78c73418d7-00646e7dab-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'qskrps-14obj1000falseobj2023-05-24T21:11:41.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-25T03:36:21.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj 2023-05-25T03:36:21.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj 2023-05-25T03:36:21.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-25T03:36:21.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/obj 2023-05-25T03:36:21.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8001 2023-05-25T03:36:21.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-25T03:36:21.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.150 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-25T03:36:21.150 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.151 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.151 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:11 GMT 2023-05-25T03:36:21.151 INFO:tasks.rgw_multisite_tests:/qskrps-14/obj 2023-05-25T03:36:21.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.152 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:Ajtt4n/GcpLB+i9gdbtAJWPJCr0= 2023-05-25T03:36:21.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Ajtt4n/GcpLB+i9gdbtAJWPJCr0=', 'Content-Length': '0'} 2023-05-25T03:36:21.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:11:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c39e473d0a3827c6-00646e7dab-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj 2023-05-25T03:36:21.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj 2023-05-25T03:36:21.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-25T03:36:21.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/obj 2023-05-25T03:36:21.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.155 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-25T03:36:21.155 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.155 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.156 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:11 GMT 2023-05-25T03:36:21.156 INFO:tasks.rgw_multisite_tests:/qskrps-14/obj 2023-05-25T03:36:21.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.156 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:Ajtt4n/GcpLB+i9gdbtAJWPJCr0= 2023-05-25T03:36:21.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:Ajtt4n/GcpLB+i9gdbtAJWPJCr0=', 'Content-Length': '0'} 2023-05-25T03:36:21.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:11:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fea591ddd5849b82-00646e7dab-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.157 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj 2023-05-25T03:36:21.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj 2023-05-25T03:36:21.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj 2023-05-25T03:36:21.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-25T03:36:21.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/obj 2023-05-25T03:36:21.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8001 2023-05-25T03:36:21.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-25T03:36:21.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.160 INFO:tasks.rgw_multisite_tests:GET 2023-05-25T03:36:21.160 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.160 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.160 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:11 GMT 2023-05-25T03:36:21.161 INFO:tasks.rgw_multisite_tests:/qskrps-14/obj 2023-05-25T03:36:21.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.161 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:nGEdrgj9zbOzOyVatPa3ZZoOjf4= 2023-05-25T03:36:21.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:nGEdrgj9zbOzOyVatPa3ZZoOjf4=', 'Content-Length': '0'} 2023-05-25T03:36:21.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:11:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002df7a91b05206540-00646e7dab-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj 2023-05-25T03:36:21.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj 2023-05-25T03:36:21.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-25T03:36:21.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/obj 2023-05-25T03:36:21.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.164 INFO:tasks.rgw_multisite_tests:GET 2023-05-25T03:36:21.165 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.165 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.165 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:11 GMT 2023-05-25T03:36:21.165 INFO:tasks.rgw_multisite_tests:/qskrps-14/obj 2023-05-25T03:36:21.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.166 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:nGEdrgj9zbOzOyVatPa3ZZoOjf4= 2023-05-25T03:36:21.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:nGEdrgj9zbOzOyVatPa3ZZoOjf4=', 'Content-Length': '0'} 2023-05-25T03:36:21.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:11:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b67a4a19d34781e1-00646e7dab-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/ 2023-05-25T03:36:21.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/ 2023-05-25T03:36:21.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/?prefix=obj 2023-05-25T03:36:21.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/?prefix=obj 2023-05-25T03:36:21.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-25T03:36:21.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/?prefix=obj 2023-05-25T03:36:21.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8001 2023-05-25T03:36:21.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-25T03:36:21.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.170 INFO:tasks.rgw_multisite_tests:GET 2023-05-25T03:36:21.170 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.170 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.170 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:11 GMT 2023-05-25T03:36:21.171 INFO:tasks.rgw_multisite_tests:/qskrps-14/ 2023-05-25T03:36:21.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.171 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:hMEQm9ksDbEDTkkWUFU5Xqblnyg= 2023-05-25T03:36:21.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:hMEQm9ksDbEDTkkWUFU5Xqblnyg=', 'Content-Length': '0'} 2023-05-25T03:36:21.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed362bb94a0c13ed-00646e7dab-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'qskrps-14obj1000falseobj2023-05-24T21:11:41.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-25T03:36:21.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/ 2023-05-25T03:36:21.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/ 2023-05-25T03:36:21.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/?prefix=obj 2023-05-25T03:36:21.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/?prefix=obj 2023-05-25T03:36:21.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-25T03:36:21.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/?prefix=obj 2023-05-25T03:36:21.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.175 INFO:tasks.rgw_multisite_tests:GET 2023-05-25T03:36:21.175 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.176 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.176 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:11 GMT 2023-05-25T03:36:21.176 INFO:tasks.rgw_multisite_tests:/qskrps-14/ 2023-05-25T03:36:21.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.176 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:hMEQm9ksDbEDTkkWUFU5Xqblnyg= 2023-05-25T03:36:21.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:hMEQm9ksDbEDTkkWUFU5Xqblnyg=', 'Content-Length': '0'} 2023-05-25T03:36:21.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055a8b54a978ae57e-00646e7dab-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'qskrps-14obj1000falseobj2023-05-24T21:11:41.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-25T03:36:21.177 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=qskrps-14 zones={a2, a1} 2023-05-25T03:36:21.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/ 2023-05-25T03:36:21.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/ 2023-05-25T03:36:21.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/?versioning 2023-05-25T03:36:21.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/?versioning 2023-05-25T03:36:21.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-25T03:36:21.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/?versioning 2023-05-25T03:36:21.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.179 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.180 INFO:tasks.rgw_multisite_tests: Enabled 2023-05-25T03:36:21.180 INFO:tasks.rgw_multisite_tests: Disabled 2023-05-25T03:36:21.180 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.182 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:21.182 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.182 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.182 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:11 GMT 2023-05-25T03:36:21.183 INFO:tasks.rgw_multisite_tests:/qskrps-14/?versioning 2023-05-25T03:36:21.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.183 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:+EPPPAfh130nDTh0RSPNdv4Prnw= 2023-05-25T03:36:21.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:+EPPPAfh130nDTh0RSPNdv4Prnw=', 'Content-Length': '228'} 2023-05-25T03:36:21.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000fa48085c351d9319-00646e7dab-110d-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.184 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.184 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-25T03:36:21.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-25T03:36:21.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-25T03:36:21.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: No --period given, using current period=5c04936b-70ea-4415-b4eb-3aad3dc612e5 2023-05-25T03:36:21.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: [ 2023-05-25T03:36:21.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "1_1684962731.592071_145.1", 2023-05-25T03:36:21.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "last_update": "2023-05-24T21:12:11.592071Z" 2023-05-25T03:36:21.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "1_1684962392.333844_91.1", 2023-05-25T03:36:21.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "last_update": "2023-05-24T21:06:32.333844Z" 2023-05-25T03:36:21.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "", 2023-05-25T03:36:21.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "last_update": "0.000000" 2023-05-25T03:36:21.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "", 2023-05-25T03:36:21.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "last_update": "0.000000" 2023-05-25T03:36:21.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: ] 2023-05-25T03:36:21.190 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1684962731.592071_145.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-25T03:36:21.190 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-05-25T03:36:21.190 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.190 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-25T03:36:21.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-25T03:36:21.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-25T03:36:21.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:12:11.809+0000 7fbdc0be4780 20 RGW-SYNC:meta: read sync status 2023-05-25T03:36:21.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:12:11.810+0000 7fbdc0be4780 20 RGW-SYNC:meta: read sync status 2023-05-25T03:36:21.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "sync_status": { 2023-05-25T03:36:21.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "info": { 2023-05-25T03:36:21.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "status": "sync", 2023-05-25T03:36:21.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "num_shards": 4, 2023-05-25T03:36:21.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-25T03:36:21.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "realm_epoch": 2 2023-05-25T03:36:21.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "markers": [ 2023-05-25T03:36:21.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 0, 2023-05-25T03:36:21.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "state": 1, 2023-05-25T03:36:21.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "1_1684962731.592071_145.1", 2023-05-25T03:36:21.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 1, 2023-05-25T03:36:21.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "2023-05-24T21:12:11.592071Z", 2023-05-25T03:36:21.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "realm_epoch": 2 2023-05-25T03:36:21.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 1, 2023-05-25T03:36:21.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "state": 1, 2023-05-25T03:36:21.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "1_1684962392.333844_91.1", 2023-05-25T03:36:21.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-25T03:36:21.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "realm_epoch": 2 2023-05-25T03:36:21.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 2, 2023-05-25T03:36:21.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "state": 1, 2023-05-25T03:36:21.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "", 2023-05-25T03:36:21.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "0.000000", 2023-05-25T03:36:21.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "realm_epoch": 2 2023-05-25T03:36:21.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 3, 2023-05-25T03:36:21.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "state": 1, 2023-05-25T03:36:21.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "", 2023-05-25T03:36:21.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "0.000000", 2023-05-25T03:36:21.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "realm_epoch": 2 2023-05-25T03:36:21.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: ] 2023-05-25T03:36:21.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "full_sync": { 2023-05-25T03:36:21.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total": 1, 2023-05-25T03:36:21.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "complete": 1 2023-05-25T03:36:21.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "current_time": "2023-05-24T21:12:11Z" 2023-05-25T03:36:21.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.207 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-25T03:36:21.208 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-25T03:36:21.208 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-25T03:36:21.208 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-25T03:36:21.208 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-25T03:36:21.209 INFO:tasks.rgw_multisite_tests: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-25T03:36:21.209 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-25T03:36:21.209 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.209 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-25T03:36:21.209 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.210 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-25T03:36:21.210 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.210 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-25T03:36:21.210 INFO:tasks.rgw_multisite_tests: "marker": "1_1684962731.592071_145.1", 2023-05-25T03:36:21.211 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.211 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-25T03:36:21.211 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.211 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-24T21:12:11.592071Z", 2023-05-25T03:36:21.212 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-25T03:36:21.212 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.212 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.212 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.213 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-25T03:36:21.213 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.213 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-25T03:36:21.213 INFO:tasks.rgw_multisite_tests: "marker": "1_1684962392.333844_91.1", 2023-05-25T03:36:21.213 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.214 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.214 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.214 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-25T03:36:21.214 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-25T03:36:21.215 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.215 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.215 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.215 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-25T03:36:21.216 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.216 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-25T03:36:21.216 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-25T03:36:21.216 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.217 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.217 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.217 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-25T03:36:21.217 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-25T03:36:21.217 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.218 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.218 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.218 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-25T03:36:21.218 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.219 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-25T03:36:21.219 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-25T03:36:21.219 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.219 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.220 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.220 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-25T03:36:21.220 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-25T03:36:21.220 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.221 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.221 INFO:tasks.rgw_multisite_tests: ] 2023-05-25T03:36:21.221 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.221 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-25T03:36:21.222 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-25T03:36:21.222 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-25T03:36:21.222 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.222 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-24T21:12:11Z" 2023-05-25T03:36:21.222 INFO:tasks.rgw_multisite_tests:} 2023-05-25T03:36:21.223 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.223 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962731.592071_145.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:12:11.592071Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-25T03:36:21.223 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1684962731.592071_145.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-25T03:36:21.223 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1684962731.592071_145.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-25T03:36:21.224 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-05-25T03:36:21.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/ 2023-05-25T03:36:21.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/ 2023-05-25T03:36:21.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-25T03:36:21.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/ 2023-05-25T03:36:21.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.226 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-25T03:36:21.227 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.227 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.227 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:11 GMT 2023-05-25T03:36:21.227 INFO:tasks.rgw_multisite_tests:/qskrps-14/ 2023-05-25T03:36:21.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.228 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:MzwBoA7no3UPTBSmsVhvqJO7UXI= 2023-05-25T03:36:21.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:MzwBoA7no3UPTBSmsVhvqJO7UXI=', 'Content-Length': '0'} 2023-05-25T03:36:21.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d303e78ffea3088f-00646e7dab-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj 2023-05-25T03:36:21.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj 2023-05-25T03:36:21.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-25T03:36:21.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/obj 2023-05-25T03:36:21.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-05-25T03:36:21.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.231 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:21.232 INFO:tasks.rgw_multisite_tests:1B2M2Y8AsgTpgAmY7PhCfg== 2023-05-25T03:36:21.232 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-25T03:36:21.232 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:11 GMT 2023-05-25T03:36:21.232 INFO:tasks.rgw_multisite_tests:/qskrps-14/obj 2023-05-25T03:36:21.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.233 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:98Kmj+pdBFIbUoQLH8CtlwXJNcU= 2023-05-25T03:36:21.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:12:11 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:98Kmj+pdBFIbUoQLH8CtlwXJNcU='} 2023-05-25T03:36:21.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ'), ('x-amz-request-id', 'tx00000984d0741f0fd46d7-00646e7dab-110d-a1'), ('Date', 'Wed, 24 May 2023 21:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.234 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created new version id=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-25T03:36:21.234 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=qskrps-14 2023-05-25T03:36:21.234 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-25T03:36:21.234 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-25T03:36:21.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-14 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-25T03:36:21.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-25T03:36:21.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-25T03:36:21.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:12:11.971+0000 7f4b6b563780 1 waiting for incremental sync to catch up: 2023-05-25T03:36:21.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: local status: [, , , , , , , , 00000000001.6.6, , ] 2023-05-25T03:36:21.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: remote markers: [, , , , , , , , 00000000003.9.13, , ] 2023-05-25T03:36:21.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:12:16.973+0000 7f4b6b563780 1 waiting for incremental sync to catch up: 2023-05-25T03:36:21.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: local status: [, , , , , , , , 00000000001.6.6, , ] 2023-05-25T03:36:21.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: remote markers: [, , , , , , , , 00000000003.9.13, , ] 2023-05-25T03:36:21.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:12:21.974+0000 7f4b6b563780 1 waiting for incremental sync to catch up: 2023-05-25T03:36:21.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: local status: [, , , , , , , , 00000000001.6.6, , ] 2023-05-25T03:36:21.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: remote markers: [, , , , , , , , 00000000003.9.13, , ] 2023-05-25T03:36:21.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:12:26.975+0000 7f4b6b563780 1 waiting for incremental sync to catch up: 2023-05-25T03:36:21.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: local status: [, , , , , , , , 00000000001.6.6, , ] 2023-05-25T03:36:21.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: remote markers: [, , , , , , , , 00000000003.9.13, , ] 2023-05-25T03:36:21.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:12:31.976+0000 7f4b6b563780 1 bucket sync caught up with source: 2023-05-25T03:36:21.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: local status: [, , , , , , , , 00000000003.9.13, , ] 2023-05-25T03:36:21.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: remote markers: [, , , , , , , , 00000000003.9.13, , ] 2023-05-25T03:36:21.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:12:31.976+0000 7f4b6b563780 0 bucket checkpoint complete 2023-05-25T03:36:21.239 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=qskrps-14 2023-05-25T03:36:21.239 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-25T03:36:21.239 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-25T03:36:21.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-25T03:36:21.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-25T03:36:21.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-25T03:36:21.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:12:32.110+0000 7f9050f0f780 1 waiting for incremental sync to catch up: 2023-05-25T03:36:21.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: local status: [, , , , , , , , 00000000001.14.6, , ] 2023-05-25T03:36:21.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: remote markers: [, , , , , , , , 00000000003.19.13, , ] 2023-05-25T03:36:21.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:12:37.111+0000 7f9050f0f780 1 waiting for incremental sync to catch up: 2023-05-25T03:36:21.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: local status: [, , , , , , , , 00000000001.14.6, , ] 2023-05-25T03:36:21.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: remote markers: [, , , , , , , , 00000000003.19.13, , ] 2023-05-25T03:36:21.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:12:42.112+0000 7f9050f0f780 1 waiting for incremental sync to catch up: 2023-05-25T03:36:21.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: local status: [, , , , , , , , 00000000001.14.6, , ] 2023-05-25T03:36:21.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: remote markers: [, , , , , , , , 00000000003.19.13, , ] 2023-05-25T03:36:21.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:12:47.113+0000 7f9050f0f780 1 waiting for incremental sync to catch up: 2023-05-25T03:36:21.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: local status: [, , , , , , , , 00000000001.14.6, , ] 2023-05-25T03:36:21.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: remote markers: [, , , , , , , , 00000000003.19.13, , ] 2023-05-25T03:36:21.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:12:52.116+0000 7f9050f0f780 1 bucket sync caught up with source: 2023-05-25T03:36:21.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: local status: [, , , , , , , , 00000000003.19.13, , ] 2023-05-25T03:36:21.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: remote markers: [, , , , , , , , 00000000003.19.13, , ] 2023-05-25T03:36:21.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:12:52.116+0000 7f9050f0f780 0 bucket checkpoint complete 2023-05-25T03:36:21.244 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=qskrps-14 zones={a2, a1} 2023-05-25T03:36:21.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/ 2023-05-25T03:36:21.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/ 2023-05-25T03:36:21.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-25T03:36:21.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/ 2023-05-25T03:36:21.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8001 2023-05-25T03:36:21.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-25T03:36:21.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.247 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-25T03:36:21.247 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.247 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.248 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:52 GMT 2023-05-25T03:36:21.248 INFO:tasks.rgw_multisite_tests:/qskrps-14/ 2023-05-25T03:36:21.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.248 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:0Cy+rIDC9q1jv0yqhIaIOr9Zt7A= 2023-05-25T03:36:21.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:0Cy+rIDC9q1jv0yqhIaIOr9Zt7A=', 'Content-Length': '0'} 2023-05-25T03:36:21.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008de07933e0eb021a-00646e7dd4-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/ 2023-05-25T03:36:21.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/ 2023-05-25T03:36:21.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-25T03:36:21.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/ 2023-05-25T03:36:21.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.252 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-25T03:36:21.252 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.252 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.253 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:52 GMT 2023-05-25T03:36:21.253 INFO:tasks.rgw_multisite_tests:/qskrps-14/ 2023-05-25T03:36:21.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.253 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:0Cy+rIDC9q1jv0yqhIaIOr9Zt7A= 2023-05-25T03:36:21.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:0Cy+rIDC9q1jv0yqhIaIOr9Zt7A=', 'Content-Length': '0'} 2023-05-25T03:36:21.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ec67a3315e81f04-00646e7dd4-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.254 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2023-05-25T03:36:21.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/ 2023-05-25T03:36:21.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/ 2023-05-25T03:36:21.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/?versions&max-keys=999 2023-05-25T03:36:21.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/?versions&max-keys=999 2023-05-25T03:36:21.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-25T03:36:21.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/?versions&max-keys=999 2023-05-25T03:36:21.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8001 2023-05-25T03:36:21.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-25T03:36:21.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.257 INFO:tasks.rgw_multisite_tests:GET 2023-05-25T03:36:21.257 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.258 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.258 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:52 GMT 2023-05-25T03:36:21.258 INFO:tasks.rgw_multisite_tests:/qskrps-14/?versions 2023-05-25T03:36:21.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.259 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:CWB4xyHRVKNVc7Wqh3Tn8xuA6Yo= 2023-05-25T03:36:21.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:CWB4xyHRVKNVc7Wqh3Tn8xuA6Yo=', 'Content-Length': '0'} 2023-05-25T03:36:21.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001098095c98bd5ce8-00646e7dd4-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'qskrps-14999falseobjpgpY3GcrHqwBxmj52IgrlMvqJeqfWWQtrue2023-05-24T21:12:11.852Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-05-24T21:11:41.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-25T03:36:21.260 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj 2023-05-25T03:36:21.260 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj 2023-05-25T03:36:21.260 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2023-05-25T03:36:21.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/ 2023-05-25T03:36:21.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/ 2023-05-25T03:36:21.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/?versions&max-keys=999 2023-05-25T03:36:21.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/?versions&max-keys=999 2023-05-25T03:36:21.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-25T03:36:21.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/?versions&max-keys=999 2023-05-25T03:36:21.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.263 INFO:tasks.rgw_multisite_tests:GET 2023-05-25T03:36:21.264 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.264 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.264 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:52 GMT 2023-05-25T03:36:21.264 INFO:tasks.rgw_multisite_tests:/qskrps-14/?versions 2023-05-25T03:36:21.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.265 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:CWB4xyHRVKNVc7Wqh3Tn8xuA6Yo= 2023-05-25T03:36:21.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:CWB4xyHRVKNVc7Wqh3Tn8xuA6Yo=', 'Content-Length': '0'} 2023-05-25T03:36:21.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000952e6182db509d6e-00646e7dd4-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'qskrps-14999falseobjpgpY3GcrHqwBxmj52IgrlMvqJeqfWWQtrue2023-05-24T21:12:11.852Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-05-24T21:11:41.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-25T03:36:21.266 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj 2023-05-25T03:36:21.266 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj 2023-05-25T03:36:21.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/ 2023-05-25T03:36:21.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/ 2023-05-25T03:36:21.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/?versions&max-keys=999 2023-05-25T03:36:21.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/?versions&max-keys=999 2023-05-25T03:36:21.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-25T03:36:21.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/?versions&max-keys=999 2023-05-25T03:36:21.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8001 2023-05-25T03:36:21.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-25T03:36:21.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.269 INFO:tasks.rgw_multisite_tests:GET 2023-05-25T03:36:21.270 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.270 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.270 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:52 GMT 2023-05-25T03:36:21.270 INFO:tasks.rgw_multisite_tests:/qskrps-14/?versions 2023-05-25T03:36:21.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.271 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:CWB4xyHRVKNVc7Wqh3Tn8xuA6Yo= 2023-05-25T03:36:21.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:CWB4xyHRVKNVc7Wqh3Tn8xuA6Yo=', 'Content-Length': '0'} 2023-05-25T03:36:21.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f88e9da453e89024-00646e7dd4-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'qskrps-14999falseobjpgpY3GcrHqwBxmj52IgrlMvqJeqfWWQtrue2023-05-24T21:12:11.852Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-05-24T21:11:41.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-25T03:36:21.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/ 2023-05-25T03:36:21.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/ 2023-05-25T03:36:21.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/?versions&max-keys=999 2023-05-25T03:36:21.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/?versions&max-keys=999 2023-05-25T03:36:21.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-25T03:36:21.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/?versions&max-keys=999 2023-05-25T03:36:21.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.275 INFO:tasks.rgw_multisite_tests:GET 2023-05-25T03:36:21.275 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.275 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.276 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:52 GMT 2023-05-25T03:36:21.276 INFO:tasks.rgw_multisite_tests:/qskrps-14/?versions 2023-05-25T03:36:21.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.276 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:CWB4xyHRVKNVc7Wqh3Tn8xuA6Yo= 2023-05-25T03:36:21.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:CWB4xyHRVKNVc7Wqh3Tn8xuA6Yo=', 'Content-Length': '0'} 2023-05-25T03:36:21.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fa150a230e5df29-00646e7dd4-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'qskrps-14999falseobjpgpY3GcrHqwBxmj52IgrlMvqJeqfWWQtrue2023-05-24T21:12:11.852Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-05-24T21:11:41.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-25T03:36:21.277 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj 2023-05-25T03:36:21.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj 2023-05-25T03:36:21.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj 2023-05-25T03:36:21.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-25T03:36:21.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-25T03:36:21.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-25T03:36:21.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-25T03:36:21.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8001 2023-05-25T03:36:21.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-25T03:36:21.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.280 INFO:tasks.rgw_multisite_tests:GET 2023-05-25T03:36:21.281 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.281 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.281 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:52 GMT 2023-05-25T03:36:21.281 INFO:tasks.rgw_multisite_tests:/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-25T03:36:21.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.282 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:wsFZcAhU2JLBSCRMGFMrN1vubfI= 2023-05-25T03:36:21.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wsFZcAhU2JLBSCRMGFMrN1vubfI=', 'Content-Length': '0'} 2023-05-25T03:36:21.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:12:11 GMT'), ('x-amz-version-id', 'pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f020e489be7834b0-00646e7dd4-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj 2023-05-25T03:36:21.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj 2023-05-25T03:36:21.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-25T03:36:21.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-25T03:36:21.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-25T03:36:21.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-25T03:36:21.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.286 INFO:tasks.rgw_multisite_tests:GET 2023-05-25T03:36:21.286 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.286 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.287 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:52 GMT 2023-05-25T03:36:21.287 INFO:tasks.rgw_multisite_tests:/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-25T03:36:21.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.287 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:wsFZcAhU2JLBSCRMGFMrN1vubfI= 2023-05-25T03:36:21.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wsFZcAhU2JLBSCRMGFMrN1vubfI=', 'Content-Length': '0'} 2023-05-25T03:36:21.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:12:11 GMT'), ('x-amz-version-id', 'pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000045b3a8eb15606305-00646e7dd4-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj 2023-05-25T03:36:21.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj 2023-05-25T03:36:21.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-25T03:36:21.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-25T03:36:21.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-25T03:36:21.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-25T03:36:21.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8001 2023-05-25T03:36:21.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-25T03:36:21.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.291 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-25T03:36:21.292 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.292 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.292 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:52 GMT 2023-05-25T03:36:21.293 INFO:tasks.rgw_multisite_tests:/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-25T03:36:21.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.293 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:ArfXbmIiadu1cLfI1lKemiMrfRI= 2023-05-25T03:36:21.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ArfXbmIiadu1cLfI1lKemiMrfRI=', 'Content-Length': '0'} 2023-05-25T03:36:21.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:12:11 GMT'), ('x-amz-version-id', 'pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d278122f4806f89b-00646e7dd4-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj 2023-05-25T03:36:21.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj 2023-05-25T03:36:21.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-25T03:36:21.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-25T03:36:21.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-25T03:36:21.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-25T03:36:21.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.297 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-25T03:36:21.297 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.297 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.298 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:52 GMT 2023-05-25T03:36:21.298 INFO:tasks.rgw_multisite_tests:/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-25T03:36:21.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.298 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:ArfXbmIiadu1cLfI1lKemiMrfRI= 2023-05-25T03:36:21.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:ArfXbmIiadu1cLfI1lKemiMrfRI=', 'Content-Length': '0'} 2023-05-25T03:36:21.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:12:11 GMT'), ('x-amz-version-id', 'pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f3ecb63a23b0271a-00646e7dd4-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.299 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj 2023-05-25T03:36:21.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj 2023-05-25T03:36:21.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj 2023-05-25T03:36:21.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-25T03:36:21.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-25T03:36:21.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-25T03:36:21.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-25T03:36:21.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8001 2023-05-25T03:36:21.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-25T03:36:21.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.303 INFO:tasks.rgw_multisite_tests:GET 2023-05-25T03:36:21.303 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.303 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.303 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:52 GMT 2023-05-25T03:36:21.304 INFO:tasks.rgw_multisite_tests:/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-25T03:36:21.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.304 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:wsFZcAhU2JLBSCRMGFMrN1vubfI= 2023-05-25T03:36:21.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wsFZcAhU2JLBSCRMGFMrN1vubfI=', 'Content-Length': '0'} 2023-05-25T03:36:21.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:12:11 GMT'), ('x-amz-version-id', 'pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e2144d34bd63e8b5-00646e7dd4-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj 2023-05-25T03:36:21.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj 2023-05-25T03:36:21.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-25T03:36:21.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-25T03:36:21.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-25T03:36:21.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-25T03:36:21.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.308 INFO:tasks.rgw_multisite_tests:GET 2023-05-25T03:36:21.308 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.308 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.308 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:52 GMT 2023-05-25T03:36:21.309 INFO:tasks.rgw_multisite_tests:/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-25T03:36:21.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.309 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:wsFZcAhU2JLBSCRMGFMrN1vubfI= 2023-05-25T03:36:21.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wsFZcAhU2JLBSCRMGFMrN1vubfI=', 'Content-Length': '0'} 2023-05-25T03:36:21.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:12:11 GMT'), ('x-amz-version-id', 'pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004a3e8d1e784cb911-00646e7dd4-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/ 2023-05-25T03:36:21.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/ 2023-05-25T03:36:21.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/?prefix=obj 2023-05-25T03:36:21.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/?prefix=obj 2023-05-25T03:36:21.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-25T03:36:21.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/?prefix=obj 2023-05-25T03:36:21.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8001 2023-05-25T03:36:21.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-25T03:36:21.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.313 INFO:tasks.rgw_multisite_tests:GET 2023-05-25T03:36:21.313 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.313 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.314 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:52 GMT 2023-05-25T03:36:21.314 INFO:tasks.rgw_multisite_tests:/qskrps-14/ 2023-05-25T03:36:21.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.314 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:MX4R/2vj3cFapNs0cBkZB8d2G/E= 2023-05-25T03:36:21.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:MX4R/2vj3cFapNs0cBkZB8d2G/E=', 'Content-Length': '0'} 2023-05-25T03:36:21.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a952da7ad7c291c7-00646e7dd4-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'qskrps-14obj1000falseobj2023-05-24T21:12:11.852Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-25T03:36:21.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/ 2023-05-25T03:36:21.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/ 2023-05-25T03:36:21.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/?prefix=obj 2023-05-25T03:36:21.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/?prefix=obj 2023-05-25T03:36:21.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-25T03:36:21.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/?prefix=obj 2023-05-25T03:36:21.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.318 INFO:tasks.rgw_multisite_tests:GET 2023-05-25T03:36:21.319 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.319 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.319 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:52 GMT 2023-05-25T03:36:21.319 INFO:tasks.rgw_multisite_tests:/qskrps-14/ 2023-05-25T03:36:21.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.320 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:MX4R/2vj3cFapNs0cBkZB8d2G/E= 2023-05-25T03:36:21.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:MX4R/2vj3cFapNs0cBkZB8d2G/E=', 'Content-Length': '0'} 2023-05-25T03:36:21.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005885a6b6faf3374b-00646e7dd4-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'qskrps-14obj1000falseobj2023-05-24T21:12:11.852Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-25T03:36:21.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj 2023-05-25T03:36:21.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj 2023-05-25T03:36:21.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-25T03:36:21.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/obj 2023-05-25T03:36:21.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8001 2023-05-25T03:36:21.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-25T03:36:21.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.323 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-25T03:36:21.324 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.324 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.324 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:52 GMT 2023-05-25T03:36:21.324 INFO:tasks.rgw_multisite_tests:/qskrps-14/obj 2023-05-25T03:36:21.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.325 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:u2r/WGlaLQ93C4498vxuOA54DW0= 2023-05-25T03:36:21.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:u2r/WGlaLQ93C4498vxuOA54DW0=', 'Content-Length': '0'} 2023-05-25T03:36:21.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:12:11 GMT'), ('x-amz-version-id', 'pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ed591ba734ec38dd-00646e7dd4-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj 2023-05-25T03:36:21.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj 2023-05-25T03:36:21.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-25T03:36:21.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/obj 2023-05-25T03:36:21.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.328 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-25T03:36:21.328 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.329 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.329 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:52 GMT 2023-05-25T03:36:21.329 INFO:tasks.rgw_multisite_tests:/qskrps-14/obj 2023-05-25T03:36:21.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.330 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:u2r/WGlaLQ93C4498vxuOA54DW0= 2023-05-25T03:36:21.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:u2r/WGlaLQ93C4498vxuOA54DW0=', 'Content-Length': '0'} 2023-05-25T03:36:21.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:12:11 GMT'), ('x-amz-version-id', 'pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f8f7196d697433a9-00646e7dd4-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.330 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj 2023-05-25T03:36:21.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj 2023-05-25T03:36:21.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj 2023-05-25T03:36:21.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-25T03:36:21.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-25T03:36:21.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-25T03:36:21.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-25T03:36:21.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8001 2023-05-25T03:36:21.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-25T03:36:21.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.334 INFO:tasks.rgw_multisite_tests:GET 2023-05-25T03:36:21.334 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.334 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.334 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:52 GMT 2023-05-25T03:36:21.335 INFO:tasks.rgw_multisite_tests:/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-25T03:36:21.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.335 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:wsFZcAhU2JLBSCRMGFMrN1vubfI= 2023-05-25T03:36:21.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wsFZcAhU2JLBSCRMGFMrN1vubfI=', 'Content-Length': '0'} 2023-05-25T03:36:21.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:12:11 GMT'), ('x-amz-version-id', 'pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000021e3fbdf75ca9184-00646e7dd4-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj 2023-05-25T03:36:21.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj 2023-05-25T03:36:21.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-25T03:36:21.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-25T03:36:21.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-25T03:36:21.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-25T03:36:21.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.339 INFO:tasks.rgw_multisite_tests:GET 2023-05-25T03:36:21.339 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.339 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.340 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:52 GMT 2023-05-25T03:36:21.340 INFO:tasks.rgw_multisite_tests:/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-25T03:36:21.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.340 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:wsFZcAhU2JLBSCRMGFMrN1vubfI= 2023-05-25T03:36:21.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wsFZcAhU2JLBSCRMGFMrN1vubfI=', 'Content-Length': '0'} 2023-05-25T03:36:21.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:12:11 GMT'), ('x-amz-version-id', 'pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009022ac476830c28e-00646e7dd4-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/ 2023-05-25T03:36:21.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/ 2023-05-25T03:36:21.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/?prefix=obj 2023-05-25T03:36:21.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/?prefix=obj 2023-05-25T03:36:21.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-25T03:36:21.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/?prefix=obj 2023-05-25T03:36:21.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8001 2023-05-25T03:36:21.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-25T03:36:21.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.344 INFO:tasks.rgw_multisite_tests:GET 2023-05-25T03:36:21.345 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.345 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.345 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:52 GMT 2023-05-25T03:36:21.345 INFO:tasks.rgw_multisite_tests:/qskrps-14/ 2023-05-25T03:36:21.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.346 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:MX4R/2vj3cFapNs0cBkZB8d2G/E= 2023-05-25T03:36:21.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:MX4R/2vj3cFapNs0cBkZB8d2G/E=', 'Content-Length': '0'} 2023-05-25T03:36:21.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fcdd6b4c65f7410-00646e7dd4-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'qskrps-14obj1000falseobj2023-05-24T21:12:11.852Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-25T03:36:21.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/ 2023-05-25T03:36:21.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/ 2023-05-25T03:36:21.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/?prefix=obj 2023-05-25T03:36:21.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/?prefix=obj 2023-05-25T03:36:21.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-25T03:36:21.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/?prefix=obj 2023-05-25T03:36:21.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.350 INFO:tasks.rgw_multisite_tests:GET 2023-05-25T03:36:21.350 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.350 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.351 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:52 GMT 2023-05-25T03:36:21.351 INFO:tasks.rgw_multisite_tests:/qskrps-14/ 2023-05-25T03:36:21.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.351 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:MX4R/2vj3cFapNs0cBkZB8d2G/E= 2023-05-25T03:36:21.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:MX4R/2vj3cFapNs0cBkZB8d2G/E=', 'Content-Length': '0'} 2023-05-25T03:36:21.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055089a2e7b63bac2-00646e7dd4-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'qskrps-14obj1000falseobj2023-05-24T21:12:11.852Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-25T03:36:21.352 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj 2023-05-25T03:36:21.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj 2023-05-25T03:36:21.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj 2023-05-25T03:36:21.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-25T03:36:21.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/obj?versionId=null 2023-05-25T03:36:21.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8001 2023-05-25T03:36:21.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-25T03:36:21.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.356 INFO:tasks.rgw_multisite_tests:GET 2023-05-25T03:36:21.356 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.356 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.356 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:52 GMT 2023-05-25T03:36:21.357 INFO:tasks.rgw_multisite_tests:/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.357 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:5mIWh1Gf07Go8j8HH2/IZVW9MfM= 2023-05-25T03:36:21.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5mIWh1Gf07Go8j8HH2/IZVW9MfM=', 'Content-Length': '0'} 2023-05-25T03:36:21.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:11:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f8b6f315834a15f7-00646e7dd4-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj 2023-05-25T03:36:21.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj 2023-05-25T03:36:21.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-25T03:36:21.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/obj?versionId=null 2023-05-25T03:36:21.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.361 INFO:tasks.rgw_multisite_tests:GET 2023-05-25T03:36:21.361 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.361 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.361 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:52 GMT 2023-05-25T03:36:21.362 INFO:tasks.rgw_multisite_tests:/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.362 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:5mIWh1Gf07Go8j8HH2/IZVW9MfM= 2023-05-25T03:36:21.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5mIWh1Gf07Go8j8HH2/IZVW9MfM=', 'Content-Length': '0'} 2023-05-25T03:36:21.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:11:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000017a3ec15b731da42-00646e7dd4-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj 2023-05-25T03:36:21.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj 2023-05-25T03:36:21.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-25T03:36:21.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/obj?versionId=null 2023-05-25T03:36:21.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8001 2023-05-25T03:36:21.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-25T03:36:21.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.366 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-25T03:36:21.366 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.366 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.367 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:52 GMT 2023-05-25T03:36:21.367 INFO:tasks.rgw_multisite_tests:/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.367 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:L/ZM2TaoP3N0cMpaQNyQawz+D78= 2023-05-25T03:36:21.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:L/ZM2TaoP3N0cMpaQNyQawz+D78=', 'Content-Length': '0'} 2023-05-25T03:36:21.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:11:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f7afac507df8d0f7-00646e7dd4-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj 2023-05-25T03:36:21.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj 2023-05-25T03:36:21.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-25T03:36:21.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/obj?versionId=null 2023-05-25T03:36:21.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.371 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-25T03:36:21.371 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.372 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.372 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:52 GMT 2023-05-25T03:36:21.372 INFO:tasks.rgw_multisite_tests:/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.372 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:L/ZM2TaoP3N0cMpaQNyQawz+D78= 2023-05-25T03:36:21.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:L/ZM2TaoP3N0cMpaQNyQawz+D78=', 'Content-Length': '0'} 2023-05-25T03:36:21.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:11:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c2800c1d3f39af60-00646e7dd4-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.373 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj 2023-05-25T03:36:21.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj 2023-05-25T03:36:21.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj 2023-05-25T03:36:21.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-25T03:36:21.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/obj?versionId=null 2023-05-25T03:36:21.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8001 2023-05-25T03:36:21.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-25T03:36:21.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.377 INFO:tasks.rgw_multisite_tests:GET 2023-05-25T03:36:21.377 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.377 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.377 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:52 GMT 2023-05-25T03:36:21.377 INFO:tasks.rgw_multisite_tests:/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.378 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:5mIWh1Gf07Go8j8HH2/IZVW9MfM= 2023-05-25T03:36:21.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5mIWh1Gf07Go8j8HH2/IZVW9MfM=', 'Content-Length': '0'} 2023-05-25T03:36:21.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:11:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000126a7f7051ffbc8f-00646e7dd4-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj 2023-05-25T03:36:21.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj 2023-05-25T03:36:21.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-25T03:36:21.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/obj?versionId=null 2023-05-25T03:36:21.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.382 INFO:tasks.rgw_multisite_tests:GET 2023-05-25T03:36:21.382 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.382 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.382 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:52 GMT 2023-05-25T03:36:21.383 INFO:tasks.rgw_multisite_tests:/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.383 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:5mIWh1Gf07Go8j8HH2/IZVW9MfM= 2023-05-25T03:36:21.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5mIWh1Gf07Go8j8HH2/IZVW9MfM=', 'Content-Length': '0'} 2023-05-25T03:36:21.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:11:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000029fd533e44d37849-00646e7dd4-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/ 2023-05-25T03:36:21.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/ 2023-05-25T03:36:21.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/?prefix=obj 2023-05-25T03:36:21.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/?prefix=obj 2023-05-25T03:36:21.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-25T03:36:21.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/?prefix=obj 2023-05-25T03:36:21.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8001 2023-05-25T03:36:21.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-25T03:36:21.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.387 INFO:tasks.rgw_multisite_tests:GET 2023-05-25T03:36:21.387 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.387 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.388 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:52 GMT 2023-05-25T03:36:21.388 INFO:tasks.rgw_multisite_tests:/qskrps-14/ 2023-05-25T03:36:21.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.388 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:MX4R/2vj3cFapNs0cBkZB8d2G/E= 2023-05-25T03:36:21.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:MX4R/2vj3cFapNs0cBkZB8d2G/E=', 'Content-Length': '0'} 2023-05-25T03:36:21.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015b0f6efd3848946-00646e7dd4-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'qskrps-14obj1000falseobj2023-05-24T21:12:11.852Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-25T03:36:21.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/ 2023-05-25T03:36:21.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/ 2023-05-25T03:36:21.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/?prefix=obj 2023-05-25T03:36:21.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/?prefix=obj 2023-05-25T03:36:21.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-25T03:36:21.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/?prefix=obj 2023-05-25T03:36:21.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.392 INFO:tasks.rgw_multisite_tests:GET 2023-05-25T03:36:21.393 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.393 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.393 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:52 GMT 2023-05-25T03:36:21.393 INFO:tasks.rgw_multisite_tests:/qskrps-14/ 2023-05-25T03:36:21.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.394 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:MX4R/2vj3cFapNs0cBkZB8d2G/E= 2023-05-25T03:36:21.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:MX4R/2vj3cFapNs0cBkZB8d2G/E=', 'Content-Length': '0'} 2023-05-25T03:36:21.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037f8ee460fe36ef1-00646e7dd4-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'qskrps-14obj1000falseobj2023-05-24T21:12:11.852Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-25T03:36:21.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj 2023-05-25T03:36:21.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj 2023-05-25T03:36:21.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-25T03:36:21.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/obj 2023-05-25T03:36:21.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8001 2023-05-25T03:36:21.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-25T03:36:21.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.397 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-25T03:36:21.397 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.398 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.398 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:52 GMT 2023-05-25T03:36:21.398 INFO:tasks.rgw_multisite_tests:/qskrps-14/obj 2023-05-25T03:36:21.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.399 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:u2r/WGlaLQ93C4498vxuOA54DW0= 2023-05-25T03:36:21.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:u2r/WGlaLQ93C4498vxuOA54DW0=', 'Content-Length': '0'} 2023-05-25T03:36:21.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:12:11 GMT'), ('x-amz-version-id', 'pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006144a267bcf3946e-00646e7dd4-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj 2023-05-25T03:36:21.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj 2023-05-25T03:36:21.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-25T03:36:21.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/obj 2023-05-25T03:36:21.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.402 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-25T03:36:21.402 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.402 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.403 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:52 GMT 2023-05-25T03:36:21.403 INFO:tasks.rgw_multisite_tests:/qskrps-14/obj 2023-05-25T03:36:21.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.403 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:u2r/WGlaLQ93C4498vxuOA54DW0= 2023-05-25T03:36:21.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:u2r/WGlaLQ93C4498vxuOA54DW0=', 'Content-Length': '0'} 2023-05-25T03:36:21.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:12:11 GMT'), ('x-amz-version-id', 'pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000a78f4ceec3cfd3a-00646e7dd4-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.404 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj 2023-05-25T03:36:21.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj 2023-05-25T03:36:21.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj 2023-05-25T03:36:21.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-25T03:36:21.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-25T03:36:21.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-25T03:36:21.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-25T03:36:21.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8001 2023-05-25T03:36:21.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-25T03:36:21.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.407 INFO:tasks.rgw_multisite_tests:GET 2023-05-25T03:36:21.408 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.408 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.408 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:52 GMT 2023-05-25T03:36:21.408 INFO:tasks.rgw_multisite_tests:/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-25T03:36:21.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.409 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:wsFZcAhU2JLBSCRMGFMrN1vubfI= 2023-05-25T03:36:21.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wsFZcAhU2JLBSCRMGFMrN1vubfI=', 'Content-Length': '0'} 2023-05-25T03:36:21.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:12:11 GMT'), ('x-amz-version-id', 'pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000016204aa5eb88de54-00646e7dd4-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj 2023-05-25T03:36:21.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj 2023-05-25T03:36:21.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-25T03:36:21.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-25T03:36:21.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-25T03:36:21.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-25T03:36:21.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.413 INFO:tasks.rgw_multisite_tests:GET 2023-05-25T03:36:21.413 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.413 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.413 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:52 GMT 2023-05-25T03:36:21.414 INFO:tasks.rgw_multisite_tests:/qskrps-14/obj?versionId=pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ 2023-05-25T03:36:21.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.414 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:wsFZcAhU2JLBSCRMGFMrN1vubfI= 2023-05-25T03:36:21.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:wsFZcAhU2JLBSCRMGFMrN1vubfI=', 'Content-Length': '0'} 2023-05-25T03:36:21.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:12:11 GMT'), ('x-amz-version-id', 'pgpY3GcrHqwBxmj52IgrlMvqJeqfWWQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ec8961372c7903a8-00646e7dd4-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/ 2023-05-25T03:36:21.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/ 2023-05-25T03:36:21.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/?prefix=obj 2023-05-25T03:36:21.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/?prefix=obj 2023-05-25T03:36:21.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-25T03:36:21.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/?prefix=obj 2023-05-25T03:36:21.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8001 2023-05-25T03:36:21.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-25T03:36:21.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.418 INFO:tasks.rgw_multisite_tests:GET 2023-05-25T03:36:21.418 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.418 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.419 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:52 GMT 2023-05-25T03:36:21.419 INFO:tasks.rgw_multisite_tests:/qskrps-14/ 2023-05-25T03:36:21.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.419 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:MX4R/2vj3cFapNs0cBkZB8d2G/E= 2023-05-25T03:36:21.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:MX4R/2vj3cFapNs0cBkZB8d2G/E=', 'Content-Length': '0'} 2023-05-25T03:36:21.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098dde42f3beaac93-00646e7dd4-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'qskrps-14obj1000falseobj2023-05-24T21:12:11.852Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-25T03:36:21.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/ 2023-05-25T03:36:21.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/ 2023-05-25T03:36:21.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/?prefix=obj 2023-05-25T03:36:21.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/?prefix=obj 2023-05-25T03:36:21.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-25T03:36:21.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/?prefix=obj 2023-05-25T03:36:21.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.423 INFO:tasks.rgw_multisite_tests:GET 2023-05-25T03:36:21.423 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.424 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.424 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:52 GMT 2023-05-25T03:36:21.424 INFO:tasks.rgw_multisite_tests:/qskrps-14/ 2023-05-25T03:36:21.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.425 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:MX4R/2vj3cFapNs0cBkZB8d2G/E= 2023-05-25T03:36:21.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:MX4R/2vj3cFapNs0cBkZB8d2G/E=', 'Content-Length': '0'} 2023-05-25T03:36:21.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067da7d2ad336a08d-00646e7dd4-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'qskrps-14obj1000falseobj2023-05-24T21:12:11.852Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-25T03:36:21.426 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=qskrps-14 zones={a2, a1} 2023-05-25T03:36:21.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/ 2023-05-25T03:36:21.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/ 2023-05-25T03:36:21.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/?versioning 2023-05-25T03:36:21.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/?versioning 2023-05-25T03:36:21.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-25T03:36:21.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/?versioning 2023-05-25T03:36:21.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.427 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.428 INFO:tasks.rgw_multisite_tests: Suspended 2023-05-25T03:36:21.428 INFO:tasks.rgw_multisite_tests: Disabled 2023-05-25T03:36:21.428 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.430 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:21.430 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.430 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.430 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:52 GMT 2023-05-25T03:36:21.431 INFO:tasks.rgw_multisite_tests:/qskrps-14/?versioning 2023-05-25T03:36:21.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.431 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:GjPPJWLd7kDpY5V9WUOh14mt04c= 2023-05-25T03:36:21.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:GjPPJWLd7kDpY5V9WUOh14mt04c=', 'Content-Length': '230'} 2023-05-25T03:36:21.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000007b8d5c7b8d31b52c-00646e7dd4-110d-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.432 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.432 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-25T03:36:21.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-25T03:36:21.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-25T03:36:21.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: No --period given, using current period=5c04936b-70ea-4415-b4eb-3aad3dc612e5 2023-05-25T03:36:21.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: [ 2023-05-25T03:36:21.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "1_1684962772.506122_148.1", 2023-05-25T03:36:21.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "last_update": "2023-05-24T21:12:52.506122Z" 2023-05-25T03:36:21.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "1_1684962392.333844_91.1", 2023-05-25T03:36:21.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "last_update": "2023-05-24T21:06:32.333844Z" 2023-05-25T03:36:21.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "", 2023-05-25T03:36:21.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "last_update": "0.000000" 2023-05-25T03:36:21.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "", 2023-05-25T03:36:21.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "last_update": "0.000000" 2023-05-25T03:36:21.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: ] 2023-05-25T03:36:21.438 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1684962772.506122_148.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-25T03:36:21.438 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-05-25T03:36:21.438 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.438 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-25T03:36:21.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-25T03:36:21.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-25T03:36:21.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:12:52.704+0000 7fb3a5737780 20 RGW-SYNC:meta: read sync status 2023-05-25T03:36:21.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:12:52.705+0000 7fb3a5737780 20 RGW-SYNC:meta: read sync status 2023-05-25T03:36:21.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "sync_status": { 2023-05-25T03:36:21.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "info": { 2023-05-25T03:36:21.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "status": "sync", 2023-05-25T03:36:21.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "num_shards": 4, 2023-05-25T03:36:21.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-25T03:36:21.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "realm_epoch": 2 2023-05-25T03:36:21.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "markers": [ 2023-05-25T03:36:21.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 0, 2023-05-25T03:36:21.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "state": 1, 2023-05-25T03:36:21.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "1_1684962772.506122_148.1", 2023-05-25T03:36:21.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 1, 2023-05-25T03:36:21.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "2023-05-24T21:12:52.506122Z", 2023-05-25T03:36:21.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "realm_epoch": 2 2023-05-25T03:36:21.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 1, 2023-05-25T03:36:21.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "state": 1, 2023-05-25T03:36:21.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "1_1684962392.333844_91.1", 2023-05-25T03:36:21.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-25T03:36:21.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "realm_epoch": 2 2023-05-25T03:36:21.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 2, 2023-05-25T03:36:21.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "state": 1, 2023-05-25T03:36:21.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "", 2023-05-25T03:36:21.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "0.000000", 2023-05-25T03:36:21.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "realm_epoch": 2 2023-05-25T03:36:21.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 3, 2023-05-25T03:36:21.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "state": 1, 2023-05-25T03:36:21.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "", 2023-05-25T03:36:21.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "0.000000", 2023-05-25T03:36:21.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "realm_epoch": 2 2023-05-25T03:36:21.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: ] 2023-05-25T03:36:21.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "full_sync": { 2023-05-25T03:36:21.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total": 1, 2023-05-25T03:36:21.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "complete": 1 2023-05-25T03:36:21.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "current_time": "2023-05-24T21:12:52Z" 2023-05-25T03:36:21.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.455 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-25T03:36:21.455 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-25T03:36:21.456 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-25T03:36:21.456 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-25T03:36:21.456 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-25T03:36:21.456 INFO:tasks.rgw_multisite_tests: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-25T03:36:21.457 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-25T03:36:21.457 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.457 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-25T03:36:21.457 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.458 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-25T03:36:21.458 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.458 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-25T03:36:21.458 INFO:tasks.rgw_multisite_tests: "marker": "1_1684962772.506122_148.1", 2023-05-25T03:36:21.458 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.459 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-25T03:36:21.459 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.459 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-24T21:12:52.506122Z", 2023-05-25T03:36:21.459 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-25T03:36:21.460 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.460 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.460 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.460 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-25T03:36:21.461 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.461 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-25T03:36:21.461 INFO:tasks.rgw_multisite_tests: "marker": "1_1684962392.333844_91.1", 2023-05-25T03:36:21.461 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.462 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.462 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.462 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-25T03:36:21.462 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-25T03:36:21.462 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.463 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.463 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.463 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-25T03:36:21.463 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.464 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-25T03:36:21.464 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-25T03:36:21.464 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.464 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.465 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.465 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-25T03:36:21.465 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-25T03:36:21.465 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.465 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.466 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.466 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-25T03:36:21.466 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.466 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-25T03:36:21.467 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-25T03:36:21.467 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.467 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.467 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.468 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-25T03:36:21.468 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-25T03:36:21.468 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.468 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.469 INFO:tasks.rgw_multisite_tests: ] 2023-05-25T03:36:21.469 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.469 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-25T03:36:21.469 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-25T03:36:21.469 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-25T03:36:21.470 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.470 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-24T21:12:52Z" 2023-05-25T03:36:21.470 INFO:tasks.rgw_multisite_tests:} 2023-05-25T03:36:21.470 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.471 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962772.506122_148.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:12:52.506122Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-25T03:36:21.471 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1684962772.506122_148.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-25T03:36:21.471 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1684962772.506122_148.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-25T03:36:21.471 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-05-25T03:36:21.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/ 2023-05-25T03:36:21.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/ 2023-05-25T03:36:21.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-25T03:36:21.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/ 2023-05-25T03:36:21.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.474 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-25T03:36:21.474 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.475 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.475 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:52 GMT 2023-05-25T03:36:21.475 INFO:tasks.rgw_multisite_tests:/qskrps-14/ 2023-05-25T03:36:21.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.476 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:0Cy+rIDC9q1jv0yqhIaIOr9Zt7A= 2023-05-25T03:36:21.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:0Cy+rIDC9q1jv0yqhIaIOr9Zt7A=', 'Content-Length': '0'} 2023-05-25T03:36:21.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000054d4ce6feb2859fc-00646e7dd4-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj 2023-05-25T03:36:21.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj 2023-05-25T03:36:21.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-25T03:36:21.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/obj 2023-05-25T03:36:21.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-05-25T03:36:21.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.479 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:21.480 INFO:tasks.rgw_multisite_tests:1B2M2Y8AsgTpgAmY7PhCfg== 2023-05-25T03:36:21.480 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-25T03:36:21.480 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:12:52 GMT 2023-05-25T03:36:21.481 INFO:tasks.rgw_multisite_tests:/qskrps-14/obj 2023-05-25T03:36:21.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.481 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:5LN+6rfDBoXovo1GkJhlcj/BHhk= 2023-05-25T03:36:21.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:12:52 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:5LN+6rfDBoXovo1GkJhlcj/BHhk='} 2023-05-25T03:36:21.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000051e347def97ed52-00646e7dd4-110d-a1'), ('Date', 'Wed, 24 May 2023 21:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.483 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created null version id=None 2023-05-25T03:36:21.483 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=qskrps-14 2023-05-25T03:36:21.483 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-25T03:36:21.484 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-25T03:36:21.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-14 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-25T03:36:21.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-25T03:36:21.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-25T03:36:21.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:12:52.866+0000 7f3ad87e1780 1 waiting for incremental sync to catch up: 2023-05-25T03:36:21.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: local status: [, , , , , , , , 00000000003.9.13, , ] 2023-05-25T03:36:21.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: remote markers: [, , , , , , , , 00000000005.13.18, , ] 2023-05-25T03:36:21.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:12:57.868+0000 7f3ad87e1780 1 waiting for incremental sync to catch up: 2023-05-25T03:36:21.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: local status: [, , , , , , , , 00000000003.9.13, , ] 2023-05-25T03:36:21.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: remote markers: [, , , , , , , , 00000000005.13.18, , ] 2023-05-25T03:36:21.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:13:02.869+0000 7f3ad87e1780 1 waiting for incremental sync to catch up: 2023-05-25T03:36:21.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: local status: [, , , , , , , , 00000000003.9.13, , ] 2023-05-25T03:36:21.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: remote markers: [, , , , , , , , 00000000005.13.18, , ] 2023-05-25T03:36:21.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:13:07.870+0000 7f3ad87e1780 1 waiting for incremental sync to catch up: 2023-05-25T03:36:21.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: local status: [, , , , , , , , 00000000003.9.13, , ] 2023-05-25T03:36:21.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: remote markers: [, , , , , , , , 00000000005.13.18, , ] 2023-05-25T03:36:21.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:13:12.871+0000 7f3ad87e1780 1 bucket sync caught up with source: 2023-05-25T03:36:21.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: local status: [, , , , , , , , 00000000005.13.18, , ] 2023-05-25T03:36:21.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: remote markers: [, , , , , , , , 00000000005.13.18, , ] 2023-05-25T03:36:21.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:13:12.871+0000 7f3ad87e1780 0 bucket checkpoint complete 2023-05-25T03:36:21.492 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=qskrps-14 2023-05-25T03:36:21.492 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-25T03:36:21.492 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-25T03:36:21.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-25T03:36:21.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-25T03:36:21.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-25T03:36:21.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:13:13.002+0000 7f1121e81780 1 bucket sync caught up with source: 2023-05-25T03:36:21.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: local status: [, , , , , , , , 00000000005.25.18, , ] 2023-05-25T03:36:21.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: remote markers: [, , , , , , , , 00000000005.25.18, , ] 2023-05-25T03:36:21.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:13:13.002+0000 7f1121e81780 0 bucket checkpoint complete 2023-05-25T03:36:21.494 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=qskrps-14 zones={a2, a1} 2023-05-25T03:36:21.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/ 2023-05-25T03:36:21.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/ 2023-05-25T03:36:21.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-25T03:36:21.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/ 2023-05-25T03:36:21.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8001 2023-05-25T03:36:21.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-25T03:36:21.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.497 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-25T03:36:21.497 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.497 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.498 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:13:13 GMT 2023-05-25T03:36:21.498 INFO:tasks.rgw_multisite_tests:/qskrps-14/ 2023-05-25T03:36:21.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.498 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:+G9FIAuwt8Cy3G5h79fJJ8pZS4Q= 2023-05-25T03:36:21.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:13 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:+G9FIAuwt8Cy3G5h79fJJ8pZS4Q=', 'Content-Length': '0'} 2023-05-25T03:36:21.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000106d672a3b6aa2a3-00646e7de9-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:13:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/ 2023-05-25T03:36:21.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/ 2023-05-25T03:36:21.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-25T03:36:21.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/ 2023-05-25T03:36:21.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.502 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-25T03:36:21.502 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.502 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.502 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:13:13 GMT 2023-05-25T03:36:21.502 INFO:tasks.rgw_multisite_tests:/qskrps-14/ 2023-05-25T03:36:21.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.503 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:+G9FIAuwt8Cy3G5h79fJJ8pZS4Q= 2023-05-25T03:36:21.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:13 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:+G9FIAuwt8Cy3G5h79fJJ8pZS4Q=', 'Content-Length': '0'} 2023-05-25T03:36:21.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000054e055d6d4c732ce-00646e7de9-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:13:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.504 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2023-05-25T03:36:21.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/ 2023-05-25T03:36:21.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/ 2023-05-25T03:36:21.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/?versions&max-keys=999 2023-05-25T03:36:21.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/?versions&max-keys=999 2023-05-25T03:36:21.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-25T03:36:21.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/?versions&max-keys=999 2023-05-25T03:36:21.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8001 2023-05-25T03:36:21.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-25T03:36:21.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.507 INFO:tasks.rgw_multisite_tests:GET 2023-05-25T03:36:21.507 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.507 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.508 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:13:13 GMT 2023-05-25T03:36:21.508 INFO:tasks.rgw_multisite_tests:/qskrps-14/?versions 2023-05-25T03:36:21.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.508 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:7Wr1QbYXljXJIwCH3+iek4lhITc= 2023-05-25T03:36:21.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:13 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:7Wr1QbYXljXJIwCH3+iek4lhITc=', 'Content-Length': '0'} 2023-05-25T03:36:21.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f40f2d14507851e7-00646e7de9-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:13:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'qskrps-14999falseobjnulltrue2023-05-24T21:12:52.749Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpgpY3GcrHqwBxmj52IgrlMvqJeqfWWQfalse2023-05-24T21:12:11.852Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-25T03:36:21.509 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj 2023-05-25T03:36:21.510 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj 2023-05-25T03:36:21.510 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2023-05-25T03:36:21.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/ 2023-05-25T03:36:21.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/ 2023-05-25T03:36:21.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/?versions&max-keys=999 2023-05-25T03:36:21.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/?versions&max-keys=999 2023-05-25T03:36:21.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-25T03:36:21.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/?versions&max-keys=999 2023-05-25T03:36:21.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.513 INFO:tasks.rgw_multisite_tests:GET 2023-05-25T03:36:21.513 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.514 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.514 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:13:13 GMT 2023-05-25T03:36:21.514 INFO:tasks.rgw_multisite_tests:/qskrps-14/?versions 2023-05-25T03:36:21.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.514 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:7Wr1QbYXljXJIwCH3+iek4lhITc= 2023-05-25T03:36:21.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:13 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:7Wr1QbYXljXJIwCH3+iek4lhITc=', 'Content-Length': '0'} 2023-05-25T03:36:21.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065da53aa29eddddb-00646e7de9-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:13:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'qskrps-14999falseobjnulltrue2023-05-24T21:12:52.749Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpgpY3GcrHqwBxmj52IgrlMvqJeqfWWQfalse2023-05-24T21:12:11.852Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-25T03:36:21.515 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj 2023-05-25T03:36:21.516 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj 2023-05-25T03:36:21.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/ 2023-05-25T03:36:21.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/ 2023-05-25T03:36:21.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/?versions&max-keys=999 2023-05-25T03:36:21.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/?versions&max-keys=999 2023-05-25T03:36:21.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-25T03:36:21.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/?versions&max-keys=999 2023-05-25T03:36:21.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8001 2023-05-25T03:36:21.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-25T03:36:21.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.519 INFO:tasks.rgw_multisite_tests:GET 2023-05-25T03:36:21.519 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.519 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.520 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:13:13 GMT 2023-05-25T03:36:21.520 INFO:tasks.rgw_multisite_tests:/qskrps-14/?versions 2023-05-25T03:36:21.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.520 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:7Wr1QbYXljXJIwCH3+iek4lhITc= 2023-05-25T03:36:21.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:13 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:7Wr1QbYXljXJIwCH3+iek4lhITc=', 'Content-Length': '0'} 2023-05-25T03:36:21.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abd29942ef5053cd-00646e7de9-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:13:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'qskrps-14999falseobjnulltrue2023-05-24T21:12:52.749Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpgpY3GcrHqwBxmj52IgrlMvqJeqfWWQfalse2023-05-24T21:12:11.852Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-25T03:36:21.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/ 2023-05-25T03:36:21.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/ 2023-05-25T03:36:21.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/?versions&max-keys=999 2023-05-25T03:36:21.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/?versions&max-keys=999 2023-05-25T03:36:21.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-25T03:36:21.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/?versions&max-keys=999 2023-05-25T03:36:21.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.524 INFO:tasks.rgw_multisite_tests:GET 2023-05-25T03:36:21.525 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.525 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.525 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:13:13 GMT 2023-05-25T03:36:21.525 INFO:tasks.rgw_multisite_tests:/qskrps-14/?versions 2023-05-25T03:36:21.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.526 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:7Wr1QbYXljXJIwCH3+iek4lhITc= 2023-05-25T03:36:21.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:13 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:7Wr1QbYXljXJIwCH3+iek4lhITc=', 'Content-Length': '0'} 2023-05-25T03:36:21.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbe3d0766389b3ab-00646e7de9-110d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 24 May 2023 21:13:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'qskrps-14999falseobjnulltrue2023-05-24T21:12:52.749Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpgpY3GcrHqwBxmj52IgrlMvqJeqfWWQfalse2023-05-24T21:12:11.852Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-25T03:36:21.527 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj 2023-05-25T03:36:21.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj 2023-05-25T03:36:21.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj 2023-05-25T03:36:21.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-25T03:36:21.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/obj?versionId=null 2023-05-25T03:36:21.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8001 2023-05-25T03:36:21.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-25T03:36:21.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.530 INFO:tasks.rgw_multisite_tests:GET 2023-05-25T03:36:21.530 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.530 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.531 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:13:13 GMT 2023-05-25T03:36:21.531 INFO:tasks.rgw_multisite_tests:/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.531 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:geJrMH2kqjS8vWuG3CWZnSBL9No= 2023-05-25T03:36:21.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:13 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:geJrMH2kqjS8vWuG3CWZnSBL9No=', 'Content-Length': '0'} 2023-05-25T03:36:21.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:11:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f40d4ef35b43395b-00646e7de9-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:13:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj 2023-05-25T03:36:21.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj 2023-05-25T03:36:21.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-25T03:36:21.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/obj?versionId=null 2023-05-25T03:36:21.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.535 INFO:tasks.rgw_multisite_tests:GET 2023-05-25T03:36:21.535 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.536 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.536 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:13:13 GMT 2023-05-25T03:36:21.536 INFO:tasks.rgw_multisite_tests:/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.537 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:geJrMH2kqjS8vWuG3CWZnSBL9No= 2023-05-25T03:36:21.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:13 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:geJrMH2kqjS8vWuG3CWZnSBL9No=', 'Content-Length': '0'} 2023-05-25T03:36:21.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:12:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000008ac4d1dad44ad92-00646e7de9-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:13:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj 2023-05-25T03:36:21.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj 2023-05-25T03:36:21.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-25T03:36:21.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/obj?versionId=null 2023-05-25T03:36:21.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8001 2023-05-25T03:36:21.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-25T03:36:21.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.540 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-25T03:36:21.541 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.541 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.541 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:13:13 GMT 2023-05-25T03:36:21.541 INFO:tasks.rgw_multisite_tests:/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.542 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:CKNiyvqSSzAv/wUysbKBZN47cdg= 2023-05-25T03:36:21.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:13 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:CKNiyvqSSzAv/wUysbKBZN47cdg=', 'Content-Length': '0'} 2023-05-25T03:36:21.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:11:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001f0adb49075a613d-00646e7de9-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:13:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj 2023-05-25T03:36:21.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj 2023-05-25T03:36:21.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-25T03:36:21.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-14/obj?versionId=null 2023-05-25T03:36:21.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.545 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-25T03:36:21.546 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.546 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.546 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:13:13 GMT 2023-05-25T03:36:21.546 INFO:tasks.rgw_multisite_tests:/qskrps-14/obj?versionId=null 2023-05-25T03:36:21.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.547 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:CKNiyvqSSzAv/wUysbKBZN47cdg= 2023-05-25T03:36:21.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:13:13 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:CKNiyvqSSzAv/wUysbKBZN47cdg=', 'Content-Length': '0'} 2023-05-25T03:36:21.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 24 May 2023 21:12:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000fdf299548c9d7a5-00646e7de9-110d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 24 May 2023 21:13:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.548 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj 2023-05-25T03:36:21.548 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-05-25T03:36:21.548 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.548 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-05-25T03:36:21.549 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_datalog_autotrim 2023-05-25T03:36:21.549 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-05-25T03:36:21.549 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-05-25T03:36:21.550 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-05-25T03:36:21.550 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-05-25T03:36:21.550 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/../src/test/rgw/rgw_multi/tests.py", line 1004, in test_datalog_autotrim 2023-05-25T03:36:21.550 INFO:tasks.rgw_multisite_tests: assert before_trim < after_trim, "any datalog entries must be newer than trim" 2023-05-25T03:36:21.551 INFO:tasks.rgw_multisite_tests:AssertionError: any datalog entries must be newer than trim 2023-05-25T03:36:21.551 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-05-25T03:36:21.551 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=qskrps-33 2023-05-25T03:36:21.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-33/ 2023-05-25T03:36:21.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-33/ 2023-05-25T03:36:21.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-25T03:36:21.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-33/ 2023-05-25T03:36:21.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.554 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:21.554 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.555 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.555 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:14:40 GMT 2023-05-25T03:36:21.555 INFO:tasks.rgw_multisite_tests:/qskrps-33/ 2023-05-25T03:36:21.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.555 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:cWf2YuvvAzCJuI5VU3LWTtW/M/o= 2023-05-25T03:36:21.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:40 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:cWf2YuvvAzCJuI5VU3LWTtW/M/o=', 'Content-Length': '0'} 2023-05-25T03:36:21.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000006fcd9d2f88e07927-00646e7e40-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:14:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.556 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a2 name=qskrps-34 2023-05-25T03:36:21.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-34/ 2023-05-25T03:36:21.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-34/ 2023-05-25T03:36:21.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-25T03:36:21.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-34/ 2023-05-25T03:36:21.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8001 2023-05-25T03:36:21.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-25T03:36:21.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.560 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:21.560 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.560 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.560 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:14:40 GMT 2023-05-25T03:36:21.560 INFO:tasks.rgw_multisite_tests:/qskrps-34/ 2023-05-25T03:36:21.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.561 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:4enNehl9rBmrowZyCVg7K3B3oZQ= 2023-05-25T03:36:21.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:40 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:4enNehl9rBmrowZyCVg7K3B3oZQ=', 'Content-Length': '0'} 2023-05-25T03:36:21.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000a76520c1a3013553-00646e7e40-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:14:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-33/ 2023-05-25T03:36:21.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-33/ 2023-05-25T03:36:21.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-25T03:36:21.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-33/ 2023-05-25T03:36:21.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.564 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-25T03:36:21.564 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.565 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.565 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:14:40 GMT 2023-05-25T03:36:21.565 INFO:tasks.rgw_multisite_tests:/qskrps-33/ 2023-05-25T03:36:21.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.566 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:RPCoaNhOGgVSmf8XjfgM7KsXgNQ= 2023-05-25T03:36:21.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:40 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:RPCoaNhOGgVSmf8XjfgM7KsXgNQ=', 'Content-Length': '0'} 2023-05-25T03:36:21.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005d7dc13b603cb431-00646e7e40-110d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:14:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-33/key 2023-05-25T03:36:21.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-33/key 2023-05-25T03:36:21.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-25T03:36:21.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-33/key 2023-05-25T03:36:21.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2023-05-25T03:36:21.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.569 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:21.569 INFO:tasks.rgw_multisite_tests:hBotaJrYa9FhFEdFPCLG/A== 2023-05-25T03:36:21.569 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-25T03:36:21.570 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:14:40 GMT 2023-05-25T03:36:21.570 INFO:tasks.rgw_multisite_tests:/qskrps-33/key 2023-05-25T03:36:21.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.570 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:YKN/kBi6tQbCz3pdD8cr/7k83QA= 2023-05-25T03:36:21.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:14:40 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:YKN/kBi6tQbCz3pdD8cr/7k83QA='} 2023-05-25T03:36:21.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000338d6f1bc6be2a45-00646e7e40-110d-a1'), ('Date', 'Wed, 24 May 2023 21:14:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-34/ 2023-05-25T03:36:21.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-34/ 2023-05-25T03:36:21.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-25T03:36:21.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-34/ 2023-05-25T03:36:21.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8001 2023-05-25T03:36:21.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-25T03:36:21.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.574 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-25T03:36:21.574 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.574 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.574 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:14:40 GMT 2023-05-25T03:36:21.575 INFO:tasks.rgw_multisite_tests:/qskrps-34/ 2023-05-25T03:36:21.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.575 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:LvhaWO3P3WLldUtStskMQ7gcevw= 2023-05-25T03:36:21.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:14:40 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:LvhaWO3P3WLldUtStskMQ7gcevw=', 'Content-Length': '0'} 2023-05-25T03:36:21.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005038a4da2cbc8de5-00646e7e40-1111-a2'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:14:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-34/key 2023-05-25T03:36:21.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-34/key 2023-05-25T03:36:21.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-25T03:36:21.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-34/key 2023-05-25T03:36:21.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2023-05-25T03:36:21.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8001 2023-05-25T03:36:21.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-25T03:36:21.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.578 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:21.579 INFO:tasks.rgw_multisite_tests:hBotaJrYa9FhFEdFPCLG/A== 2023-05-25T03:36:21.579 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-25T03:36:21.579 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:14:40 GMT 2023-05-25T03:36:21.579 INFO:tasks.rgw_multisite_tests:/qskrps-34/key 2023-05-25T03:36:21.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.580 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:oVEp4OtNG9PYXMdkHiKsicBARM8= 2023-05-25T03:36:21.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:14:40 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:oVEp4OtNG9PYXMdkHiKsicBARM8='} 2023-05-25T03:36:21.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c03ceaca3dc49210-00646e7e40-1111-a2'), ('Date', 'Wed, 24 May 2023 21:14:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.580 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.581 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-25T03:36:21.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-25T03:36:21.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-25T03:36:21.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: No --period given, using current period=5c04936b-70ea-4415-b4eb-3aad3dc612e5 2023-05-25T03:36:21.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: [ 2023-05-25T03:36:21.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "1_1684962880.273629_269.1", 2023-05-25T03:36:21.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "last_update": "2023-05-24T21:14:40.273629Z" 2023-05-25T03:36:21.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "1_1684962392.333844_91.1", 2023-05-25T03:36:21.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "last_update": "2023-05-24T21:06:32.333844Z" 2023-05-25T03:36:21.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "", 2023-05-25T03:36:21.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "last_update": "0.000000" 2023-05-25T03:36:21.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "", 2023-05-25T03:36:21.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "last_update": "0.000000" 2023-05-25T03:36:21.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: ] 2023-05-25T03:36:21.586 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1684962880.273629_269.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-25T03:36:21.586 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-05-25T03:36:21.587 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.587 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-25T03:36:21.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-25T03:36:21.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-25T03:36:21.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:14:40.548+0000 7f125cfe4780 20 RGW-SYNC:meta: read sync status 2023-05-25T03:36:21.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:14:40.549+0000 7f125cfe4780 20 RGW-SYNC:meta: read sync status 2023-05-25T03:36:21.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "sync_status": { 2023-05-25T03:36:21.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "info": { 2023-05-25T03:36:21.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "status": "sync", 2023-05-25T03:36:21.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "num_shards": 4, 2023-05-25T03:36:21.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-25T03:36:21.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "realm_epoch": 2 2023-05-25T03:36:21.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "markers": [ 2023-05-25T03:36:21.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 0, 2023-05-25T03:36:21.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "state": 1, 2023-05-25T03:36:21.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "1_1684962880.243018_262.1", 2023-05-25T03:36:21.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 1, 2023-05-25T03:36:21.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "2023-05-24T21:14:40.243018Z", 2023-05-25T03:36:21.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "realm_epoch": 2 2023-05-25T03:36:21.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 1, 2023-05-25T03:36:21.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "state": 1, 2023-05-25T03:36:21.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "1_1684962392.333844_91.1", 2023-05-25T03:36:21.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-25T03:36:21.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "realm_epoch": 2 2023-05-25T03:36:21.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 2, 2023-05-25T03:36:21.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "state": 1, 2023-05-25T03:36:21.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "", 2023-05-25T03:36:21.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "0.000000", 2023-05-25T03:36:21.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "realm_epoch": 2 2023-05-25T03:36:21.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 3, 2023-05-25T03:36:21.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "state": 1, 2023-05-25T03:36:21.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "", 2023-05-25T03:36:21.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "0.000000", 2023-05-25T03:36:21.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "realm_epoch": 2 2023-05-25T03:36:21.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: ] 2023-05-25T03:36:21.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "full_sync": { 2023-05-25T03:36:21.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total": 1, 2023-05-25T03:36:21.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "complete": 1 2023-05-25T03:36:21.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "current_time": "2023-05-24T21:14:40Z" 2023-05-25T03:36:21.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.604 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-25T03:36:21.604 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-25T03:36:21.604 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-25T03:36:21.604 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-25T03:36:21.605 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-25T03:36:21.605 INFO:tasks.rgw_multisite_tests: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-25T03:36:21.605 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-25T03:36:21.605 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.606 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-25T03:36:21.606 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.606 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-25T03:36:21.606 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.606 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-25T03:36:21.607 INFO:tasks.rgw_multisite_tests: "marker": "1_1684962880.243018_262.1", 2023-05-25T03:36:21.607 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.607 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-25T03:36:21.607 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.608 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-24T21:14:40.243018Z", 2023-05-25T03:36:21.608 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-25T03:36:21.608 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.608 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.609 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.609 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-25T03:36:21.609 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.609 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-25T03:36:21.610 INFO:tasks.rgw_multisite_tests: "marker": "1_1684962392.333844_91.1", 2023-05-25T03:36:21.610 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.610 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.610 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.610 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-25T03:36:21.611 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-25T03:36:21.611 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.611 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.611 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.612 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-25T03:36:21.612 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.612 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-25T03:36:21.612 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-25T03:36:21.613 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.613 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.613 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.613 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-25T03:36:21.613 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-25T03:36:21.614 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.614 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.614 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.614 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-25T03:36:21.615 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.615 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-25T03:36:21.615 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-25T03:36:21.615 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.616 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.616 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.617 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-25T03:36:21.617 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-25T03:36:21.617 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.618 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.618 INFO:tasks.rgw_multisite_tests: ] 2023-05-25T03:36:21.618 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.618 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-25T03:36:21.619 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-25T03:36:21.619 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-25T03:36:21.619 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.619 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-24T21:14:40Z" 2023-05-25T03:36:21.620 INFO:tasks.rgw_multisite_tests:} 2023-05-25T03:36:21.620 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.620 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962880.243018_262.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:14:40.243018Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-25T03:36:21.620 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1684962880.273629_269.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-25T03:36:21.621 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1684962880.243018_262.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-25T03:36:21.621 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1684962880.273629_269.1 target=1_1684962880.243018_262.1 2023-05-25T03:36:21.621 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.621 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-25T03:36:21.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-25T03:36:21.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-25T03:36:21.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:14:45.684+0000 7f0a0b595780 20 RGW-SYNC:meta: read sync status 2023-05-25T03:36:21.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:14:45.686+0000 7f0a0b595780 20 RGW-SYNC:meta: read sync status 2023-05-25T03:36:21.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "sync_status": { 2023-05-25T03:36:21.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "info": { 2023-05-25T03:36:21.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "status": "sync", 2023-05-25T03:36:21.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "num_shards": 4, 2023-05-25T03:36:21.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-25T03:36:21.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "realm_epoch": 2 2023-05-25T03:36:21.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "markers": [ 2023-05-25T03:36:21.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 0, 2023-05-25T03:36:21.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "state": 1, 2023-05-25T03:36:21.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "1_1684962880.273629_269.1", 2023-05-25T03:36:21.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 1, 2023-05-25T03:36:21.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "2023-05-24T21:14:40.273629Z", 2023-05-25T03:36:21.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "realm_epoch": 2 2023-05-25T03:36:21.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 1, 2023-05-25T03:36:21.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "state": 1, 2023-05-25T03:36:21.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "1_1684962392.333844_91.1", 2023-05-25T03:36:21.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-25T03:36:21.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "realm_epoch": 2 2023-05-25T03:36:21.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 2, 2023-05-25T03:36:21.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "state": 1, 2023-05-25T03:36:21.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "", 2023-05-25T03:36:21.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "0.000000", 2023-05-25T03:36:21.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "realm_epoch": 2 2023-05-25T03:36:21.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 3, 2023-05-25T03:36:21.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "state": 1, 2023-05-25T03:36:21.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "", 2023-05-25T03:36:21.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "0.000000", 2023-05-25T03:36:21.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "realm_epoch": 2 2023-05-25T03:36:21.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: ] 2023-05-25T03:36:21.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "full_sync": { 2023-05-25T03:36:21.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total": 1, 2023-05-25T03:36:21.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "complete": 1 2023-05-25T03:36:21.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "current_time": "2023-05-24T21:14:45Z" 2023-05-25T03:36:21.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.638 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-25T03:36:21.638 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-25T03:36:21.638 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-25T03:36:21.639 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-25T03:36:21.639 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-25T03:36:21.639 INFO:tasks.rgw_multisite_tests: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-25T03:36:21.639 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-25T03:36:21.640 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.640 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-25T03:36:21.640 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.640 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-25T03:36:21.641 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.641 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-25T03:36:21.641 INFO:tasks.rgw_multisite_tests: "marker": "1_1684962880.273629_269.1", 2023-05-25T03:36:21.641 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.642 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-25T03:36:21.642 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.642 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-24T21:14:40.273629Z", 2023-05-25T03:36:21.642 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-25T03:36:21.643 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.643 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.643 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.643 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-25T03:36:21.643 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.644 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-25T03:36:21.644 INFO:tasks.rgw_multisite_tests: "marker": "1_1684962392.333844_91.1", 2023-05-25T03:36:21.644 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.644 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.645 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.645 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-25T03:36:21.645 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-25T03:36:21.645 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.646 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.646 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.646 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-25T03:36:21.646 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.646 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-25T03:36:21.647 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-25T03:36:21.647 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.647 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.647 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.648 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-25T03:36:21.648 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-25T03:36:21.648 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.648 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.649 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.649 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-25T03:36:21.649 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.649 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-25T03:36:21.649 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-25T03:36:21.650 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.650 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.650 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.650 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-25T03:36:21.651 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-25T03:36:21.651 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.651 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.651 INFO:tasks.rgw_multisite_tests: ] 2023-05-25T03:36:21.652 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.652 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-25T03:36:21.652 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-25T03:36:21.652 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-25T03:36:21.653 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.653 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-24T21:14:45Z" 2023-05-25T03:36:21.653 INFO:tasks.rgw_multisite_tests:} 2023-05-25T03:36:21.653 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.653 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684962880.273629_269.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:14:40.273629Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-25T03:36:21.654 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1684962880.273629_269.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-25T03:36:21.654 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1684962880.273629_269.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-25T03:36:21.654 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-05-25T03:36:21.654 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: data checkpoint: source=a1 target=a2 2023-05-25T03:36:21.655 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.655 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-25T03:36:21.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-25T03:36:21.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-25T03:36:21.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: [ 2023-05-25T03:36:21.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "00000000000000000000:00000000000000002627", 2023-05-25T03:36:21.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "last_update": "2023-05-24T21:10:45.893198Z" 2023-05-25T03:36:21.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "00000000000000000000:00000000000000001068", 2023-05-25T03:36:21.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "last_update": "2023-05-24T21:14:40.065760Z" 2023-05-25T03:36:21.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "00000000000000000000:00000000000000000651", 2023-05-25T03:36:21.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "last_update": "2023-05-24T21:06:31.177575Z" 2023-05-25T03:36:21.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "00000000000000000000:00000000000000004452", 2023-05-25T03:36:21.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "last_update": "2023-05-24T21:14:40.318777Z" 2023-05-25T03:36:21.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: ] 2023-05-25T03:36:21.660 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: data markers for zone=a1 markers={0: '00000000000000000000:00000000000000002627', 1: '00000000000000000000:00000000000000001068', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004452'} 2023-05-25T03:36:21.660 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting data checkpoint for target_zone=a2 source_zone=a1 2023-05-25T03:36:21.661 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.661 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-25T03:36:21.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-25T03:36:21.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-25T03:36:21.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "sync_status": { 2023-05-25T03:36:21.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "info": { 2023-05-25T03:36:21.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "status": "sync", 2023-05-25T03:36:21.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "num_shards": 4, 2023-05-25T03:36:21.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "instance_id": 13190874283549403954 2023-05-25T03:36:21.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "markers": [ 2023-05-25T03:36:21.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 0, 2023-05-25T03:36:21.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "status": "incremental-sync", 2023-05-25T03:36:21.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "00000000000000000000:00000000000000002627", 2023-05-25T03:36:21.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "2023-05-24T21:10:45.893198Z" 2023-05-25T03:36:21.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 1, 2023-05-25T03:36:21.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "status": "incremental-sync", 2023-05-25T03:36:21.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "00000000000000000000:00000000000000000929", 2023-05-25T03:36:21.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "2023-05-24T21:03:11.196517Z" 2023-05-25T03:36:21.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 2, 2023-05-25T03:36:21.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "status": "incremental-sync", 2023-05-25T03:36:21.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "00000000000000000000:00000000000000000651", 2023-05-25T03:36:21.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "2023-05-24T21:06:31.177575Z" 2023-05-25T03:36:21.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 3, 2023-05-25T03:36:21.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "status": "incremental-sync", 2023-05-25T03:36:21.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "00000000000000000000:00000000000000004311", 2023-05-25T03:36:21.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "2023-05-24T21:14:25.907870Z" 2023-05-25T03:36:21.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: ] 2023-05-25T03:36:21.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "full_sync": { 2023-05-25T03:36:21.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total": 0, 2023-05-25T03:36:21.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "complete": 0 2023-05-25T03:36:21.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "current_time": "2023-05-24T21:14:45Z" 2023-05-25T03:36:21.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.676 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-25T03:36:21.676 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-25T03:36:21.677 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-25T03:36:21.677 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-25T03:36:21.677 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-25T03:36:21.677 INFO:tasks.rgw_multisite_tests: "instance_id": 13190874283549403954 2023-05-25T03:36:21.678 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.678 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-25T03:36:21.678 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.678 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-25T03:36:21.678 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.679 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-25T03:36:21.679 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002627", 2023-05-25T03:36:21.679 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.679 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.680 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.680 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-24T21:10:45.893198Z" 2023-05-25T03:36:21.680 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.680 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.681 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.681 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-25T03:36:21.681 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.681 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-25T03:36:21.681 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000000929", 2023-05-25T03:36:21.682 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.682 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.682 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.682 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-24T21:03:11.196517Z" 2023-05-25T03:36:21.683 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.683 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.683 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.683 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-25T03:36:21.684 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.684 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-25T03:36:21.684 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000000651", 2023-05-25T03:36:21.684 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.684 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.685 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.685 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-24T21:06:31.177575Z" 2023-05-25T03:36:21.685 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.685 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.686 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.686 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-25T03:36:21.686 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.686 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-25T03:36:21.687 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004311", 2023-05-25T03:36:21.687 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.687 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.687 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.687 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-24T21:14:25.907870Z" 2023-05-25T03:36:21.688 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.688 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.688 INFO:tasks.rgw_multisite_tests: ] 2023-05-25T03:36:21.689 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.689 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-25T03:36:21.689 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-05-25T03:36:21.689 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-05-25T03:36:21.689 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.690 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-24T21:14:45Z" 2023-05-25T03:36:21.690 INFO:tasks.rgw_multisite_tests:} 2023-05-25T03:36:21.690 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.690 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002627', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:10:45.893198Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000929', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:03:11.196517Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:31.177575Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004311', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:14:25.907870Z'}}] 2023-05-25T03:36:21.691 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002627', 1: '00000000000000000000:00000000000000001068', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004452'} 2023-05-25T03:36:21.691 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002627', 1: '00000000000000000000:00000000000000000929', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004311'} 2023-05-25T03:36:21.691 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000001068 target=00000000000000000000:00000000000000000929, shard=3 master=00000000000000000000:00000000000000004452 target=00000000000000000000:00000000000000004311 2023-05-25T03:36:21.691 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.692 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-25T03:36:21.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-25T03:36:21.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-25T03:36:21.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "sync_status": { 2023-05-25T03:36:21.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "info": { 2023-05-25T03:36:21.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "status": "sync", 2023-05-25T03:36:21.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "num_shards": 4, 2023-05-25T03:36:21.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "instance_id": 13190874283549403954 2023-05-25T03:36:21.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "markers": [ 2023-05-25T03:36:21.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 0, 2023-05-25T03:36:21.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "status": "incremental-sync", 2023-05-25T03:36:21.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "00000000000000000000:00000000000000002627", 2023-05-25T03:36:21.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "2023-05-24T21:10:45.893198Z" 2023-05-25T03:36:21.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 1, 2023-05-25T03:36:21.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "status": "incremental-sync", 2023-05-25T03:36:21.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "00000000000000000000:00000000000000000929", 2023-05-25T03:36:21.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "2023-05-24T21:03:11.196517Z" 2023-05-25T03:36:21.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 2, 2023-05-25T03:36:21.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "status": "incremental-sync", 2023-05-25T03:36:21.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "00000000000000000000:00000000000000000651", 2023-05-25T03:36:21.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "2023-05-24T21:06:31.177575Z" 2023-05-25T03:36:21.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 3, 2023-05-25T03:36:21.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "status": "incremental-sync", 2023-05-25T03:36:21.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "00000000000000000000:00000000000000004311", 2023-05-25T03:36:21.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "2023-05-24T21:14:25.907870Z" 2023-05-25T03:36:21.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: ] 2023-05-25T03:36:21.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "full_sync": { 2023-05-25T03:36:21.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total": 0, 2023-05-25T03:36:21.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "complete": 0 2023-05-25T03:36:21.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "current_time": "2023-05-24T21:14:51Z" 2023-05-25T03:36:21.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.707 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-25T03:36:21.707 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-25T03:36:21.707 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-25T03:36:21.708 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-25T03:36:21.708 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-25T03:36:21.708 INFO:tasks.rgw_multisite_tests: "instance_id": 13190874283549403954 2023-05-25T03:36:21.708 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.709 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-25T03:36:21.709 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.709 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-25T03:36:21.709 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.710 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-25T03:36:21.710 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002627", 2023-05-25T03:36:21.710 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.710 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.711 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.711 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-24T21:10:45.893198Z" 2023-05-25T03:36:21.711 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.711 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.712 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.712 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-25T03:36:21.712 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.712 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-25T03:36:21.712 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000000929", 2023-05-25T03:36:21.713 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.713 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.713 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.713 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-24T21:03:11.196517Z" 2023-05-25T03:36:21.714 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.714 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.714 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.714 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-25T03:36:21.715 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.715 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-25T03:36:21.715 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000000651", 2023-05-25T03:36:21.715 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.716 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.716 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.716 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-24T21:06:31.177575Z" 2023-05-25T03:36:21.716 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.716 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.717 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.717 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-25T03:36:21.717 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.717 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-25T03:36:21.718 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004311", 2023-05-25T03:36:21.718 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.718 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.718 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.719 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-24T21:14:25.907870Z" 2023-05-25T03:36:21.719 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.719 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.719 INFO:tasks.rgw_multisite_tests: ] 2023-05-25T03:36:21.720 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.720 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-25T03:36:21.720 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-05-25T03:36:21.720 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-05-25T03:36:21.721 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.721 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-24T21:14:51Z" 2023-05-25T03:36:21.721 INFO:tasks.rgw_multisite_tests:} 2023-05-25T03:36:21.721 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.722 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002627', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:10:45.893198Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000929', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:03:11.196517Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:31.177575Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004311', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:14:25.907870Z'}}] 2023-05-25T03:36:21.722 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002627', 1: '00000000000000000000:00000000000000001068', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004452'} 2023-05-25T03:36:21.722 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002627', 1: '00000000000000000000:00000000000000000929', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004311'} 2023-05-25T03:36:21.722 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000001068 target=00000000000000000000:00000000000000000929, shard=3 master=00000000000000000000:00000000000000004452 target=00000000000000000000:00000000000000004311 2023-05-25T03:36:21.722 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.723 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-25T03:36:21.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-25T03:36:21.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-25T03:36:21.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "sync_status": { 2023-05-25T03:36:21.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "info": { 2023-05-25T03:36:21.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "status": "sync", 2023-05-25T03:36:21.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "num_shards": 4, 2023-05-25T03:36:21.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "instance_id": 13190874283549403954 2023-05-25T03:36:21.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "markers": [ 2023-05-25T03:36:21.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 0, 2023-05-25T03:36:21.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "status": "incremental-sync", 2023-05-25T03:36:21.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "00000000000000000000:00000000000000002627", 2023-05-25T03:36:21.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "2023-05-24T21:10:45.893198Z" 2023-05-25T03:36:21.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 1, 2023-05-25T03:36:21.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "status": "incremental-sync", 2023-05-25T03:36:21.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "00000000000000000000:00000000000000001068", 2023-05-25T03:36:21.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "2023-05-24T21:14:40.065760Z" 2023-05-25T03:36:21.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 2, 2023-05-25T03:36:21.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "status": "incremental-sync", 2023-05-25T03:36:21.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "00000000000000000000:00000000000000000651", 2023-05-25T03:36:21.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "2023-05-24T21:06:31.177575Z" 2023-05-25T03:36:21.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 3, 2023-05-25T03:36:21.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "status": "incremental-sync", 2023-05-25T03:36:21.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "00000000000000000000:00000000000000004452", 2023-05-25T03:36:21.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "2023-05-24T21:14:40.318777Z" 2023-05-25T03:36:21.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: ] 2023-05-25T03:36:21.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "full_sync": { 2023-05-25T03:36:21.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total": 0, 2023-05-25T03:36:21.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "complete": 0 2023-05-25T03:36:21.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "current_time": "2023-05-24T21:14:56Z" 2023-05-25T03:36:21.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.738 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-25T03:36:21.738 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-25T03:36:21.738 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-25T03:36:21.739 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-25T03:36:21.739 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-25T03:36:21.739 INFO:tasks.rgw_multisite_tests: "instance_id": 13190874283549403954 2023-05-25T03:36:21.739 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.740 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-25T03:36:21.740 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.740 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-25T03:36:21.740 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.741 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-25T03:36:21.741 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002627", 2023-05-25T03:36:21.741 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.741 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.741 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.742 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-24T21:10:45.893198Z" 2023-05-25T03:36:21.742 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.742 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.742 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.743 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-25T03:36:21.743 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.743 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-25T03:36:21.743 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000001068", 2023-05-25T03:36:21.744 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.744 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.744 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.744 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-24T21:14:40.065760Z" 2023-05-25T03:36:21.745 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.745 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.745 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.745 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-25T03:36:21.745 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.746 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-25T03:36:21.746 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000000651", 2023-05-25T03:36:21.746 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.746 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.747 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.747 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-24T21:06:31.177575Z" 2023-05-25T03:36:21.747 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.747 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.748 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.748 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-25T03:36:21.748 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.748 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-25T03:36:21.749 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004452", 2023-05-25T03:36:21.749 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.749 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.749 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.749 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-24T21:14:40.318777Z" 2023-05-25T03:36:21.750 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.750 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.750 INFO:tasks.rgw_multisite_tests: ] 2023-05-25T03:36:21.750 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.751 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-25T03:36:21.751 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-05-25T03:36:21.751 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-05-25T03:36:21.751 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.752 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-24T21:14:56Z" 2023-05-25T03:36:21.752 INFO:tasks.rgw_multisite_tests:} 2023-05-25T03:36:21.752 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.752 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002627', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:10:45.893198Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001068', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:14:40.065760Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:31.177575Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004452', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:14:40.318777Z'}}] 2023-05-25T03:36:21.753 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002627', 1: '00000000000000000000:00000000000000001068', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004452'} 2023-05-25T03:36:21.753 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002627', 1: '00000000000000000000:00000000000000001068', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004452'} 2023-05-25T03:36:21.753 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finished data checkpoint for target_zone=a2 source_zone=a1 2023-05-25T03:36:21.753 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: data checkpoint: source=a2 target=a1 2023-05-25T03:36:21.753 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.754 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-25T03:36:21.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-25T03:36:21.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-25T03:36:21.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: [ 2023-05-25T03:36:21.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "00000000000000000000:00000000000000002345", 2023-05-25T03:36:21.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "last_update": "2023-05-24T21:10:23.920015Z" 2023-05-25T03:36:21.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "00000000000000000000:00000000000000001210", 2023-05-25T03:36:21.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "last_update": "2023-05-24T21:14:51.309908Z" 2023-05-25T03:36:21.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "00000000000000000000:00000000000000000651", 2023-05-25T03:36:21.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "last_update": "2023-05-24T21:03:11.225401Z" 2023-05-25T03:36:21.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "00000000000000000000:00000000000000004593", 2023-05-25T03:36:21.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "last_update": "2023-05-24T21:14:40.343934Z" 2023-05-25T03:36:21.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: ] 2023-05-25T03:36:21.759 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: data markers for zone=a2 markers={0: '00000000000000000000:00000000000000002345', 1: '00000000000000000000:00000000000000001210', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004593'} 2023-05-25T03:36:21.759 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting data checkpoint for target_zone=a1 source_zone=a2 2023-05-25T03:36:21.759 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.760 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-25T03:36:21.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-25T03:36:21.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-25T03:36:21.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "sync_status": { 2023-05-25T03:36:21.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "info": { 2023-05-25T03:36:21.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "status": "sync", 2023-05-25T03:36:21.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "num_shards": 4, 2023-05-25T03:36:21.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "instance_id": 6585121019900467362 2023-05-25T03:36:21.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "markers": [ 2023-05-25T03:36:21.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 0, 2023-05-25T03:36:21.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "status": "incremental-sync", 2023-05-25T03:36:21.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "00000000000000000000:00000000000000002345", 2023-05-25T03:36:21.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "2023-05-24T21:10:23.920015Z" 2023-05-25T03:36:21.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 1, 2023-05-25T03:36:21.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "status": "incremental-sync", 2023-05-25T03:36:21.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "00000000000000000000:00000000000000001068", 2023-05-25T03:36:21.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "2023-05-24T21:14:40.136506Z" 2023-05-25T03:36:21.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 2, 2023-05-25T03:36:21.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "status": "incremental-sync", 2023-05-25T03:36:21.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "00000000000000000000:00000000000000000651", 2023-05-25T03:36:21.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "2023-05-24T21:03:11.225401Z" 2023-05-25T03:36:21.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 3, 2023-05-25T03:36:21.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "status": "incremental-sync", 2023-05-25T03:36:21.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "00000000000000000000:00000000000000004593", 2023-05-25T03:36:21.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "2023-05-24T21:14:40.343934Z" 2023-05-25T03:36:21.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: ] 2023-05-25T03:36:21.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "full_sync": { 2023-05-25T03:36:21.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total": 0, 2023-05-25T03:36:21.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "complete": 0 2023-05-25T03:36:21.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "current_time": "2023-05-24T21:14:56Z" 2023-05-25T03:36:21.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.775 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-25T03:36:21.775 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-25T03:36:21.775 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-25T03:36:21.775 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-25T03:36:21.775 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-25T03:36:21.776 INFO:tasks.rgw_multisite_tests: "instance_id": 6585121019900467362 2023-05-25T03:36:21.776 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.776 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-25T03:36:21.776 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.777 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-25T03:36:21.777 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.777 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-25T03:36:21.777 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002345", 2023-05-25T03:36:21.778 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.778 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.778 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.778 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-24T21:10:23.920015Z" 2023-05-25T03:36:21.779 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.779 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.779 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.779 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-25T03:36:21.779 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.780 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-25T03:36:21.780 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000001068", 2023-05-25T03:36:21.780 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.780 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.781 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.781 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-24T21:14:40.136506Z" 2023-05-25T03:36:21.781 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.781 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.782 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.782 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-25T03:36:21.782 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.782 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-25T03:36:21.782 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000000651", 2023-05-25T03:36:21.783 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.783 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.783 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.783 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-24T21:03:11.225401Z" 2023-05-25T03:36:21.784 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.784 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.784 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.784 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-25T03:36:21.785 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.785 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-25T03:36:21.785 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004593", 2023-05-25T03:36:21.785 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.786 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.786 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.786 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-24T21:14:40.343934Z" 2023-05-25T03:36:21.786 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.786 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.787 INFO:tasks.rgw_multisite_tests: ] 2023-05-25T03:36:21.787 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.787 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-25T03:36:21.787 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-05-25T03:36:21.788 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-05-25T03:36:21.788 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.788 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-24T21:14:56Z" 2023-05-25T03:36:21.788 INFO:tasks.rgw_multisite_tests:} 2023-05-25T03:36:21.789 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.789 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002345', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:10:23.920015Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001068', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:14:40.136506Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:03:11.225401Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004593', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:14:40.343934Z'}}] 2023-05-25T03:36:21.789 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002345', 1: '00000000000000000000:00000000000000001210', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004593'} 2023-05-25T03:36:21.789 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002345', 1: '00000000000000000000:00000000000000001068', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004593'} 2023-05-25T03:36:21.790 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a1 behind zone a2: shard=1 master=00000000000000000000:00000000000000001210 target=00000000000000000000:00000000000000001068 2023-05-25T03:36:21.790 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.790 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-25T03:36:21.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-25T03:36:21.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-25T03:36:21.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "sync_status": { 2023-05-25T03:36:21.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "info": { 2023-05-25T03:36:21.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "status": "sync", 2023-05-25T03:36:21.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "num_shards": 4, 2023-05-25T03:36:21.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "instance_id": 6585121019900467362 2023-05-25T03:36:21.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "markers": [ 2023-05-25T03:36:21.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 0, 2023-05-25T03:36:21.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "status": "incremental-sync", 2023-05-25T03:36:21.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "00000000000000000000:00000000000000002345", 2023-05-25T03:36:21.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "2023-05-24T21:10:23.920015Z" 2023-05-25T03:36:21.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 1, 2023-05-25T03:36:21.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "status": "incremental-sync", 2023-05-25T03:36:21.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "00000000000000000000:00000000000000001068", 2023-05-25T03:36:21.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "2023-05-24T21:14:40.136506Z" 2023-05-25T03:36:21.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 2, 2023-05-25T03:36:21.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "status": "incremental-sync", 2023-05-25T03:36:21.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "00000000000000000000:00000000000000000651", 2023-05-25T03:36:21.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "2023-05-24T21:03:11.225401Z" 2023-05-25T03:36:21.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 3, 2023-05-25T03:36:21.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "status": "incremental-sync", 2023-05-25T03:36:21.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "00000000000000000000:00000000000000004593", 2023-05-25T03:36:21.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "2023-05-24T21:14:40.343934Z" 2023-05-25T03:36:21.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: ] 2023-05-25T03:36:21.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "full_sync": { 2023-05-25T03:36:21.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total": 0, 2023-05-25T03:36:21.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "complete": 0 2023-05-25T03:36:21.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "current_time": "2023-05-24T21:15:01Z" 2023-05-25T03:36:21.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.805 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-25T03:36:21.805 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-25T03:36:21.806 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-25T03:36:21.806 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-25T03:36:21.806 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-25T03:36:21.806 INFO:tasks.rgw_multisite_tests: "instance_id": 6585121019900467362 2023-05-25T03:36:21.806 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.807 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-25T03:36:21.807 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.807 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-25T03:36:21.807 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.808 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-25T03:36:21.808 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002345", 2023-05-25T03:36:21.808 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.808 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.809 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.809 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-24T21:10:23.920015Z" 2023-05-25T03:36:21.809 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.809 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.810 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.810 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-25T03:36:21.810 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.811 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-25T03:36:21.811 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000001068", 2023-05-25T03:36:21.811 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.811 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.811 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.812 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-24T21:14:40.136506Z" 2023-05-25T03:36:21.812 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.812 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.812 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.813 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-25T03:36:21.813 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.813 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-25T03:36:21.813 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000000651", 2023-05-25T03:36:21.814 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.814 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.814 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.814 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-24T21:03:11.225401Z" 2023-05-25T03:36:21.815 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.815 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.815 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.815 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-25T03:36:21.815 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.816 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-25T03:36:21.816 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004593", 2023-05-25T03:36:21.816 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.816 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.817 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.817 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-24T21:14:40.343934Z" 2023-05-25T03:36:21.817 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.817 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.818 INFO:tasks.rgw_multisite_tests: ] 2023-05-25T03:36:21.818 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.818 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-25T03:36:21.818 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-05-25T03:36:21.819 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-05-25T03:36:21.819 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.819 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-24T21:15:01Z" 2023-05-25T03:36:21.819 INFO:tasks.rgw_multisite_tests:} 2023-05-25T03:36:21.819 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.820 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002345', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:10:23.920015Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001068', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:14:40.136506Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:03:11.225401Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004593', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:14:40.343934Z'}}] 2023-05-25T03:36:21.820 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002345', 1: '00000000000000000000:00000000000000001210', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004593'} 2023-05-25T03:36:21.820 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002345', 1: '00000000000000000000:00000000000000001068', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004593'} 2023-05-25T03:36:21.820 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a1 behind zone a2: shard=1 master=00000000000000000000:00000000000000001210 target=00000000000000000000:00000000000000001068 2023-05-25T03:36:21.821 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.821 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-25T03:36:21.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-25T03:36:21.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-25T03:36:21.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "sync_status": { 2023-05-25T03:36:21.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "info": { 2023-05-25T03:36:21.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "status": "sync", 2023-05-25T03:36:21.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "num_shards": 4, 2023-05-25T03:36:21.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "instance_id": 6585121019900467362 2023-05-25T03:36:21.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "markers": [ 2023-05-25T03:36:21.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 0, 2023-05-25T03:36:21.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "status": "incremental-sync", 2023-05-25T03:36:21.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "00000000000000000000:00000000000000002345", 2023-05-25T03:36:21.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "2023-05-24T21:10:23.920015Z" 2023-05-25T03:36:21.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 1, 2023-05-25T03:36:21.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "status": "incremental-sync", 2023-05-25T03:36:21.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "00000000000000000000:00000000000000001068", 2023-05-25T03:36:21.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "2023-05-24T21:14:40.136506Z" 2023-05-25T03:36:21.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 2, 2023-05-25T03:36:21.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "status": "incremental-sync", 2023-05-25T03:36:21.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "00000000000000000000:00000000000000000651", 2023-05-25T03:36:21.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "2023-05-24T21:03:11.225401Z" 2023-05-25T03:36:21.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 3, 2023-05-25T03:36:21.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "status": "incremental-sync", 2023-05-25T03:36:21.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "00000000000000000000:00000000000000004593", 2023-05-25T03:36:21.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "2023-05-24T21:14:40.343934Z" 2023-05-25T03:36:21.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: ] 2023-05-25T03:36:21.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "full_sync": { 2023-05-25T03:36:21.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total": 0, 2023-05-25T03:36:21.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "complete": 0 2023-05-25T03:36:21.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "current_time": "2023-05-24T21:15:06Z" 2023-05-25T03:36:21.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.836 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-25T03:36:21.837 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-25T03:36:21.837 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-25T03:36:21.837 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-25T03:36:21.837 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-25T03:36:21.838 INFO:tasks.rgw_multisite_tests: "instance_id": 6585121019900467362 2023-05-25T03:36:21.838 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.838 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-25T03:36:21.838 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.839 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-25T03:36:21.839 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.839 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-25T03:36:21.839 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002345", 2023-05-25T03:36:21.840 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.840 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.840 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.840 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-24T21:10:23.920015Z" 2023-05-25T03:36:21.841 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.841 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.841 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.841 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-25T03:36:21.842 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.842 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-25T03:36:21.842 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000001068", 2023-05-25T03:36:21.842 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.843 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.843 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.843 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-24T21:14:40.136506Z" 2023-05-25T03:36:21.843 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.844 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.844 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.844 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-25T03:36:21.844 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.845 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-25T03:36:21.845 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000000651", 2023-05-25T03:36:21.845 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.845 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.846 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.846 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-24T21:03:11.225401Z" 2023-05-25T03:36:21.846 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.846 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.847 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.847 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-25T03:36:21.847 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.847 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-25T03:36:21.848 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004593", 2023-05-25T03:36:21.848 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.848 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.849 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.849 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-24T21:14:40.343934Z" 2023-05-25T03:36:21.849 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.849 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.850 INFO:tasks.rgw_multisite_tests: ] 2023-05-25T03:36:21.850 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.850 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-25T03:36:21.850 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-05-25T03:36:21.850 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-05-25T03:36:21.851 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.851 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-24T21:15:06Z" 2023-05-25T03:36:21.851 INFO:tasks.rgw_multisite_tests:} 2023-05-25T03:36:21.851 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.852 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002345', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:10:23.920015Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001068', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:14:40.136506Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:03:11.225401Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004593', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:14:40.343934Z'}}] 2023-05-25T03:36:21.852 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002345', 1: '00000000000000000000:00000000000000001210', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004593'} 2023-05-25T03:36:21.852 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002345', 1: '00000000000000000000:00000000000000001068', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004593'} 2023-05-25T03:36:21.852 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a1 behind zone a2: shard=1 master=00000000000000000000:00000000000000001210 target=00000000000000000000:00000000000000001068 2023-05-25T03:36:21.853 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.853 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-25T03:36:21.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-25T03:36:21.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-25T03:36:21.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "sync_status": { 2023-05-25T03:36:21.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "info": { 2023-05-25T03:36:21.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "status": "sync", 2023-05-25T03:36:21.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "num_shards": 4, 2023-05-25T03:36:21.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "instance_id": 6585121019900467362 2023-05-25T03:36:21.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "markers": [ 2023-05-25T03:36:21.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 0, 2023-05-25T03:36:21.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "status": "incremental-sync", 2023-05-25T03:36:21.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "00000000000000000000:00000000000000002345", 2023-05-25T03:36:21.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "2023-05-24T21:10:23.920015Z" 2023-05-25T03:36:21.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 1, 2023-05-25T03:36:21.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "status": "incremental-sync", 2023-05-25T03:36:21.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "00000000000000000000:00000000000000001210", 2023-05-25T03:36:21.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "2023-05-24T21:14:51.309908Z" 2023-05-25T03:36:21.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 2, 2023-05-25T03:36:21.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "status": "incremental-sync", 2023-05-25T03:36:21.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "00000000000000000000:00000000000000000651", 2023-05-25T03:36:21.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "2023-05-24T21:03:11.225401Z" 2023-05-25T03:36:21.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 3, 2023-05-25T03:36:21.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "status": "incremental-sync", 2023-05-25T03:36:21.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "00000000000000000000:00000000000000004593", 2023-05-25T03:36:21.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "2023-05-24T21:14:40.343934Z" 2023-05-25T03:36:21.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: ] 2023-05-25T03:36:21.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "full_sync": { 2023-05-25T03:36:21.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total": 0, 2023-05-25T03:36:21.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "complete": 0 2023-05-25T03:36:21.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "current_time": "2023-05-24T21:15:11Z" 2023-05-25T03:36:21.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.868 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-25T03:36:21.869 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-25T03:36:21.869 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-25T03:36:21.869 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-25T03:36:21.869 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-25T03:36:21.870 INFO:tasks.rgw_multisite_tests: "instance_id": 6585121019900467362 2023-05-25T03:36:21.870 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.870 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-25T03:36:21.870 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.871 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-25T03:36:21.871 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.871 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-25T03:36:21.871 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002345", 2023-05-25T03:36:21.871 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.872 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.872 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.872 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-24T21:10:23.920015Z" 2023-05-25T03:36:21.872 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.873 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.873 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.873 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-25T03:36:21.873 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.874 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-25T03:36:21.874 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000001210", 2023-05-25T03:36:21.874 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.874 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.875 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.875 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-24T21:14:51.309908Z" 2023-05-25T03:36:21.875 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.875 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.875 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.876 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-25T03:36:21.876 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.876 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-25T03:36:21.876 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000000651", 2023-05-25T03:36:21.877 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.877 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.877 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.877 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-24T21:03:11.225401Z" 2023-05-25T03:36:21.878 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.878 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.878 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.878 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-25T03:36:21.879 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.879 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-25T03:36:21.879 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004593", 2023-05-25T03:36:21.879 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.880 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.880 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.880 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-24T21:14:40.343934Z" 2023-05-25T03:36:21.880 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.881 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.881 INFO:tasks.rgw_multisite_tests: ] 2023-05-25T03:36:21.881 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.881 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-25T03:36:21.881 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-05-25T03:36:21.882 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-05-25T03:36:21.882 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.882 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-24T21:15:11Z" 2023-05-25T03:36:21.882 INFO:tasks.rgw_multisite_tests:} 2023-05-25T03:36:21.883 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.883 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002345', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:10:23.920015Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:14:51.309908Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:03:11.225401Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004593', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:14:40.343934Z'}}] 2023-05-25T03:36:21.883 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002345', 1: '00000000000000000000:00000000000000001210', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004593'} 2023-05-25T03:36:21.883 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002345', 1: '00000000000000000000:00000000000000001210', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004593'} 2023-05-25T03:36:21.884 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finished data checkpoint for target_zone=a1 source_zone=a2 2023-05-25T03:36:21.884 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.884 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-25T03:36:21.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-25T03:36:21.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-25T03:36:21.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: [ 2023-05-25T03:36:21.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "00000000000000000000:00000000000000002627", 2023-05-25T03:36:21.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "last_update": "2023-05-24T21:10:45.893198Z" 2023-05-25T03:36:21.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "00000000000000000000:00000000000000001210", 2023-05-25T03:36:21.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "last_update": "2023-05-24T21:14:51.314836Z" 2023-05-25T03:36:21.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "00000000000000000000:00000000000000000651", 2023-05-25T03:36:21.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "last_update": "2023-05-24T21:06:31.177575Z" 2023-05-25T03:36:21.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "00000000000000000000:00000000000000004593", 2023-05-25T03:36:21.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "last_update": "2023-05-24T21:14:52.334363Z" 2023-05-25T03:36:21.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: ] 2023-05-25T03:36:21.889 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-25T03:36:21.890 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-25T03:36:21.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-25T03:36:21.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-25T03:36:21.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-25T03:36:21.891 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.891 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 0 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-25T03:36:21.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-25T03:36:21.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-25T03:36:21.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: [] 2023-05-25T03:36:21.892 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.892 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 1 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-25T03:36:21.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-25T03:36:21.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-25T03:36:21.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: [] 2023-05-25T03:36:21.893 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.894 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 2 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-25T03:36:21.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-25T03:36:21.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-25T03:36:21.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: [] 2023-05-25T03:36:21.895 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.895 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 3 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-25T03:36:21.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-25T03:36:21.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-25T03:36:21.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: [ 2023-05-25T03:36:21.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "entity_type": "bucket", 2023-05-25T03:36:21.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": "qskrps-34:299a3349-fae0-459e-8632-431fc36beb1b.4368.34:8", 2023-05-25T03:36:21.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "2023-05-24T21:14:52.334048Z", 2023-05-25T03:36:21.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "gen": 0 2023-05-25T03:36:21.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: ] 2023-05-25T03:36:21.898 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-05-25T03:36:21.898 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.898 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-05-25T03:36:21.899 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_encrypted_object_sync 2023-05-25T03:36:21.899 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-05-25T03:36:21.899 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-05-25T03:36:21.900 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-05-25T03:36:21.900 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-05-25T03:36:21.900 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/../src/test/rgw/rgw_multi/tests.py", line 1281, in test_encrypted_object_sync 2023-05-25T03:36:21.900 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(zone2.zone, zone1.zone, bucket_name) 2023-05-25T03:36:21.901 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2023-05-25T03:36:21.901 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2023-05-25T03:36:21.901 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/tasks/rgw_multisite.py", line 197, in admin 2023-05-25T03:36:21.901 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-05-25T03:36:21.902 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-05-25T03:36:21.902 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-05-25T03:36:21.902 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=qskrps-52 2023-05-25T03:36:21.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-52/ 2023-05-25T03:36:21.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-52/ 2023-05-25T03:36:21.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-25T03:36:21.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-52/ 2023-05-25T03:36:21.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-25T03:36:21.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.905 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:21.905 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.905 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.906 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:29:02 GMT 2023-05-25T03:36:21.906 INFO:tasks.rgw_multisite_tests:/qskrps-52/ 2023-05-25T03:36:21.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.906 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:S33yJGZOVB/Ai/BHOrSsR5D43Zo= 2023-05-25T03:36:21.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 24 May 2023 21:29:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:S33yJGZOVB/Ai/BHOrSsR5D43Zo=', 'Content-Length': '0'} 2023-05-25T03:36:21.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000ff344861062aa898-00646e819e-1627-a1'), ('Content-Length', '0'), ('Date', 'Wed, 24 May 2023 21:29:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-52/testobj-sse-c 2023-05-25T03:36:21.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-52/testobj-sse-c 2023-05-25T03:36:21.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-25T03:36:21.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-52/testobj-sse-c 2023-05-25T03:36:21.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-05-25T03:36:21.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.910 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:21.910 INFO:tasks.rgw_multisite_tests:3FCGuEcom6i4veFJuDiBdQ== 2023-05-25T03:36:21.910 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-25T03:36:21.911 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:29:02 GMT 2023-05-25T03:36:21.911 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption-customer-algorithm:AES256 2023-05-25T03:36:21.911 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= 2023-05-25T03:36:21.911 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== 2023-05-25T03:36:21.912 INFO:tasks.rgw_multisite_tests:/qskrps-52/testobj-sse-c 2023-05-25T03:36:21.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.912 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:N7YAD17l+Tu23Z3Kwa4hlhObGeI= 2023-05-25T03:36:21.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:29:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:N7YAD17l+Tu23Z3Kwa4hlhObGeI='} 2023-05-25T03:36:21.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('x-amz-request-id', 'tx0000096c7f1ae667a8e25-00646e819e-1627-a1'), ('Date', 'Wed, 24 May 2023 21:29:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/qskrps-52/testobj-sse-kms 2023-05-25T03:36:21.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/qskrps-52/testobj-sse-kms 2023-05-25T03:36:21.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-25T03:36:21.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /qskrps-52/testobj-sse-kms 2023-05-25T03:36:21.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-25T03:36:21.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-05-25T03:36:21.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2023-05-25T03:36:21.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-25T03:36:21.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-25T03:36:21.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-25T03:36:21.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-25T03:36:21.915 INFO:tasks.rgw_multisite_tests:PUT 2023-05-25T03:36:21.916 INFO:tasks.rgw_multisite_tests:3FCGuEcom6i4veFJuDiBdQ== 2023-05-25T03:36:21.916 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-25T03:36:21.916 INFO:tasks.rgw_multisite_tests:Wed, 24 May 2023 21:29:02 GMT 2023-05-25T03:36:21.916 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption:aws:kms 2023-05-25T03:36:21.917 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption-aws-kms-key-id:testkey-1 2023-05-25T03:36:21.917 INFO:tasks.rgw_multisite_tests:/qskrps-52/testobj-sse-kms 2023-05-25T03:36:21.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-25T03:36:21.917 INFO:tasks.rgw_multisite_tests:AWS YAS2LRYUNTM46AEFZJ98:vE2nYt9i/CoIpvRdfKUx6PhXML8= 2023-05-25T03:36:21.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Wed, 24 May 2023 21:29:02 GMT', 'Authorization': 'AWS YAS2LRYUNTM46AEFZJ98:vE2nYt9i/CoIpvRdfKUx6PhXML8='} 2023-05-25T03:36:21.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('x-amz-server-side-encryption-context', 'eyJhd3M6czM6YXJuIjoiYXJuOmF3czpzMzo6cmd3LW11bHRpc2l0ZS10ZXN0LXVzZXI6cXNrcnBzLTUyL3Rlc3RvYmotc3NlLWttcyJ9'), ('x-amz-request-id', 'tx00000372e5bf4ac75afb8-00646e819e-1627-a1'), ('Date', 'Wed, 24 May 2023 21:29:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-25T03:36:21.918 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.918 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-25T03:36:21.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-25T03:36:21.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-25T03:36:21.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: No --period given, using current period=5c04936b-70ea-4415-b4eb-3aad3dc612e5 2023-05-25T03:36:21.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: [ 2023-05-25T03:36:21.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "1_1684963742.659011_575.1", 2023-05-25T03:36:21.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "last_update": "2023-05-24T21:29:02.659011Z" 2023-05-25T03:36:21.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "1_1684962392.333844_91.1", 2023-05-25T03:36:21.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "last_update": "2023-05-24T21:06:32.333844Z" 2023-05-25T03:36:21.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "", 2023-05-25T03:36:21.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "last_update": "0.000000" 2023-05-25T03:36:21.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "", 2023-05-25T03:36:21.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "last_update": "0.000000" 2023-05-25T03:36:21.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: ] 2023-05-25T03:36:21.924 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1684963742.659011_575.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-25T03:36:21.924 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-05-25T03:36:21.924 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.924 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-25T03:36:21.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-25T03:36:21.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-25T03:36:21.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:29:02.864+0000 7fd7bd33d780 20 RGW-SYNC:meta: read sync status 2023-05-25T03:36:21.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:29:02.865+0000 7fd7bd33d780 20 RGW-SYNC:meta: read sync status 2023-05-25T03:36:21.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "sync_status": { 2023-05-25T03:36:21.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "info": { 2023-05-25T03:36:21.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "status": "sync", 2023-05-25T03:36:21.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "num_shards": 4, 2023-05-25T03:36:21.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-25T03:36:21.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "realm_epoch": 2 2023-05-25T03:36:21.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "markers": [ 2023-05-25T03:36:21.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 0, 2023-05-25T03:36:21.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "state": 1, 2023-05-25T03:36:21.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "1_1684963742.651710_572.1", 2023-05-25T03:36:21.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 1, 2023-05-25T03:36:21.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "2023-05-24T21:29:02.651710Z", 2023-05-25T03:36:21.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "realm_epoch": 2 2023-05-25T03:36:21.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 1, 2023-05-25T03:36:21.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "state": 1, 2023-05-25T03:36:21.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "1_1684962392.333844_91.1", 2023-05-25T03:36:21.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-25T03:36:21.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "realm_epoch": 2 2023-05-25T03:36:21.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 2, 2023-05-25T03:36:21.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "state": 1, 2023-05-25T03:36:21.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "", 2023-05-25T03:36:21.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "0.000000", 2023-05-25T03:36:21.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "realm_epoch": 2 2023-05-25T03:36:21.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 3, 2023-05-25T03:36:21.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "state": 1, 2023-05-25T03:36:21.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "", 2023-05-25T03:36:21.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "0.000000", 2023-05-25T03:36:21.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "realm_epoch": 2 2023-05-25T03:36:21.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: ] 2023-05-25T03:36:21.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "full_sync": { 2023-05-25T03:36:21.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total": 1, 2023-05-25T03:36:21.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "complete": 1 2023-05-25T03:36:21.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "current_time": "2023-05-24T21:29:02Z" 2023-05-25T03:36:21.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.941 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-25T03:36:21.941 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-25T03:36:21.941 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-25T03:36:21.941 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-25T03:36:21.942 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-25T03:36:21.942 INFO:tasks.rgw_multisite_tests: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-25T03:36:21.942 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-25T03:36:21.942 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.943 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-25T03:36:21.943 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.943 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-25T03:36:21.943 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.943 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-25T03:36:21.944 INFO:tasks.rgw_multisite_tests: "marker": "1_1684963742.651710_572.1", 2023-05-25T03:36:21.944 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.944 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-25T03:36:21.944 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.945 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-24T21:29:02.651710Z", 2023-05-25T03:36:21.945 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-25T03:36:21.945 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.945 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.946 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.946 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-25T03:36:21.946 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.946 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-25T03:36:21.946 INFO:tasks.rgw_multisite_tests: "marker": "1_1684962392.333844_91.1", 2023-05-25T03:36:21.947 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.947 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.947 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.947 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-25T03:36:21.948 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-25T03:36:21.948 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.948 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.948 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.949 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-25T03:36:21.949 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.949 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-25T03:36:21.949 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-25T03:36:21.949 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.950 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.950 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.950 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-25T03:36:21.950 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-25T03:36:21.951 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.951 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.951 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.951 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-25T03:36:21.952 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.952 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-25T03:36:21.952 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-25T03:36:21.952 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.953 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.953 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.953 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-25T03:36:21.953 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-25T03:36:21.953 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.954 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.954 INFO:tasks.rgw_multisite_tests: ] 2023-05-25T03:36:21.954 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.954 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-25T03:36:21.955 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-25T03:36:21.955 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-25T03:36:21.955 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.955 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-24T21:29:02Z" 2023-05-25T03:36:21.955 INFO:tasks.rgw_multisite_tests:} 2023-05-25T03:36:21.956 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.956 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963742.651710_572.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:29:02.651710Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-25T03:36:21.956 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1684963742.659011_575.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-25T03:36:21.956 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1684963742.651710_572.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-25T03:36:21.957 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1684963742.659011_575.1 target=1_1684963742.651710_572.1 2023-05-25T03:36:21.957 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.957 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:21.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-25T03:36:21.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-25T03:36:21.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-25T03:36:21.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:29:08.010+0000 7fb50ba5f780 20 RGW-SYNC:meta: read sync status 2023-05-25T03:36:21.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:29:08.011+0000 7fb50ba5f780 20 RGW-SYNC:meta: read sync status 2023-05-25T03:36:21.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "sync_status": { 2023-05-25T03:36:21.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "info": { 2023-05-25T03:36:21.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "status": "sync", 2023-05-25T03:36:21.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "num_shards": 4, 2023-05-25T03:36:21.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-25T03:36:21.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "realm_epoch": 2 2023-05-25T03:36:21.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "markers": [ 2023-05-25T03:36:21.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 0, 2023-05-25T03:36:21.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "state": 1, 2023-05-25T03:36:21.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "1_1684963742.659011_575.1", 2023-05-25T03:36:21.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 1, 2023-05-25T03:36:21.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "2023-05-24T21:29:02.659011Z", 2023-05-25T03:36:21.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "realm_epoch": 2 2023-05-25T03:36:21.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 1, 2023-05-25T03:36:21.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "state": 1, 2023-05-25T03:36:21.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "1_1684962392.333844_91.1", 2023-05-25T03:36:21.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-25T03:36:21.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "realm_epoch": 2 2023-05-25T03:36:21.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 2, 2023-05-25T03:36:21.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "state": 1, 2023-05-25T03:36:21.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "", 2023-05-25T03:36:21.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "0.000000", 2023-05-25T03:36:21.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "realm_epoch": 2 2023-05-25T03:36:21.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: { 2023-05-25T03:36:21.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "key": 3, 2023-05-25T03:36:21.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "val": { 2023-05-25T03:36:21.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "state": 1, 2023-05-25T03:36:21.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "marker": "", 2023-05-25T03:36:21.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "next_step_marker": "", 2023-05-25T03:36:21.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total_entries": 0, 2023-05-25T03:36:21.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "pos": 0, 2023-05-25T03:36:21.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "timestamp": "0.000000", 2023-05-25T03:36:21.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "realm_epoch": 2 2023-05-25T03:36:21.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: ] 2023-05-25T03:36:21.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "full_sync": { 2023-05-25T03:36:21.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "total": 1, 2023-05-25T03:36:21.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "complete": 1 2023-05-25T03:36:21.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: }, 2023-05-25T03:36:21.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: "current_time": "2023-05-24T21:29:08Z" 2023-05-25T03:36:21.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: } 2023-05-25T03:36:21.974 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-25T03:36:21.974 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-25T03:36:21.974 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-25T03:36:21.975 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-25T03:36:21.975 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-25T03:36:21.975 INFO:tasks.rgw_multisite_tests: "period": "5c04936b-70ea-4415-b4eb-3aad3dc612e5", 2023-05-25T03:36:21.975 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-25T03:36:21.975 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.976 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-25T03:36:21.976 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.976 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-25T03:36:21.976 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.977 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-25T03:36:21.977 INFO:tasks.rgw_multisite_tests: "marker": "1_1684963742.659011_575.1", 2023-05-25T03:36:21.977 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.977 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-25T03:36:21.978 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.978 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-24T21:29:02.659011Z", 2023-05-25T03:36:21.978 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-25T03:36:21.978 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.979 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.979 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.979 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-25T03:36:21.979 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.980 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-25T03:36:21.980 INFO:tasks.rgw_multisite_tests: "marker": "1_1684962392.333844_91.1", 2023-05-25T03:36:21.980 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.980 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.980 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.981 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-24T21:06:32.333844Z", 2023-05-25T03:36:21.981 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-25T03:36:21.981 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.981 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.982 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.982 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-25T03:36:21.982 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.982 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-25T03:36:21.983 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-25T03:36:21.983 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.983 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.983 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.983 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-25T03:36:21.984 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-25T03:36:21.984 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.984 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.984 INFO:tasks.rgw_multisite_tests: { 2023-05-25T03:36:21.985 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-25T03:36:21.985 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-25T03:36:21.985 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-25T03:36:21.985 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-25T03:36:21.986 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-25T03:36:21.986 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-25T03:36:21.986 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-25T03:36:21.986 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-25T03:36:21.986 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-25T03:36:21.987 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.987 INFO:tasks.rgw_multisite_tests: } 2023-05-25T03:36:21.987 INFO:tasks.rgw_multisite_tests: ] 2023-05-25T03:36:21.987 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.988 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-25T03:36:21.988 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-25T03:36:21.988 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-25T03:36:21.988 INFO:tasks.rgw_multisite_tests: }, 2023-05-25T03:36:21.989 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-24T21:29:08Z" 2023-05-25T03:36:21.989 INFO:tasks.rgw_multisite_tests:} 2023-05-25T03:36:21.989 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:21.989 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1684963742.659011_575.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-24T21:29:02.659011Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1684962392.333844_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-24T21:06:32.333844Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-25T03:36:21.990 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1684963742.659011_575.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-25T03:36:21.990 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1684963742.659011_575.1', 1: '1_1684962392.333844_91.1', 2: '', 3: ''} 2023-05-25T03:36:21.990 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-05-25T03:36:21.990 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-52', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-25T03:36:21.991 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qskrps-52', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-25T03:36:21.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qskrps-52 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-25T03:36:21.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-25T03:36:21.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-25T03:36:21.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:29:08.143+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:21.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:29:13.144+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:21.992 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: tcmalloc: large alloc 9825697792 bytes == 0x55e5f074e000 @ 0x7f4a63069760 0x7f4a6308ac64 0x7f492584d166 0x7f492584be93 0x55e5e42eb399 0x55e5e45b9124 0x55e5e4599a56 0x55e5e45d6d11 0x55e5e45d75e4 0x55e5e4381402 0x55e5e4175a1c 0x55e5e4179134 0x55e5e40b49b6 0x55e5e40b5684 0x55e5e4f056cf 2023-05-25T03:36:21.992 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: tcmalloc: large alloc 15572418560 bytes == 0x55e83c146000 @ 0x7f4a63069760 0x7f4a6308ac64 0x7f492584d166 0x7f492584be93 0x55e5e42eb399 0x55e5e45b9124 0x55e5e4599a56 0x55e5e45d6d11 0x55e5e45d75e4 0x55e5e4381402 0x55e5e4175a1c 0x55e5e4179134 0x55e5e40b49b6 0x55e5e40b5684 0x55e5e4f056cf 2023-05-25T03:36:21.992 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: terminate called after throwing an instance of 'ceph::buffer::v15_2_0::end_of_buffer' 2023-05-25T03:36:21.993 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: what(): End of buffer [buffer:2] 2023-05-25T03:36:21.993 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: *** Caught signal (Aborted) ** 2023-05-25T03:36:21.993 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: in thread 7f497d388700 thread_name:radosgw 2023-05-25T03:36:21.993 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: ceph version 18.0.0-3443-g3395903b (3395903b291ea37c38b39a05e6d9aa4f40306c80) reef (dev) 2023-05-25T03:36:21.994 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f4a622c0cf0] 2023-05-25T03:36:21.994 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 2: gsignal() 2023-05-25T03:36:21.994 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 3: abort() 2023-05-25T03:36:21.994 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7f4a61a0609b] 2023-05-25T03:36:21.995 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 5: /lib64/libstdc++.so.6(+0x9653c) [0x7f4a61a0c53c] 2023-05-25T03:36:21.995 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 6: /lib64/libstdc++.so.6(+0x95559) [0x7f4a61a0b559] 2023-05-25T03:36:21.995 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 7: __gxx_personality_v0() 2023-05-25T03:36:21.995 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 8: /lib64/libgcc_s.so.1(+0x10b03) [0x7f4a613ecb03] 2023-05-25T03:36:21.995 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 9: _Unwind_Resume() 2023-05-25T03:36:21.996 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 10: radosgw(+0x50fa2e) [0x55e5e3f71a2e] 2023-05-25T03:36:21.996 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 11: radosgw(+0x6432cd) [0x55e5e40a52cd] 2023-05-25T03:36:21.996 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 12: /lib64/libstdc++.so.6(+0xc2b13) [0x7f4a61a38b13] 2023-05-25T03:36:21.996 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 13: /lib64/libpthread.so.0(+0x81ca) [0x7f4a622b61ca] 2023-05-25T03:36:21.997 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 14: clone() 2023-05-25T03:36:21.997 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 2023-05-24T21:29:17.401+0000 7f497d388700 -1 *** Caught signal (Aborted) ** 2023-05-25T03:36:21.997 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: in thread 7f497d388700 thread_name:radosgw 2023-05-25T03:36:21.997 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 2023-05-25T03:36:21.998 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: ceph version 18.0.0-3443-g3395903b (3395903b291ea37c38b39a05e6d9aa4f40306c80) reef (dev) 2023-05-25T03:36:21.998 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f4a622c0cf0] 2023-05-25T03:36:21.998 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 2: gsignal() 2023-05-25T03:36:21.998 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 3: abort() 2023-05-25T03:36:21.999 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7f4a61a0609b] 2023-05-25T03:36:21.999 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 5: /lib64/libstdc++.so.6(+0x9653c) [0x7f4a61a0c53c] 2023-05-25T03:36:21.999 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 6: /lib64/libstdc++.so.6(+0x95559) [0x7f4a61a0b559] 2023-05-25T03:36:21.999 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 7: __gxx_personality_v0() 2023-05-25T03:36:22.000 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 8: /lib64/libgcc_s.so.1(+0x10b03) [0x7f4a613ecb03] 2023-05-25T03:36:22.000 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 9: _Unwind_Resume() 2023-05-25T03:36:22.000 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 10: radosgw(+0x50fa2e) [0x55e5e3f71a2e] 2023-05-25T03:36:22.000 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 11: radosgw(+0x6432cd) [0x55e5e40a52cd] 2023-05-25T03:36:22.000 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 12: /lib64/libstdc++.so.6(+0xc2b13) [0x7f4a61a38b13] 2023-05-25T03:36:22.001 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 13: /lib64/libpthread.so.0(+0x81ca) [0x7f4a622b61ca] 2023-05-25T03:36:22.001 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 14: clone() 2023-05-25T03:36:22.001 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-05-25T03:36:22.001 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 2023-05-25T03:36:22.002 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 0> 2023-05-24T21:29:17.401+0000 7f497d388700 -1 *** Caught signal (Aborted) ** 2023-05-25T03:36:22.002 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: in thread 7f497d388700 thread_name:radosgw 2023-05-25T03:36:22.002 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 2023-05-25T03:36:22.002 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: ceph version 18.0.0-3443-g3395903b (3395903b291ea37c38b39a05e6d9aa4f40306c80) reef (dev) 2023-05-25T03:36:22.003 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f4a622c0cf0] 2023-05-25T03:36:22.003 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 2: gsignal() 2023-05-25T03:36:22.003 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 3: abort() 2023-05-25T03:36:22.003 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7f4a61a0609b] 2023-05-25T03:36:22.003 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 5: /lib64/libstdc++.so.6(+0x9653c) [0x7f4a61a0c53c] 2023-05-25T03:36:22.004 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 6: /lib64/libstdc++.so.6(+0x95559) [0x7f4a61a0b559] 2023-05-25T03:36:22.004 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 7: __gxx_personality_v0() 2023-05-25T03:36:22.004 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 8: /lib64/libgcc_s.so.1(+0x10b03) [0x7f4a613ecb03] 2023-05-25T03:36:22.004 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 9: _Unwind_Resume() 2023-05-25T03:36:22.005 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 10: radosgw(+0x50fa2e) [0x55e5e3f71a2e] 2023-05-25T03:36:22.005 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 11: radosgw(+0x6432cd) [0x55e5e40a52cd] 2023-05-25T03:36:22.005 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 12: /lib64/libstdc++.so.6(+0xc2b13) [0x7f4a61a38b13] 2023-05-25T03:36:22.005 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 13: /lib64/libpthread.so.0(+0x81ca) [0x7f4a622b61ca] 2023-05-25T03:36:22.006 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 14: clone() 2023-05-25T03:36:22.006 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-05-25T03:36:22.006 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 2023-05-25T03:36:22.006 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 0> 2023-05-24T21:29:17.401+0000 7f497d388700 -1 *** Caught signal (Aborted) ** 2023-05-25T03:36:22.007 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: in thread 7f497d388700 thread_name:radosgw 2023-05-25T03:36:22.007 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 2023-05-25T03:36:22.007 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: ceph version 18.0.0-3443-g3395903b (3395903b291ea37c38b39a05e6d9aa4f40306c80) reef (dev) 2023-05-25T03:36:22.007 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f4a622c0cf0] 2023-05-25T03:36:22.007 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 2: gsignal() 2023-05-25T03:36:22.008 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 3: abort() 2023-05-25T03:36:22.008 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7f4a61a0609b] 2023-05-25T03:36:22.008 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 5: /lib64/libstdc++.so.6(+0x9653c) [0x7f4a61a0c53c] 2023-05-25T03:36:22.008 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 6: /lib64/libstdc++.so.6(+0x95559) [0x7f4a61a0b559] 2023-05-25T03:36:22.009 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 7: __gxx_personality_v0() 2023-05-25T03:36:22.009 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 8: /lib64/libgcc_s.so.1(+0x10b03) [0x7f4a613ecb03] 2023-05-25T03:36:22.009 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 9: _Unwind_Resume() 2023-05-25T03:36:22.009 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 10: radosgw(+0x50fa2e) [0x55e5e3f71a2e] 2023-05-25T03:36:22.010 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 11: radosgw(+0x6432cd) [0x55e5e40a52cd] 2023-05-25T03:36:22.010 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 12: /lib64/libstdc++.so.6(+0xc2b13) [0x7f4a61a38b13] 2023-05-25T03:36:22.010 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 13: /lib64/libpthread.so.0(+0x81ca) [0x7f4a622b61ca] 2023-05-25T03:36:22.010 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 14: clone() 2023-05-25T03:36:22.011 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-05-25T03:36:22.011 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stdout: INFO: 2023-05-25T03:36:22.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:29:18.144+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.011 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi071.stderr: INFO: daemon-helper: command crashed with signal 6 2023-05-25T03:36:22.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:29:23.144+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.012 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~0s 2023-05-25T03:36:22.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:29:28.144+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.012 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~6s 2023-05-25T03:36:22.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:29:33.144+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.013 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~11s 2023-05-25T03:36:22.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:29:38.145+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.013 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~17s 2023-05-25T03:36:22.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:29:43.146+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.014 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~22s 2023-05-25T03:36:22.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:29:48.147+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.014 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~28s 2023-05-25T03:36:22.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:29:53.147+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.014 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~34s 2023-05-25T03:36:22.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:29:58.148+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:30:03.149+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.015 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~39s 2023-05-25T03:36:22.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:30:08.149+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.016 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~45s 2023-05-25T03:36:22.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:30:13.150+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.016 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~50s 2023-05-25T03:36:22.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:30:18.150+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.017 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~56s 2023-05-25T03:36:22.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:30:23.150+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.017 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~62s 2023-05-25T03:36:22.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:30:28.151+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.018 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~67s 2023-05-25T03:36:22.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:30:33.152+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.018 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~73s 2023-05-25T03:36:22.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:30:38.153+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.018 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~78s 2023-05-25T03:36:22.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:30:43.154+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.019 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi071.stdout: INFO: 2023-05-24T21:30:46.608+0000 7f1a4f21b700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-05-25T03:36:22.019 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~84s 2023-05-25T03:36:22.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:30:48.154+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:30:53.155+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.020 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~90s 2023-05-25T03:36:22.020 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.1.smithi157.stdout: INFO: 2023-05-24T21:30:57.874+0000 7f37d6f3f700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-05-25T03:36:22.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:30:58.156+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.021 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~95s 2023-05-25T03:36:22.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:31:03.156+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.021 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~101s 2023-05-25T03:36:22.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:31:08.157+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.022 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~106s 2023-05-25T03:36:22.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:31:13.156+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.022 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~112s 2023-05-25T03:36:22.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:31:18.157+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.022 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~118s 2023-05-25T03:36:22.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:31:23.158+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.023 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~123s 2023-05-25T03:36:22.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:31:28.158+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.023 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~129s 2023-05-25T03:36:22.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:31:33.160+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:31:38.161+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.024 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~134s 2023-05-25T03:36:22.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:31:43.161+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.025 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~140s 2023-05-25T03:36:22.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:31:48.162+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.025 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~146s 2023-05-25T03:36:22.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:31:53.162+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.026 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~151s 2023-05-25T03:36:22.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:31:58.163+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.026 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~157s 2023-05-25T03:36:22.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:32:03.164+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.027 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~163s 2023-05-25T03:36:22.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:32:08.163+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.027 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~168s 2023-05-25T03:36:22.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:32:13.164+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.027 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~174s 2023-05-25T03:36:22.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:32:18.165+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:32:23.166+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.028 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~179s 2023-05-25T03:36:22.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:32:28.167+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.029 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~185s 2023-05-25T03:36:22.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:32:33.167+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.029 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~191s 2023-05-25T03:36:22.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:32:38.168+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.030 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~196s 2023-05-25T03:36:22.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:32:43.169+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.030 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~202s 2023-05-25T03:36:22.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:32:48.169+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.030 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~207s 2023-05-25T03:36:22.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:32:53.170+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.031 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~213s 2023-05-25T03:36:22.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:32:58.170+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.031 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~219s 2023-05-25T03:36:22.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:33:03.170+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.032 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~224s 2023-05-25T03:36:22.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:33:08.171+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:33:13.171+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.033 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~230s 2023-05-25T03:36:22.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:33:18.172+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.033 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~235s 2023-05-25T03:36:22.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:33:23.173+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.034 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~241s 2023-05-25T03:36:22.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:33:28.174+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.034 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~247s 2023-05-25T03:36:22.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:33:33.175+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.035 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~252s 2023-05-25T03:36:22.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:33:38.175+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.035 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~258s 2023-05-25T03:36:22.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:33:43.176+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.035 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~263s 2023-05-25T03:36:22.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:33:48.177+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.036 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~269s 2023-05-25T03:36:22.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:33:53.177+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:33:58.177+0000 7fee02333780 1 waiting to reach incremental sync.. 2023-05-25T03:36:22.037 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~275s 2023-05-25T03:36:22.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:34:03.178+0000 7fee02333780 -1 bucket checkpoint timed out waiting to reach incremental sync 2023-05-25T03:36:22.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:34:03.178+0000 7fee02333780 0 bucket sync checkpoint failed: (110) Connection timed out 2023-05-25T03:36:22.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:34:03.178+0000 7fee02333780 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-05-25T03:36:22.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2023-05-25T03:36:22.038 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-05-25T03:36:22.038 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:22.038 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-05-25T03:36:22.039 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_creation_time 2023-05-25T03:36:22.039 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-05-25T03:36:22.039 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-05-25T03:36:22.039 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-05-25T03:36:22.040 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-05-25T03:36:22.040 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/../src/test/rgw/rgw_multi/tests.py", line 1504, in test_bucket_creation_time 2023-05-25T03:36:22.040 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2023-05-25T03:36:22.040 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/../src/test/rgw/rgw_multi/tests.py", line 203, in zonegroup_meta_checkpoint 2023-05-25T03:36:22.041 INFO:tasks.rgw_multisite_tests: master_status = meta_master_log_status(meta_master_zone) 2023-05-25T03:36:22.041 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/../src/test/rgw/rgw_multi/tests.py", line 149, in meta_master_log_status 2023-05-25T03:36:22.041 INFO:tasks.rgw_multisite_tests: mdlog_status_json, retcode = master_zone.cluster.admin(cmd, read_only=True) 2023-05-25T03:36:22.041 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/tasks/rgw_multisite.py", line 197, in admin 2023-05-25T03:36:22.042 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-05-25T03:36:22.042 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-05-25T03:36:22.042 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-05-25T03:36:22.042 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:22.042 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:22.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-25T03:36:22.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-25T03:36:22.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-25T03:36:22.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:42:20.726+0000 7f812a6bf780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:47:20.727+0000 7f812a6bf780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:52:20.728+0000 7f812a6bf780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T21:57:20.728+0000 7f812a6bf780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T22:02:20.729+0000 7f812a6bf780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T22:07:20.729+0000 7f812a6bf780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T22:12:20.729+0000 7f812a6bf780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T22:17:20.730+0000 7f812a6bf780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T22:22:20.730+0000 7f812a6bf780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T22:27:20.731+0000 7f812a6bf780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: failed to fetch mon config (--no-mon-config to skip) 2023-05-25T03:36:22.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 1 2023-05-25T03:36:22.046 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-05-25T03:36:22.046 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:22.047 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-05-25T03:36:22.047 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_policy_config_zonegroup: 2023-05-25T03:36:22.047 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-05-25T03:36:22.048 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-05-25T03:36:22.048 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-05-25T03:36:22.048 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-05-25T03:36:22.048 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/../src/test/rgw/rgw_multi/tests.py", line 2064, in test_sync_policy_config_zonegroup 2023-05-25T03:36:22.048 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2023-05-25T03:36:22.049 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/../src/test/rgw/rgw_multi/tests.py", line 203, in zonegroup_meta_checkpoint 2023-05-25T03:36:22.049 INFO:tasks.rgw_multisite_tests: master_status = meta_master_log_status(meta_master_zone) 2023-05-25T03:36:22.049 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/../src/test/rgw/rgw_multi/tests.py", line 149, in meta_master_log_status 2023-05-25T03:36:22.049 INFO:tasks.rgw_multisite_tests: mdlog_status_json, retcode = master_zone.cluster.admin(cmd, read_only=True) 2023-05-25T03:36:22.050 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/tasks/rgw_multisite.py", line 197, in admin 2023-05-25T03:36:22.050 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-05-25T03:36:22.050 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-05-25T03:36:22.050 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-05-25T03:36:22.051 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:22.051 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:22.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-25T03:36:22.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-25T03:36:22.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-25T03:36:22.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T22:41:19.225+0000 7f92ac2f4780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T22:46:19.225+0000 7f92ac2f4780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T22:51:19.226+0000 7f92ac2f4780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T22:56:19.226+0000 7f92ac2f4780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T23:01:19.227+0000 7f92ac2f4780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T23:06:19.228+0000 7f92ac2f4780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T23:11:19.227+0000 7f92ac2f4780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T23:16:19.228+0000 7f92ac2f4780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T23:21:19.229+0000 7f92ac2f4780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T23:26:19.230+0000 7f92ac2f4780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: failed to fetch mon config (--no-mon-config to skip) 2023-05-25T03:36:22.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 1 2023-05-25T03:36:22.055 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-05-25T03:36:22.055 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:22.055 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-05-25T03:36:22.055 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_flow_symmetrical_zonegroup_all: 2023-05-25T03:36:22.056 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-05-25T03:36:22.056 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-05-25T03:36:22.056 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-05-25T03:36:22.056 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-05-25T03:36:22.057 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/../src/test/rgw/rgw_multi/tests.py", line 2136, in test_sync_flow_symmetrical_zonegroup_all 2023-05-25T03:36:22.057 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2023-05-25T03:36:22.057 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/../src/test/rgw/rgw_multi/tests.py", line 203, in zonegroup_meta_checkpoint 2023-05-25T03:36:22.057 INFO:tasks.rgw_multisite_tests: master_status = meta_master_log_status(meta_master_zone) 2023-05-25T03:36:22.057 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/../src/test/rgw/rgw_multi/tests.py", line 149, in meta_master_log_status 2023-05-25T03:36:22.058 INFO:tasks.rgw_multisite_tests: mdlog_status_json, retcode = master_zone.cluster.admin(cmd, read_only=True) 2023-05-25T03:36:22.058 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/tasks/rgw_multisite.py", line 197, in admin 2023-05-25T03:36:22.058 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-05-25T03:36:22.058 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-05-25T03:36:22.059 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-05-25T03:36:22.059 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:22.059 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:22.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-25T03:36:22.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-25T03:36:22.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-25T03:36:22.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T23:31:19.326+0000 7f3853eaa780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T23:36:19.326+0000 7f3853eaa780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T23:41:19.327+0000 7f3853eaa780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T23:46:19.327+0000 7f3853eaa780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T23:51:19.328+0000 7f3853eaa780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-24T23:56:19.328+0000 7f3853eaa780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-25T00:01:19.329+0000 7f3853eaa780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-25T00:06:19.329+0000 7f3853eaa780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-25T00:11:19.331+0000 7f3853eaa780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-25T00:16:19.331+0000 7f3853eaa780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: failed to fetch mon config (--no-mon-config to skip) 2023-05-25T03:36:22.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 1 2023-05-25T03:36:22.063 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-05-25T03:36:22.063 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:22.063 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-05-25T03:36:22.064 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_single_bucket: 2023-05-25T03:36:22.064 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-05-25T03:36:22.064 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-05-25T03:36:22.065 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-05-25T03:36:22.065 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-05-25T03:36:22.065 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/../src/test/rgw/rgw_multi/tests.py", line 2382, in test_sync_single_bucket 2023-05-25T03:36:22.065 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2023-05-25T03:36:22.065 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/../src/test/rgw/rgw_multi/tests.py", line 203, in zonegroup_meta_checkpoint 2023-05-25T03:36:22.066 INFO:tasks.rgw_multisite_tests: master_status = meta_master_log_status(meta_master_zone) 2023-05-25T03:36:22.066 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/../src/test/rgw/rgw_multi/tests.py", line 149, in meta_master_log_status 2023-05-25T03:36:22.066 INFO:tasks.rgw_multisite_tests: mdlog_status_json, retcode = master_zone.cluster.admin(cmd, read_only=True) 2023-05-25T03:36:22.066 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/tasks/rgw_multisite.py", line 197, in admin 2023-05-25T03:36:22.067 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-05-25T03:36:22.067 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-05-25T03:36:22.067 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-05-25T03:36:22.067 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:22.068 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:22.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-25T03:36:22.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-25T03:36:22.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-25T03:36:22.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-25T00:21:19.403+0000 7f94180e8780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-25T00:26:19.403+0000 7f94180e8780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-25T00:31:19.404+0000 7f94180e8780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-25T00:36:19.404+0000 7f94180e8780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-25T00:41:19.405+0000 7f94180e8780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-25T00:46:19.405+0000 7f94180e8780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-25T00:51:19.406+0000 7f94180e8780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-25T00:56:19.406+0000 7f94180e8780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-25T01:01:19.406+0000 7f94180e8780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-25T01:06:19.406+0000 7f94180e8780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: failed to fetch mon config (--no-mon-config to skip) 2023-05-25T03:36:22.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 1 2023-05-25T03:36:22.071 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-05-25T03:36:22.072 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:22.072 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-05-25T03:36:22.072 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_different_buckets: 2023-05-25T03:36:22.072 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-05-25T03:36:22.073 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-05-25T03:36:22.073 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-05-25T03:36:22.073 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-05-25T03:36:22.073 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/../src/test/rgw/rgw_multi/tests.py", line 2499, in test_sync_different_buckets 2023-05-25T03:36:22.074 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2023-05-25T03:36:22.074 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/../src/test/rgw/rgw_multi/tests.py", line 203, in zonegroup_meta_checkpoint 2023-05-25T03:36:22.074 INFO:tasks.rgw_multisite_tests: master_status = meta_master_log_status(meta_master_zone) 2023-05-25T03:36:22.074 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/../src/test/rgw/rgw_multi/tests.py", line 149, in meta_master_log_status 2023-05-25T03:36:22.075 INFO:tasks.rgw_multisite_tests: mdlog_status_json, retcode = master_zone.cluster.admin(cmd, read_only=True) 2023-05-25T03:36:22.075 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/tasks/rgw_multisite.py", line 197, in admin 2023-05-25T03:36:22.075 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-05-25T03:36:22.075 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-05-25T03:36:22.076 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-05-25T03:36:22.076 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:22.076 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:22.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-25T03:36:22.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-25T03:36:22.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-25T03:36:22.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-25T01:11:19.692+0000 7f015f792780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-25T01:16:19.693+0000 7f015f792780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-25T01:21:19.693+0000 7f015f792780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-25T01:26:19.694+0000 7f015f792780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-25T01:31:19.694+0000 7f015f792780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-25T01:36:19.695+0000 7f015f792780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-25T01:41:19.694+0000 7f015f792780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-25T01:46:19.696+0000 7f015f792780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-25T01:51:19.696+0000 7f015f792780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-25T01:56:19.697+0000 7f015f792780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: failed to fetch mon config (--no-mon-config to skip) 2023-05-25T03:36:22.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 1 2023-05-25T03:36:22.080 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-05-25T03:36:22.080 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:22.081 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-05-25T03:36:22.081 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_multiple_buckets_to_single: 2023-05-25T03:36:22.082 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-05-25T03:36:22.082 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-05-25T03:36:22.082 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-05-25T03:36:22.083 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-05-25T03:36:22.083 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/../src/test/rgw/rgw_multi/tests.py", line 2645, in test_sync_multiple_buckets_to_single 2023-05-25T03:36:22.083 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2023-05-25T03:36:22.083 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/../src/test/rgw/rgw_multi/tests.py", line 203, in zonegroup_meta_checkpoint 2023-05-25T03:36:22.083 INFO:tasks.rgw_multisite_tests: master_status = meta_master_log_status(meta_master_zone) 2023-05-25T03:36:22.084 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/../src/test/rgw/rgw_multi/tests.py", line 149, in meta_master_log_status 2023-05-25T03:36:22.084 INFO:tasks.rgw_multisite_tests: mdlog_status_json, retcode = master_zone.cluster.admin(cmd, read_only=True) 2023-05-25T03:36:22.084 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/tasks/rgw_multisite.py", line 197, in admin 2023-05-25T03:36:22.084 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-05-25T03:36:22.085 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-05-25T03:36:22.085 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-05-25T03:36:22.085 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:22.085 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:22.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-25T03:36:22.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-25T03:36:22.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-25T03:36:22.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-25T02:01:20.115+0000 7fb9e9a76780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-25T02:06:20.116+0000 7fb9e9a76780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-25T02:11:20.116+0000 7fb9e9a76780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-25T02:16:20.117+0000 7fb9e9a76780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-25T02:21:20.117+0000 7fb9e9a76780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-25T02:26:20.119+0000 7fb9e9a76780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-25T02:31:20.119+0000 7fb9e9a76780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-25T02:36:20.120+0000 7fb9e9a76780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-25T02:41:20.120+0000 7fb9e9a76780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-25T02:46:20.120+0000 7fb9e9a76780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: failed to fetch mon config (--no-mon-config to skip) 2023-05-25T03:36:22.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 1 2023-05-25T03:36:22.089 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-05-25T03:36:22.089 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:22.090 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-05-25T03:36:22.090 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_single_bucket_to_multiple: 2023-05-25T03:36:22.090 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-05-25T03:36:22.090 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-05-25T03:36:22.091 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-05-25T03:36:22.091 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-05-25T03:36:22.091 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/../src/test/rgw/rgw_multi/tests.py", line 2766, in test_sync_single_bucket_to_multiple 2023-05-25T03:36:22.091 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2023-05-25T03:36:22.092 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/../src/test/rgw/rgw_multi/tests.py", line 203, in zonegroup_meta_checkpoint 2023-05-25T03:36:22.092 INFO:tasks.rgw_multisite_tests: master_status = meta_master_log_status(meta_master_zone) 2023-05-25T03:36:22.092 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/../src/test/rgw/rgw_multi/tests.py", line 149, in meta_master_log_status 2023-05-25T03:36:22.093 INFO:tasks.rgw_multisite_tests: mdlog_status_json, retcode = master_zone.cluster.admin(cmd, read_only=True) 2023-05-25T03:36:22.093 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/tasks/rgw_multisite.py", line 197, in admin 2023-05-25T03:36:22.093 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-05-25T03:36:22.093 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-05-25T03:36:22.094 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-05-25T03:36:22.094 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:22.094 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-25T03:36:22.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-25T03:36:22.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-25T03:36:22.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-25T03:36:22.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-25T02:51:20.510+0000 7f04a241b780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-25T02:56:20.510+0000 7f04a241b780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-25T03:01:20.511+0000 7f04a241b780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-25T03:06:20.511+0000 7f04a241b780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-25T03:11:20.512+0000 7f04a241b780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-25T03:16:20.512+0000 7f04a241b780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-25T03:21:20.513+0000 7f04a241b780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-25T03:26:20.514+0000 7f04a241b780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-25T03:31:20.514+0000 7f04a241b780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2023-05-25T03:36:20.515+0000 7f04a241b780 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:36:22.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: failed to fetch mon config (--no-mon-config to skip) 2023-05-25T03:36:22.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 1 2023-05-25T03:36:22.098 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-05-25T03:36:22.098 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:22.099 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-05-25T03:36:22.099 INFO:tasks.rgw_multisite_tests:Ran 47 tests in 23624.064s 2023-05-25T03:36:22.099 INFO:tasks.rgw_multisite_tests: 2023-05-25T03:36:22.099 INFO:tasks.rgw_multisite_tests:FAILED (SKIP=4, errors=12, failures=10) 2023-05-25T03:36:22.123 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/tasks/rgw_multisite_tests.py", line 95, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2023-05-25T03:36:23.659 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=63f94308abe849c5a88590818a2c9577 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/github.com_ceph_ceph-c_3395903b291ea37c38b39a05e6d9aa4f40306c80/qa/tasks/rgw_multisite_tests.py", line 95, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2023-05-25T03:36:23.669 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite-tests 2023-05-25T03:36:23.682 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite 2023-05-25T03:36:23.691 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2023-05-25T03:36:23.707 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2023-05-25T03:36:23.743 INFO:teuthology.orchestra.run:waiting for 300 2023-05-25T03:36:23.746 INFO:tasks.rgw.c1.client.0:Stopped 2023-05-25T03:36:23.747 DEBUG:teuthology.orchestra.run.smithi071:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock 2023-05-25T03:36:23.783 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f /etc/ceph/vault-root-token 2023-05-25T03:36:23.859 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-05-25T03:36:23.859 INFO:teuthology.orchestra.run:waiting for 300 2023-05-25T03:36:23.860 INFO:tasks.rgw.c1.client.1:Stopped 2023-05-25T03:36:23.860 DEBUG:teuthology.orchestra.run.smithi071:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock 2023-05-25T03:36:23.919 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f /etc/ceph/vault-root-token 2023-05-25T03:36:23.990 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2023-05-25T03:36:23.991 INFO:teuthology.orchestra.run:waiting for 300 2023-05-25T03:36:24.003 INFO:tasks.rgw.c2.client.0.smithi157.stdout:2023-05-25T03:36:23.988+0000 7f70c8e2a700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm (PID: 111857) UID: 0 2023-05-25T03:36:24.013 INFO:tasks.rgw.c2.client.0.smithi157.stdout:2023-05-25T03:36:23.988+0000 7f70cd0937c0 -1 shutting down 2023-05-25T03:36:25.149 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2023-05-25T03:36:30.095 INFO:tasks.rgw.c2.client.0:Stopped 2023-05-25T03:36:30.096 DEBUG:teuthology.orchestra.run.smithi157:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock 2023-05-25T03:36:30.305 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f /etc/ceph/vault-root-token 2023-05-25T03:36:30.341 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2023-05-25T03:36:30.342 INFO:teuthology.orchestra.run:waiting for 300 2023-05-25T03:36:30.382 INFO:tasks.rgw.c2.client.1.smithi157.stdout:2023-05-25T03:36:30.378+0000 7f37f477a700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm (PID: 113152) UID: 0 2023-05-25T03:36:30.400 INFO:tasks.rgw.c2.client.1.smithi157.stdout:2023-05-25T03:36:30.378+0000 7f37f89e37c0 -1 shutting down 2023-05-25T03:36:30.777 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2023-05-25T03:36:36.282 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2023-05-25T03:36:36.443 INFO:tasks.rgw.c2.client.1:Stopped 2023-05-25T03:36:36.444 DEBUG:teuthology.orchestra.run.smithi157:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock 2023-05-25T03:36:36.470 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f /etc/ceph/vault-root-token 2023-05-25T03:36:36.542 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-05-25T03:36:36.556 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster c2 config set global mon_health_to_clog false 2023-05-25T03:36:37.258 INFO:teuthology.misc:Shutting down mds daemons... 2023-05-25T03:36:37.258 INFO:teuthology.misc:Shutting down osd daemons... 2023-05-25T03:36:37.259 DEBUG:tasks.ceph.c2.osd.0:waiting for process to exit 2023-05-25T03:36:37.259 INFO:teuthology.orchestra.run:waiting for 300 2023-05-25T03:36:37.323 INFO:tasks.ceph.c2.osd.0:Stopped 2023-05-25T03:36:37.324 DEBUG:tasks.ceph.c2.osd.1:waiting for process to exit 2023-05-25T03:36:37.324 INFO:teuthology.orchestra.run:waiting for 300 2023-05-25T03:36:37.378 INFO:tasks.ceph.c2.osd.1:Stopped 2023-05-25T03:36:37.379 DEBUG:tasks.ceph.c2.osd.2:waiting for process to exit 2023-05-25T03:36:37.380 INFO:teuthology.orchestra.run:waiting for 300 2023-05-25T03:36:37.445 INFO:tasks.ceph.c2.osd.2:Stopped 2023-05-25T03:36:37.446 INFO:teuthology.misc:Shutting down mgr daemons... 2023-05-25T03:36:37.447 DEBUG:tasks.ceph.c2.mgr.x:waiting for process to exit 2023-05-25T03:36:37.447 INFO:teuthology.orchestra.run:waiting for 300 2023-05-25T03:36:37.472 INFO:tasks.ceph.c2.mgr.x:Stopped 2023-05-25T03:36:37.472 INFO:teuthology.misc:Shutting down mon daemons... 2023-05-25T03:36:37.473 DEBUG:tasks.ceph.c2.mon.a:waiting for process to exit 2023-05-25T03:36:37.473 INFO:teuthology.orchestra.run:waiting for 300 2023-05-25T03:36:37.548 INFO:tasks.ceph.c2.mon.a:Stopped 2023-05-25T03:36:37.549 INFO:tasks.ceph:Checking cluster log for badness... 2023-05-25T03:36:37.549 DEBUG:teuthology.orchestra.run.smithi157:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-05-25T03:36:37.670 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-0 on ubuntu@smithi157.front.sepia.ceph.com 2023-05-25T03:36:37.670 DEBUG:teuthology.orchestra.run.smithi157:> sync && sudo umount -f /var/lib/ceph/osd/c2-0 2023-05-25T03:36:37.952 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-1 on ubuntu@smithi157.front.sepia.ceph.com 2023-05-25T03:36:37.952 DEBUG:teuthology.orchestra.run.smithi157:> sync && sudo umount -f /var/lib/ceph/osd/c2-1 2023-05-25T03:36:38.052 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-2 on ubuntu@smithi157.front.sepia.ceph.com 2023-05-25T03:36:38.053 DEBUG:teuthology.orchestra.run.smithi157:> sync && sudo umount -f /var/lib/ceph/osd/c2-2 2023-05-25T03:36:38.130 INFO:tasks.ceph:Archiving mon data... 2023-05-25T03:36:38.132 DEBUG:teuthology.misc:Transferring archived files from smithi157:/var/lib/ceph/mon/c2-a to /home/teuthworker/archive/yuriw-2023-05-24_20:32:11-rgw-wip-yuri-testing-2023-05-24-0738-reef-distro-default-smithi/7285505/data/c2.mon.a.tgz 2023-05-25T03:36:38.132 DEBUG:teuthology.orchestra.run.smithi157:> mktemp 2023-05-25T03:36:38.163 INFO:teuthology.orchestra.run.smithi157.stdout:/tmp/tmp.RCIUod1eNL 2023-05-25T03:36:38.163 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar cz -f - -C /var/lib/ceph/mon/c2-a -- . > /tmp/tmp.RCIUod1eNL 2023-05-25T03:36:38.382 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0666 /tmp/tmp.RCIUod1eNL 2023-05-25T03:36:38.503 DEBUG:teuthology.orchestra.remote:smithi157:/tmp/tmp.RCIUod1eNL is 593KB 2023-05-25T03:36:38.575 DEBUG:teuthology.orchestra.run.smithi157:> rm -fr /tmp/tmp.RCIUod1eNL 2023-05-25T03:36:38.592 INFO:tasks.ceph:Cleaning ceph cluster... 2023-05-25T03:36:38.593 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2023-05-25T03:36:38.596 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2023-05-25T03:36:38.659 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-05-25T03:36:38.673 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster c1 config set global mon_health_to_clog false 2023-05-25T03:36:41.283 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2023-05-25T03:41:38.803 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T03:41:38.798+0000 7ff5c3bde700 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:46:38.804 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T03:46:38.799+0000 7ff5c3bde700 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:51:38.805 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T03:51:38.800+0000 7ff5c3bde700 0 monclient(hunting): authenticate timed out after 300 2023-05-25T03:56:38.805 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T03:56:38.801+0000 7ff5c3bde700 0 monclient(hunting): authenticate timed out after 300 2023-05-25T04:01:38.806 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T04:01:38.801+0000 7ff5c3bde700 0 monclient(hunting): authenticate timed out after 300 2023-05-25T04:06:38.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T04:06:38.802+0000 7ff5c3bde700 0 monclient(hunting): authenticate timed out after 300 2023-05-25T04:11:38.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T04:11:38.803+0000 7ff5c3bde700 0 monclient(hunting): authenticate timed out after 300 2023-05-25T04:16:38.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T04:16:38.804+0000 7ff5c3bde700 0 monclient(hunting): authenticate timed out after 300 2023-05-25T04:21:38.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T04:21:38.805+0000 7ff5c3bde700 0 monclient(hunting): authenticate timed out after 300 2023-05-25T04:26:38.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-05-25T04:26:38.805+0000 7ff5c3bde700 0 monclient(hunting): authenticate timed out after 300 2023-05-25T04:26:38.812 INFO:teuthology.orchestra.run.smithi071.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-05-25T04:26:38.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-25T04:26:38.815 INFO:teuthology.misc:Shutting down mds daemons... 2023-05-25T04:26:38.816 INFO:teuthology.misc:Shutting down osd daemons... 2023-05-25T04:26:38.816 DEBUG:tasks.ceph.c1.osd.0:waiting for process to exit 2023-05-25T04:26:38.817 INFO:teuthology.orchestra.run:waiting for 300 2023-05-25T04:26:38.817 INFO:tasks.ceph.c1.osd.0:Stopped 2023-05-25T04:26:38.818 DEBUG:tasks.ceph.c1.osd.1:waiting for process to exit 2023-05-25T04:26:38.818 INFO:teuthology.orchestra.run:waiting for 300 2023-05-25T04:26:38.818 INFO:tasks.ceph.c1.osd.1:Stopped 2023-05-25T04:26:38.819 DEBUG:tasks.ceph.c1.osd.2:waiting for process to exit 2023-05-25T04:26:38.819 INFO:teuthology.orchestra.run:waiting for 300 2023-05-25T04:26:38.819 INFO:tasks.ceph.c1.osd.2:Stopped 2023-05-25T04:26:38.820 INFO:teuthology.misc:Shutting down mgr daemons... 2023-05-25T04:26:38.820 DEBUG:tasks.ceph.c1.mgr.x:waiting for process to exit 2023-05-25T04:26:38.820 INFO:teuthology.orchestra.run:waiting for 300 2023-05-25T04:26:38.821 INFO:tasks.ceph.c1.mgr.x:Stopped 2023-05-25T04:26:38.821 INFO:teuthology.misc:Shutting down mon daemons... 2023-05-25T04:26:38.822 DEBUG:tasks.ceph.c1.mon.a:waiting for process to exit 2023-05-25T04:26:38.822 INFO:teuthology.orchestra.run:waiting for 300 2023-05-25T04:26:38.822 INFO:tasks.ceph.c1.mon.a:Stopped 2023-05-25T04:26:38.823 INFO:tasks.ceph:Checking cluster log for badness... 2023-05-25T04:26:38.823 DEBUG:teuthology.orchestra.run.smithi071:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-05-25T04:26:38.864 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-0 on ubuntu@smithi071.front.sepia.ceph.com 2023-05-25T04:26:38.864 DEBUG:teuthology.orchestra.run.smithi071:> sync && sudo umount -f /var/lib/ceph/osd/c1-0 2023-05-25T04:26:39.078 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-1 on ubuntu@smithi071.front.sepia.ceph.com 2023-05-25T04:26:39.078 DEBUG:teuthology.orchestra.run.smithi071:> sync && sudo umount -f /var/lib/ceph/osd/c1-1 2023-05-25T04:26:39.181 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-2 on ubuntu@smithi071.front.sepia.ceph.com 2023-05-25T04:26:39.182 DEBUG:teuthology.orchestra.run.smithi071:> sync && sudo umount -f /var/lib/ceph/osd/c1-2 2023-05-25T04:26:39.253 INFO:tasks.ceph:Archiving mon data... 2023-05-25T04:26:39.254 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/lib/ceph/mon/c1-a to /home/teuthworker/archive/yuriw-2023-05-24_20:32:11-rgw-wip-yuri-testing-2023-05-24-0738-reef-distro-default-smithi/7285505/data/c1.mon.a.tgz 2023-05-25T04:26:39.255 DEBUG:teuthology.orchestra.run.smithi071:> mktemp 2023-05-25T04:26:39.273 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.8WstKW7K0f 2023-05-25T04:26:39.273 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar cz -f - -C /var/lib/ceph/mon/c1-a -- . > /tmp/tmp.8WstKW7K0f 2023-05-25T04:26:39.493 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.8WstKW7K0f 2023-05-25T04:26:39.580 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.8WstKW7K0f is 520KB 2023-05-25T04:26:39.675 DEBUG:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.8WstKW7K0f 2023-05-25T04:26:39.692 INFO:tasks.ceph:Cleaning ceph cluster... 2023-05-25T04:26:39.693 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2023-05-25T04:26:39.734 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2023-05-25T04:26:39.773 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-05-25T04:26:39.774 DEBUG:teuthology.orchestra.run.smithi071:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-05-25T04:26:39.801 DEBUG:teuthology.orchestra.run.smithi157:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-05-25T04:26:39.826 INFO:teuthology.orchestra.run.smithi071.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-05-25T04:26:39.845 INFO:teuthology.orchestra.run.smithi157.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-05-25T04:26:40.043 INFO:tasks.ceph:Archiving crash dumps... 2023-05-25T04:26:40.045 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-05-24_20:32:11-rgw-wip-yuri-testing-2023-05-24-0738-reef-distro-default-smithi/7285505/remote/smithi071/crash 2023-05-25T04:26:40.046 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-05-25T04:26:40.096 DEBUG:teuthology.misc:Transferring archived files from smithi157:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-05-24_20:32:11-rgw-wip-yuri-testing-2023-05-24-0738-reef-distro-default-smithi/7285505/remote/smithi157/crash 2023-05-25T04:26:40.097 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-05-25T04:26:40.129 INFO:tasks.ceph:Compressing logs... 2023-05-25T04:26:40.129 DEBUG:teuthology.orchestra.run.smithi071:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-05-25T04:26:40.131 DEBUG:teuthology.orchestra.run.smithi157:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-05-25T04:33:07.501 INFO:tasks.ceph:Archiving logs... 2023-05-25T04:33:07.503 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-05-24_20:32:11-rgw-wip-yuri-testing-2023-05-24-0738-reef-distro-default-smithi/7285505/remote/smithi071/log 2023-05-25T04:33:07.504 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar c -f - -C /var/log/ceph -- . 2023-05-25T04:33:12.505 DEBUG:teuthology.misc:Transferring archived files from smithi157:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-05-24_20:32:11-rgw-wip-yuri-testing-2023-05-24-0738-reef-distro-default-smithi/7285505/remote/smithi157/log 2023-05-25T04:33:12.506 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /var/log/ceph -- . 2023-05-25T04:33:47.094 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-05-25T04:33:47.111 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2023-05-25T04:33:47.112 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2023-05-25T04:33:47.114 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2023-05-25T04:33:47.236 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-05-25T04:33:47.245 INFO:teuthology.task.clock:Checking final clock skew... 2023-05-25T04:33:47.246 DEBUG:teuthology.orchestra.run.smithi071:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-05-25T04:33:47.248 DEBUG:teuthology.orchestra.run.smithi157:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-05-25T04:33:47.266 INFO:teuthology.orchestra.run.smithi157.stderr:bash: ntpq: command not found 2023-05-25T04:33:47.266 INFO:teuthology.orchestra.run.smithi071.stderr:bash: ntpq: command not found 2023-05-25T04:33:47.272 INFO:teuthology.orchestra.run.smithi071.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-05-25T04:33:47.272 INFO:teuthology.orchestra.run.smithi071.stdout:=============================================================================== 2023-05-25T04:33:47.272 INFO:teuthology.orchestra.run.smithi071.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 38 -2678us[-2702us] +/- 79ms 2023-05-25T04:33:47.272 INFO:teuthology.orchestra.run.smithi071.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 85 -429us[ -453us] +/- 87ms 2023-05-25T04:33:47.273 INFO:teuthology.orchestra.run.smithi071.stdout:^* hv03.front.sepia.ceph.com 3 8 377 6 +2532us[+2509us] +/- 73ms 2023-05-25T04:33:47.273 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-05-25T04:33:47.312 INFO:teuthology.orchestra.run.smithi157.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-05-25T04:33:47.313 INFO:teuthology.orchestra.run.smithi157.stdout:=============================================================================== 2023-05-25T04:33:47.313 INFO:teuthology.orchestra.run.smithi157.stdout:^+ hv01.front.sepia.ceph.com 2 8 377 256 -2712us[-2773us] +/- 76ms 2023-05-25T04:33:47.313 INFO:teuthology.orchestra.run.smithi157.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 45 -541us[ -541us] +/- 88ms 2023-05-25T04:33:47.313 INFO:teuthology.orchestra.run.smithi157.stdout:^* hv03.front.sepia.ceph.com 3 7 377 88 +2441us[+2430us] +/- 72ms 2023-05-25T04:33:47.313 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-05-25T04:33:47.314 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-05-25T04:33:47.323 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-05-25T04:33:47.324 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-05-25T04:33:47.334 DEBUG:teuthology.orchestra.run.smithi071:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-05-25T04:33:47.451 DEBUG:teuthology.orchestra.run.smithi157:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-05-25T04:33:47.604 DEBUG:teuthology.orchestra.run.smithi071:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-05-25T04:33:47.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-25T04:33:47.639 DEBUG:teuthology.orchestra.run.smithi157:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-05-25T04:33:47.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-25T04:33:47.671 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-05-25T04:33:47.682 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-05-25T04:33:47.692 INFO:teuthology.task.internal:Duration was 28046.731913 seconds 2023-05-25T04:33:47.692 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-05-25T04:33:47.702 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-05-25T04:33:47.702 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-05-25T04:33:47.704 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-05-25T04:33:47.751 INFO:teuthology.orchestra.run.smithi071.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-05-25T04:33:47.818 INFO:teuthology.orchestra.run.smithi157.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-05-25T04:33:48.218 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-05-25T04:33:48.219 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi071.front.sepia.ceph.com 2023-05-25T04:33:48.219 DEBUG:teuthology.orchestra.run.smithi071:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-05-25T04:33:48.245 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi157.front.sepia.ceph.com 2023-05-25T04:33:48.246 DEBUG:teuthology.orchestra.run.smithi157:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-05-25T04:33:48.269 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-05-25T04:33:48.269 DEBUG:teuthology.orchestra.run.smithi071:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-05-25T04:33:48.288 DEBUG:teuthology.orchestra.run.smithi157:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-05-25T04:33:48.348 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-05-25T04:33:48.349 DEBUG:teuthology.orchestra.run.smithi071:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-05-25T04:33:48.368 DEBUG:teuthology.orchestra.run.smithi157:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-05-25T04:33:48.463 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-05-25T04:33:48.474 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-05-25T04:33:48.474 DEBUG:teuthology.orchestra.run.smithi071:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-05-25T04:33:48.501 DEBUG:teuthology.orchestra.run.smithi157:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-05-25T04:33:48.562 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-05-25T04:33:48.573 DEBUG:teuthology.orchestra.run.smithi071:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-05-25T04:33:48.575 DEBUG:teuthology.orchestra.run.smithi157:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-05-25T04:33:48.600 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern = core 2023-05-25T04:33:48.623 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern = core 2023-05-25T04:33:48.646 DEBUG:teuthology.orchestra.run.smithi071:> test -e /home/ubuntu/cephtest/archive/coredump 2023-05-25T04:33:48.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-25T04:33:48.703 DEBUG:teuthology.orchestra.run.smithi157:> test -e /home/ubuntu/cephtest/archive/coredump 2023-05-25T04:33:48.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-25T04:33:48.721 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-05-25T04:33:48.731 INFO:teuthology.task.internal:Transferring archived files... 2023-05-25T04:33:48.733 DEBUG:teuthology.misc:Transferring archived files from smithi071:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-05-24_20:32:11-rgw-wip-yuri-testing-2023-05-24-0738-reef-distro-default-smithi/7285505/remote/smithi071 2023-05-25T04:33:48.733 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-05-25T04:33:48.778 DEBUG:teuthology.misc:Transferring archived files from smithi157:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-05-24_20:32:11-rgw-wip-yuri-testing-2023-05-24-0738-reef-distro-default-smithi/7285505/remote/smithi157 2023-05-25T04:33:48.778 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-05-25T04:33:48.813 INFO:teuthology.task.internal:Removing archive directory... 2023-05-25T04:33:48.813 DEBUG:teuthology.orchestra.run.smithi071:> rm -rf -- /home/ubuntu/cephtest/archive 2023-05-25T04:33:48.815 DEBUG:teuthology.orchestra.run.smithi157:> rm -rf -- /home/ubuntu/cephtest/archive 2023-05-25T04:33:48.865 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-05-25T04:33:48.877 INFO:teuthology.task.internal:Not uploading archives. 2023-05-25T04:33:48.877 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-05-25T04:33:48.895 INFO:teuthology.task.internal:Tidying up after the test... 2023-05-25T04:33:48.896 DEBUG:teuthology.orchestra.run.smithi071:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-05-25T04:33:48.898 DEBUG:teuthology.orchestra.run.smithi157:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-05-25T04:33:48.913 INFO:teuthology.orchestra.run.smithi071.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 25 04:33 /home/ubuntu/cephtest 2023-05-25T04:33:48.923 INFO:teuthology.orchestra.run.smithi157.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 25 04:33 /home/ubuntu/cephtest 2023-05-25T04:33:48.925 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-05-25T04:33:48.936 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-05-25T04:33:48.957 INFO:teuthology.nuke:Checking targets against current locks 2023-05-25T04:33:48.986 DEBUG:teuthology.nuke:shortname: smithi071 2023-05-25T04:33:48.986 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-05-25T04:33:49.016 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi071.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-05-24_20:32:11-rgw-wip-yuri-testing-2023-05-24-0738-reef-distro-default-smithi/7285505', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-05-24 20:36:17.087875', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA5CM/4hsfiqXE11jZAHgpOo2FbW4ZOzUXKXNJNz9fWWeDtJTYStjEV6efAd9vMM91OzJdMX3iMjxJTS8Y5i2uY='} 2023-05-25T04:33:49.019 DEBUG:teuthology.nuke:shortname: smithi157 2023-05-25T04:33:49.019 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-05-25T04:33:49.044 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi157.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-05-24_20:32:11-rgw-wip-yuri-testing-2023-05-24-0738-reef-distro-default-smithi/7285505', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-05-24 20:36:17.089397', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO+lmSy90RyAYeFfSD6aXgxRUJPCkLZkHkQxsxqLXA3D4XXntKSNvOhxkzpRSCW+Uo3pl5PdjBfPz+AOeUYtCwU='} 2023-05-25T04:33:49.070 INFO:teuthology.orchestra.console:Power off smithi071 2023-05-25T04:33:49.071 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-05-25T04:33:49.106 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-05-25T04:33:49.129 INFO:teuthology.orchestra.console:Power off smithi157 2023-05-25T04:33:49.130 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-05-25T04:33:49.153 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-05-25T04:33:53.142 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-05-25T04:33:53.176 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-05-25T04:33:57.258 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-05-25T04:33:57.290 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-05-25T04:33:57.471 INFO:teuthology.orchestra.console:Power off for smithi071 completed 2023-05-25T04:33:57.502 INFO:teuthology.orchestra.console:Power off for smithi157 completed 2023-05-25T04:33:57.606 INFO:teuthology.run:Summary data: description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8} tasks/test_multi} duration: 28046.731913089752 failure_reason: rgw multisite test failures flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=63f94308abe849c5a88590818a2c9577 status: fail success: false 2023-05-25T04:33:57.607 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-05-25T04:33:57.607 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-05-25T04:33:57.743 INFO:teuthology.run:FAIL